RocketMQ on F嘉阳

RocketMQ学习笔记6——集群实战

Wed, 23 Dec 2020 19:15:00 +0800

组建高性能MQ服务

双主双从模式
保障无单点故障问题的高可用集群

集群拓扑图

IP	资源	服务
10.139.12.52	1C2G20GB	rocketmq-nameserver1、rocketmq-master1
10.139.12.53	1C2G20GB	rocketmq-nameserver2、rocketmq-master2
10.139.12.54	1C2G20GB	rocketmq-nameserver3、rocketmq-master1-slave
10.139.12.55	1C2G20GB	rocketmq-nameserver4、rocketmqmaster2-slave

配置文件最优设置解析

修改host文件映射

10.139.12.52 rocketmq-nameserver1 rocketmq-master1
10.139.12.53 rocketmq-nameserver2 rocketmq-master2
10.139.12.54 rocketmq-nameserver3 rocketmq-master1-slave
10.139.12.55 rocketmq-nameserver4 rocketmq-master2-slave

将4个节点的hosts均改为如上内容，并将4.4.0版本包放入环境中解压

创建存储目录

$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index

建立软连接

$ cd ~/rocketmq-test
$ ln -s rocketmq-all-4.4.0-bin-release rocketmq

修改配置文件

解压后默认的配置文件如下

[user@301 ~/rocketmq-test/rocketmq/conf]$ll
total 36
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync
drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave
-rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf
-rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml
-rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml
-rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml
-rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml
-rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml

若想保障生产环境高可用，推荐使用双主双从同步双写模式，可以让一个主节点对应多个从节点，但从节点越多，同步效率越慢

本次采用双主双从同步双写模式，一个主节点对应一个从节点

[user@301 ~/rocketmq-test/rocketmq/conf/2m-2s-sync]$ll
total 16
-rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-a.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties
-rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-b.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties

用途如下

IP	资源	服务	配置文件
10.139.12.52	1C2G20GB	rocketmq-nameserver1、rocketmq-master1	broker-a.properties
10.139.12.53	1C2G20GB	rocketmq-nameserver2、rocketmq-master2	broker-b.properties
10.139.12.54	1C2G20GB	rocketmq-nameserver3、rocketmq-slave1	broker-a-s.properties
10.139.12.55	1C2G20GB	rocketmq-nameserver4、rocketmq-slave2	broker-b-s.properties

`broker-a.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=false
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=false
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间，最大100%
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-b.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-b
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=false
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=false
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间，最大100%
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-a-s.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-b-s.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对
brokerName=broker-b
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

分发配置文件

$ scp broker-* ocmuser@rocketmq-master2-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync
$ scp broker-* ocmuser@rocketmq-master2:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync
$ scp broker-* ocmuser@rocketmq-master1-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync

修改日志配置文件

$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq/logs
$ cd /home/ocmuser/rocketmq-test/rocketmq/conf && sed -i 's#${user.home}#/home/ocmuser/rocketmq-test/#g' *.xml

修改脚本启动参数

/home/ocmuser/rocketmq-test/rocketmq/bin/runbroker.sh内容如下，至少保障1g内存

#======================================================================================
# 开发环境JVM Configuration
#======================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"

/home/ocmuser/rocketmq-test/rocketmq/bin/runserver.sh内容如下

#======================================================================================
# JVM Configuration
#======================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

启动 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin
$ nohup sh mqnamesrv &

启动 Master1:BrokerServerA

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master2:BrokerServerB

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master1-Slave:BrokerServerC

$ cd cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a-s.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master2-Slave:BrokerServerD

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b-s.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

查看进程和日志，可见部署完成

角色为同步双写，异步刷盘，4个节点通信正常

控制台监控验证

修改连接配置

rocketmq.config.namesrvAddr=10.139.12.51:9876;10.139.12.52:9876;10.139.12.53:9876;10.139.12.54:9876

连接正常

查看各节点角色和状态

服务停止

首先关闭 4 个 BrokerServer，再关闭 4 个 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ sh mqshutdown broker 
$ sh mqshutdown namesrv

双主双从环境构建与调优

为取得可靠性与效率的平衡，建议采用同步双写，异步刷盘的模式

RocketMQ学习笔记5——原理解析

Wed, 23 Dec 2020 19:13:00 +0800

消息存储结构

CommitLog：消息主体以及元数据的存储主体，存储Producer端写入的消息主体内容,消息内容不是定长的。消息主要是顺序写入日志文件，当文件满了，写入下一个文件；其日志信息科用于数据恢复，作为MQ数据保障兜底策略。
ConsumerQueue：消息消费队列，存储消息在commit log 中所在的这个位置，也可以说是一个偏移量.引入的目的主要是提高消息消费的性能。
IndexFile：IndexFile（索引文件）提供了一种可以通过key或时间区间来查询消息的方法。
每一个topic下的每一个message queue都对应一个ConsumerQueue文件，具体存储位置通过配置文件配置。
RocketMQ采用的是混合型的存储结构，即为Broker单个实例下所有的队列（ConsumerQueue）共用一个日志数据文件（即为CommitLog）来存储。
刷盘采用顺序写随机读的模式。
消费端采用“0拷贝”模式提升性能。

同步\异步刷盘

消息存储：内存+磁盘存储，两种刷盘方式

异步刷盘

能够充分利用OS的PageCache的优势，只要消息写入PageCache（内存）即可将成功的ACK（成功信息）返回给Producer端。
消息堆积到一定量才触发刷盘。
消息刷盘采用后台异步线程提交的方式进行，降低了读写延迟，提高了MQ的性能和吞吐量。

同步刷盘

只有在消息真正持久化至磁盘后RocketMQ的Broker端才会真正返回给Producer端一个成功的ACK响应。
同步刷盘对MQ消息可靠性来说是一种不错的保障，但是性能上会有较大影响，一般适用于金融业务应用该模式较多。

同步\异步复制

针对集群模式设计，同一组Broker有Master-Slave角色
推荐采用同步双写，异步复制的模式保持性能与可靠性的平衡

同步复制

Master与Slave都成功才返回ACK（成功信息）给Producer

异步复制

Master节点刷盘成功后立即返回ACK（成功信息）给Producer
再通过异步线程将信息同步给其他Slave节点
高性能，但存在数据丢失风险

同步双写

每个Master配置一个Slave，有多对Master-Slave，HA采用同步双写方式，即只有主备都写成功，才向应用返回成功，这种模式的优缺点如下：

优点：数据与服务都无单点故障，Master宕机情况下，消息无延迟，服务可用性与数据可用性都非常高；
缺点：性能比异步复制模式略低（大约低10%左右），发送单个消息的RT会略高，且目前版本在主节点宕机后，备机不能自动切换为主机。

高可用机制

Master-Slave搭配实现高可用机制
通过BrokerId区分节点
Master支持读写，Slave只读
当Master繁忙或者不可用时，可以自动切换到Slave读取消息

最佳实践

创建Topic时，将Topic的Message Queue创建到多个Broker组中，使得逻辑上高可用
由于RocketMQ不支持主从切换，故可以考虑使用监控程序监听主节点的有效性，若连接超时，则将Slave节点的配置改为Master并重启，升级为主节点，实现高可用。

NameServer协调者

NameServer是一个非常简单的Topic路由注册中心，其角色类似Dubbo中的zookeeper，支持Broker的动态注册与发现，是整个集群的状态服务器。
主要包括两个功能：
- Broker管理，NameServer接受Broker集群的注册信息并且保存下来作为路由信息的基本数据。然后提供心跳检测机制，检查Broker是否还存活；
- 路由信息管理，每个NameServer将保存关于Broker集群的整个路由信息和用于客户端查询的队列信息。然后Producer和Conumser通过NameServer就可以知道整个Broker集群的路由信息，从而进行消息的投递和消费。
NameServer通常也是集群的方式部署，各实例间相互不进行信息通讯。Broker是向每一台NameServer注册自己的路由信息，所以每一个NameServer实例上面都保存一份完整的路由信息。
当某个NameServer因某种原因下线了，Broker仍然可以向其它NameServer同步其路由信息，Producer,Consumer仍然可以动态感知Broker的路由的信息。

为什么不用zookeeper

RocketMQ 3.x以前使用zookeeper实现，后来改为nameserver实现
zookeeper提供主从选举，广播原子性等特性在MQ中用不上，功能冗余
zk使用主从自动切换提供高可用保障，nameserver使用多副本节点保障高可用
nameserver只提供路由等基本状态信息维护，代码量小，更加轻量，便于维护。

NameServer信息维护源码

package org.apache.rocketmq.namesrv.routeinfo;

public class RouteInfoManager {
 private final ReadWriteLock lock = new ReentrantReadWriteLock();
 private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;
 private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
 private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
 private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
 private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;
}

RocketMQ学习笔记4——消费者核心

Wed, 23 Dec 2020 19:11:00 +0800

配置参数详解

PushConsumer核心参数

ConsumeFromWhere源码，支持三种参数配置，默认是CONSUME_FROM_LAST_OFFSET

package org.apache.rocketmq.common.consumer;

public enum ConsumeFromWhere {
 /**
 * 第一次启动时从队列最后开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_LAST_OFFSET,

 @Deprecated
 CONSUME_FROM_LAST_OFFSET_AND_FROM_MIN_WHEN_BOOT_FIRST,
 @Deprecated
 CONSUME_FROM_MIN_OFFSET,
 @Deprecated
 CONSUME_FROM_MAX_OFFSET,
 /**
 * 从队列最开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_FIRST_OFFSET,
 /**
 * 指定具体时间点开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_TIMESTAMP,
}

AllocateMessageQueueStrategy接口用于集群模式下消息策略，默认配置为平均消息分配器AllocateMessageQueueAveragely

默认配置源码

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Constructor specifying consumer group and enabled msg trace flag.
 *
 * @param consumerGroup Consumer group.
 * @param enableMsgTrace Switch flag instance for message trace.
 */
 public DefaultMQPushConsumer(final String consumerGroup, boolean enableMsgTrace) {
 this(null, consumerGroup, null, new AllocateMessageQueueAveragely(), enableMsgTrace, null);
 }
}

subscription订阅参数，传递两个值，一个是主题，一个是表达式tag过滤

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Subscription relationship
 */
 private Map<String /* topic */, String /* sub expression */> subscription = new HashMap<String, String>();
}

样例：

consumer.subscribe("test_model_topic2", "TagA||TagB");

consumer.subscribe("test_model_topic2", "TagA");

consumer.subscribe("test_model_topic2", "*");

offsetStore配置核心偏移量，有两个实现，用于支持远程存储和本地存储，即支持集群和单机模式

consumeThreadMin与consumeThreadMax线程池自动调整参数，默认为20

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Minimum consumer thread number
 */
 private int consumeThreadMin = 20;

 /**
 * Max consumer thread number
 */
 private int consumeThreadMax = 20;
}

流控参数consumeConcurrentlyMaxSpan与pullThresholdForQueue

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Concurrently max span offset.it has no effect on sequential consumption
 * 单个队列并行消费的最大跨度
 */
 private int consumeConcurrentlyMaxSpan = 2000;

 /**
 * Flow control threshold on queue level, each message queue will cache at most 1000 messages by default,
 * Consider the {@code pullBatchSize}, the instantaneous value may exceed the limit
 * 一个队列最大的消息消费个数
 */
 private int pullThresholdForQueue = 1000;
}

消息拉取参数pullInterval与pullBatchSize

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Message pull Interval
 * 消息拉取的时间间隔
 */
 private long pullInterval = 0;

 /**
 * Batch pull size
 * 一次拉取的数据量
 */
 private int pullBatchSize = 32;
}

consumeMessageBatchMaxSize一次消息最大拉取的数据数量配置

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Batch consumption size
 * 一次消息最大拉取的数据数量
 */
 private int consumeMessageBatchMaxSize = 1;
}

MessageListener用于消息监听配置，包括并发、顺序读取等

MessageModel配置消息模式，广播或者集群，默认为集群模式

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Message model defines the way how messages are delivered to each consumer clients.
 * </p>
 *
 * RocketMQ supports two message models: clustering and broadcasting. If clustering is set, consumer clients with
 * the same {@link #consumerGroup} would only consume shards of the messages subscribed, which achieves load
 * balances; Conversely, if the broadcasting is set, each consumer client will consume all subscribed messages
 * separately.
 * </p>
 *
 * This field defaults to clustering.
 */
 private MessageModel messageModel = MessageModel.CLUSTERING;
}

集群与广播模式

集群模式

MessageModel源码支持两种模式，集群和广播

package org.apache.rocketmq.common.protocol.heartbeat;

/**
 * Message model
 */
public enum MessageModel {
 /**
 * broadcast
 */
 BROADCASTING("BROADCASTING"),
 /**
 * clustering（默认）
 */
 CLUSTERING("CLUSTERING");

 private String modeCN;

 MessageModel(String modeCN) {
 this.modeCN = modeCN;
 }

 public String getModeCN() {
 return modeCN;
 }
}

集群模式下GroupName用于把多个Consumer组织到一起，所以同一个集群，GroupName必须相同
相同的GroupName的Consumer只消费所订阅消息的一部分，即支持负载均衡
优势：集群模式下增加节点能动态增加负载均衡
一条消息只会被一个Consumer消费

样例

消息消费与奇偶性无关，随机负载均衡，并且负载均衡存在一定的差值，并非100%平分
为提高附在均衡度，可以用2个consume监听4个队列，或者3个consume监听6个队列
Producer只能保证平均分配到各个队列，不能保证被平均的消费

生产者

public class Producer {
	
 public static void main(String[] args) throws MQClientException, InterruptedException {
 	String group_name = "test_model_producer_name";
 DefaultMQProducer producer = new DefaultMQProducer(group_name);
 producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 producer.start();
 
 for (int i = 0; i < 10; i++) {
 try {
 	String tag = (i % 2 ==0) ? "TagA":"TagB";
 Message msg = new Message("test_model_topic2",// topic
 		tag,// tag
 ("信息内容" + i).getBytes()// body
 );
 SendResult sendResult = producer.send(msg);
 System.out.println(sendResult);
 }
 catch (Exception e) {
 e.printStackTrace();
 Thread.sleep(1000);
 }
 }
 
 producer.shutdown();
 }
}

消费者1

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.CLUSTERING);
//			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					//if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					//}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
		
	}
}

消费者2

public class Consumer2 {

	public Consumer2() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.CLUSTERING);
//			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					//if(tags.equals("TagB")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					//}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer2 c2 = new Consumer2();
		System.out.println("c2 start..");
		
	}
}

发送10条消息结果

消费者1结果

消费者2结果

注意，若使用集群模式，则tag表达式应该为*号，才能支持多consumer消费，若集群模式下指定tag表达式，则会导致消费结果不确定，原因如下

consumerA监听4条队列中的2条，consumerB监听4条队列中的另外2条。
Producer将包含TagA和TagB的消息均分到4条队列中。
consumerA消费TagA的消息，那么此时consumerA监听的2条队列中的TagB消息则永远不会被消费

样例

consumer1改成监听TagA，其他内容不变

consumer.subscribe("test_model_topic2", "TagA");

此时发送10条消息，consumer1监听的结果

此时将consumer2改成监听TagB并启动

consumer.subscribe("test_model_topic2", "TagB");

此时发送10条消息，consumer1监听的结果

consumer2监听结果为空，大量消息丢失

经过实测发现，只要集群模式下指定了tag，那么只有一个consumer能正常进行消费，其他consumer都为空

广播模式

同一个ConsumeGroup里的Consumer都消费订阅Topic全部信息
也就是一条消息会被每个Consume消费

样例

将consumer1和2改成广播模式，并监听所有tag

consumer.setMessageModel(MessageModel.BROADCASTING);
consumer.subscribe("test_model_topic2", "*");

发送10条消息consumer1消费结果

consumer2消费结果

注意广播模式下使用服务端tag过滤仍然有bug

将consumer1设置为只监听tagA消息

consumer.subscribe("test_model_topic2", "TagA");

consumer2设置为只监听tag2消息

consumer.subscribe("test_model_topic2", "TagB");

发送10条消息consumer1消费结果

consumer2消费结果

结果只有其中一个consumer能正常消费，与集群模式一样的问题

目前（4.4.0）若想要实现同一个topic的tag过滤，则只能让consumer监听所有消息并主动判断

样例：

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
	}
}

发送10条消息结果

消费者1结果

消费者2结果

对于consumer消费指定队列消息，推荐做法是将消息投递到专属的队列中，由同一个consumer监听，即相当于用groupname+topic唯一定位一个consumer，该方法集群与广播模式均适用

样例

此时consumer1变为test_model_consumer_name1

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name1";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "TagA");
//			consumer.setMessageModel(MessageModel.CLUSTERING);
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
	}
}

consumer2变为test_model_consumer_name2

public class Consumer2 {

	public Consumer2() {
		try {
			String group_name = "test_model_consumer_name2";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "TagB");
//			consumer.setMessageModel(MessageModel.CLUSTERING);
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagB")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer2 c2 = new Consumer2();
		System.out.println("c2 start..");
	}
}

发送10条消息结果

消费者1结果

消费者2结果

消费存储核心-OffSet

Offset是消费者消费进度的核心
Offset指某个topic下的一条消息在某个MessageQueue里的位置
通过Offset可以进行定位到这条消息
Offset的存储实现分为远程文件类型和本地文件类型两种

集群模式-`RemoteBrokerOffsetStore`

默认集群模式clustering，采用远程文件存储Offset
本质上因为多消费模式，每个Consumer消费所订阅主题的一部分

广播模式-`LocalFileOffsetStore`

广播模式下，由于每个Consumer都会收到消息且消费
各个Consumer之间没有任何干扰，独立线程消费
所以使用LocalFileOffsetStore，也就是把Offset存储到本地

PushConsumer长轮询模式分析

DefaultMQPushConsumer是使用长轮询模式进行实现的
通常主流的消息获取模式：Push消息推送模式与Pull消息拉取模式
Push模式消耗broker端性能，若消费端消费速度不一致，可能导致消息堆积
pull模式消耗消费端性能，消费端不知道消息达到时间
RocketMQ采用长轮询机制（消费端主动获取）

长轮询过程

客户端发起获取消息请求，假设一次请求超时时间为15s
服务端判断是否有消息，有则直接返回
服务端无消息，则等待5s（阈值），5s后再次检查是否有消息，有则返回，无则继续等待
15s后循环检查结束，若仍然无消息，则返回给客户端

优势：

主动权在客户端，当服务端消息堆积，不会全推送到客户端

缺点：

连接返回时间较长，占用客户端资源（请求阻塞，等待服务端返回）

源码分析

源码实现位置

PullRequestHoldService继承自Runnable接口

核心类PullRequestHoldService源码全貌

package org.apache.rocketmq.broker.longpolling;

public class PullRequestHoldService extends ServiceThread {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
 private static final String TOPIC_QUEUEID_SEPARATOR = "@";
 private final BrokerController brokerController;
 private final SystemClock systemClock = new SystemClock();
 private ConcurrentMap<String/* topic@queueId */, ManyPullRequest> pullRequestTable =
 new ConcurrentHashMap<String, ManyPullRequest>(1024);

 public PullRequestHoldService(final BrokerController brokerController) {
 this.brokerController = brokerController;
 }

 public void suspendPullRequest(final String topic, final int queueId, final PullRequest pullRequest) {
 String key = this.buildKey(topic, queueId);
 ManyPullRequest mpr = this.pullRequestTable.get(key);
 if (null == mpr) {
 mpr = new ManyPullRequest();
 ManyPullRequest prev = this.pullRequestTable.putIfAbsent(key, mpr);
 if (prev != null) {
 mpr = prev;
 }
 }

 mpr.addPullRequest(pullRequest);
 }

 private String buildKey(final String topic, final int queueId) {
 StringBuilder sb = new StringBuilder();
 sb.append(topic);
 sb.append(TOPIC_QUEUEID_SEPARATOR);
 sb.append(queueId);
 return sb.toString();
 }

 @Override
 public void run() {
 log.info("{} service started", this.getServiceName());
 while (!this.isStopped()) {
 try {
 // 判断是否启用长轮询
 if (this.brokerController.getBrokerConfig().isLongPollingEnable()) {
 // 等待5s
 this.waitForRunning(5 * 1000);
 } else {
 this.waitForRunning(this.brokerController.getBrokerConfig().getShortPollingTimeMills());
 }

 long beginLockTimestamp = this.systemClock.now();
 // 检查请求
 this.checkHoldRequest();
 long costTime = this.systemClock.now() - beginLockTimestamp;
 if (costTime > 5 * 1000) {
 log.info("[NOTIFYME] check hold request cost {} ms.", costTime);
 }
 } catch (Throwable e) {
 log.warn(this.getServiceName() + " service has exception. ", e);
 }
 }

 log.info("{} service end", this.getServiceName());
 }

 @Override
 public String getServiceName() {
 return PullRequestHoldService.class.getSimpleName();
 }

 private void checkHoldRequest() {
 // 遍历所有请求的channel
 for (String key : this.pullRequestTable.keySet()) {
 String[] kArray = key.split(TOPIC_QUEUEID_SEPARATOR);
 if (2 == kArray.length) {
 String topic = kArray[0];
 int queueId = Integer.parseInt(kArray[1]);
 final long offset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
 try {
 // 唤醒
 this.notifyMessageArriving(topic, queueId, offset);
 } catch (Throwable e) {
 log.error("check hold request failed. topic={}, queueId={}", topic, queueId, e);
 }
 }
 }
 }

 public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset) {
 notifyMessageArriving(topic, queueId, maxOffset, null, 0, null, null);
 }

 public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset, final Long tagsCode,
 long msgStoreTime, byte[] filterBitMap, Map<String, String> properties) {
 String key = this.buildKey(topic, queueId);
 // 取出等待的channel（请求）
 ManyPullRequest mpr = this.pullRequestTable.get(key);
 if (mpr != null) {
 List<PullRequest> requestList = mpr.cloneListAndClear();
 if (requestList != null) {
 List<PullRequest> replayList = new ArrayList<PullRequest>();
 // 循环所有的请求
 for (PullRequest request : requestList) {
 long newestOffset = maxOffset;
 if (newestOffset <= request.getPullFromThisOffset()) {
 newestOffset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
 }

 if (newestOffset > request.getPullFromThisOffset()) {
 boolean match = request.getMessageFilter().isMatchedByConsumeQueue(tagsCode,
 new ConsumeQueueExt.CqExtUnit(tagsCode, msgStoreTime, filterBitMap));
 // match by bit map, need eval again when properties is not null.
 if (match && properties != null) {
 match = request.getMessageFilter().isMatchedByCommitLog(null, properties);
 }
 // 如果匹配上
 if (match) {
 try {
 // 唤醒request
 this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
 request.getRequestCommand());
 } catch (Throwable e) {
 log.error("execute request when wakeup failed.", e);
 }
 continue;
 }
 }

 if (System.currentTimeMillis() >= (request.getSuspendTimestamp() + request.getTimeoutMillis())) {
 try {
 this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
 request.getRequestCommand());
 } catch (Throwable e) {
 log.error("execute request when wakeup failed.", e);
 }
 continue;
 }

 replayList.add(request);
 }

 if (!replayList.isEmpty()) {
 mpr.addPullRequest(replayList);
 }
 }
 }
 }
}

ManyPullRequest里存储的实际为PullRequest数组

public class ManyPullRequest {
 private final ArrayList<PullRequest> pullRequestList = new ArrayList<>();
	...
}

PullRequest实际存储NIO请求

public class PullRequest {
 private final RemotingCommand requestCommand;
 /** NIO channel */
 private final Channel clientChannel;
 private final long timeoutMillis;
 private final long suspendTimestamp;
 private final long pullFromThisOffset;
 private final SubscriptionData subscriptionData;
 private final MessageFilter messageFilter;
	...
}

`DefaultMQPullConsumer`消息拉取模式

实现类DefaultMQPullConsumer
主要逻辑
- 获取Message Queue并遍历
- 维护OffsetStore
- 根据不同消息状态做不同处理

生产者样例

public class Producer {
 public static void main(String[] args) throws MQClientException, InterruptedException {
 	
 	String group_name = "test_pull_producer_name";
 DefaultMQProducer producer = new DefaultMQProducer(group_name);
 producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 producer.start();
 
 for (int i = 0; i < 10; i++) {
 try {
 Message msg = new Message("test_pull_topic",// topic
 		"TagA",// tag
 		("Hello RocketMQ " + i).getBytes()// body
 );
 // 配置延迟
 msg.setDelayTimeLevel(2);
 
 SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
 // 外部方法的arg会传递到Object中
 @Override
 public MessageQueue select(List<MessageQueue> list, Message message, Object o) {
 Integer queueNumber = (Integer) o;
 return list.get(queueNumber);
 }
 },1);

 SendStatus status = sendResult.getSendStatus();
 System.out.println(status);

 Thread.sleep(1000);
 }
 catch (Exception e) {
 e.printStackTrace();
 Thread.sleep(3000);
 }
 }
 producer.shutdown();
 }
}

发送消息后的结果，在队列1有45条消息

使用普通Pull轮询模式

public class PullConsumer {
	//Map<key, value> key为指定的队列，value为这个队列拉取数据的最后位置
 private static final Map<MessageQueue, Long> offseTable = new HashMap<MessageQueue, Long>();
 
 public static void main(String[] args) throws MQClientException {
 	
 	String group_name = "test_pull_consumer_name";
 DefaultMQPullConsumer consumer = new DefaultMQPullConsumer(group_name);
 consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 consumer.start();
 System.err.println("consumer start");
 //	从TopicTest这个主题去获取所有的队列（默认会有4个队列）
 Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("test_pull_topic");
 //	遍历每一个队列，进行拉取数据
 for (MessageQueue mq : mqs) {
 System.out.println("Consume from the queue: " + mq);
 
 SINGLE_MQ: while (true) {
 try {
 	//	从queue中获取数据，从什么位置开始拉取数据 单次对多拉取32条记录
 PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
 System.out.println(pullResult);
 System.out.println(pullResult.getPullStatus());
 System.out.println();
 // 客户端自行维护Offset
 putMessageQueueOffset(mq, pullResult.getNextBeginOffset());
 switch (pullResult.getPullStatus()) {
	 case FOUND:
	 	List<MessageExt> list = pullResult.getMsgFoundList();
	 	for(MessageExt msg : list){
	 		System.out.println(new String(msg.getBody()));
	 	}
	 break;
	 case NO_MATCHED_MSG:
	 break;
	 case NO_NEW_MSG:
	 	System.out.println("没有新的数据啦...");
	 break SINGLE_MQ;
	 case OFFSET_ILLEGAL:
	 break;
	 default:
	 break;
 }
 }
 catch (Exception e) {
 e.printStackTrace();
 }
 }
 }
 consumer.shutdown();
 }
 
 
 private static void putMessageQueueOffset(MessageQueue mq, long offset) {
 offseTable.put(mq, offset);
 }
 
 
 private static long getMessageQueueOffset(MessageQueue mq) {
 Long offset = offseTable.get(mq);
 if (offset != null)
 return offset;
 return 0;
 }
}

消费结果，其会随机选择队列读取，导致服务端积压的消息得不到有效的消费

而且该模式需要客户端编码实现Offset存储

Pull定时任务实现样例

public class PullScheduleService {

 public static void main(String[] args) throws MQClientException {
 	
 	String group_name = "test_pull_consumer_name";
 	// 使用定时任务
 final MQPullConsumerScheduleService scheduleService = new MQPullConsumerScheduleService(group_name);
 
 scheduleService.getDefaultMQPullConsumer().setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 
 scheduleService.setMessageModel(MessageModel.CLUSTERING);
 
 scheduleService.registerPullTaskCallback("test_pull_topic", new PullTaskCallback() {
 
 @Override
 public void doPullTask(MessageQueue mq, PullTaskContext context) {
 MQPullConsumer consumer = context.getPullConsumer();
 System.err.println("-------------- queueId: " + mq.getQueueId() + "-------------");
 try {
 // 获取从哪里拉取
 long offset = consumer.fetchConsumeOffset(mq, false);
 if (offset < 0)
 offset = 0;
 
 PullResult pullResult = consumer.pull(mq, "*", offset, 32);
 switch (pullResult.getPullStatus()) {
 case FOUND:
 	List<MessageExt> list = pullResult.getMsgFoundList();
 	for(MessageExt msg : list){
 		//消费数据...
 		System.out.println(new String(msg.getBody()));
 	}
 break;
 case NO_MATCHED_MSG:
 break;
 case NO_NEW_MSG:
 case OFFSET_ILLEGAL:
 break;
 default:
 break;
 }
 // 客户端自动更新Offset，无需本地持久化维护
 consumer.updateConsumeOffset(mq, pullResult.getNextBeginOffset());
 // 设置再过3000ms后重新拉取
 context.setPullNextDelayTimeMillis(3000);
 
 }
 catch (Exception e) {
 e.printStackTrace();
 }
 }
 });
 
 scheduleService.start();
 }
}

运行结果，由于采用多线程读取所有队列，所以当服务端有消息时可以及时获取

实现原理

MQPullConsumerScheduleService中将PullTaskCallback方法作为Map存储
并使用ScheduledThreadPoolExecutor作为线程池，配置线程数为10
线程通过调度taskTable中的PullTaskImpl任务，实现方法回调

package org.apache.rocketmq.client.consumer;

public class MQPullConsumerScheduleService {
 private final InternalLogger log = ClientLogger.getLog();
 private final MessageQueueListener messageQueueListener = new MessageQueueListenerImpl();
 private final ConcurrentMap<MessageQueue, PullTaskImpl> taskTable =
 new ConcurrentHashMap<MessageQueue, PullTaskImpl>();
 private DefaultMQPullConsumer defaultMQPullConsumer;
 private int pullThreadNums = 20;
 private ConcurrentMap<String /* topic */, PullTaskCallback> callbackTable =
 new ConcurrentHashMap<String, PullTaskCallback>();
 private ScheduledThreadPoolExecutor scheduledThreadPoolExecutor;

 public MQPullConsumerScheduleService(final String consumerGroup) {
 this.defaultMQPullConsumer = new DefaultMQPullConsumer(consumerGroup);
 this.defaultMQPullConsumer.setMessageModel(MessageModel.CLUSTERING);
 }

 public void putTask(String topic, Set<MessageQueue> mqNewSet) {
 Iterator<Entry<MessageQueue, PullTaskImpl>> it = this.taskTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<MessageQueue, PullTaskImpl> next = it.next();
 if (next.getKey().getTopic().equals(topic)) {
 if (!mqNewSet.contains(next.getKey())) {
 next.getValue().setCancelled(true);
 it.remove();
 }
 }
 }

 for (MessageQueue mq : mqNewSet) {
 if (!this.taskTable.containsKey(mq)) {
 // 需要调度的任务
 PullTaskImpl command = new PullTaskImpl(mq);
 this.taskTable.put(mq, command);
 // 使用多线程调度
 this.scheduledThreadPoolExecutor.schedule(command, 0, TimeUnit.MILLISECONDS);

 }
 }
 }

 public void start() throws MQClientException {
 final String group = this.defaultMQPullConsumer.getConsumerGroup();
 this.scheduledThreadPoolExecutor = new ScheduledThreadPoolExecutor(
 this.pullThreadNums,
 new ThreadFactoryImpl("PullMsgThread-" + group)
 );

 this.defaultMQPullConsumer.setMessageQueueListener(this.messageQueueListener);

 this.defaultMQPullConsumer.start();

 log.info("MQPullConsumerScheduleService start OK, {} {}",
 this.defaultMQPullConsumer.getConsumerGroup(), this.callbackTable);
 }

 public void registerPullTaskCallback(final String topic, final PullTaskCallback callback) {
 // 放入回调任务集合
 this.callbackTable.put(topic, callback);
 this.defaultMQPullConsumer.registerMessageQueueListener(topic, null);
 }

 class MessageQueueListenerImpl implements MessageQueueListener {
 @Override
 public void messageQueueChanged(String topic, Set<MessageQueue> mqAll, Set<MessageQueue> mqDivided) {
 MessageModel messageModel =
 MQPullConsumerScheduleService.this.defaultMQPullConsumer.getMessageModel();
 switch (messageModel) {
 case BROADCASTING:
 MQPullConsumerScheduleService.this.putTask(topic, mqAll);
 break;
 case CLUSTERING:
 MQPullConsumerScheduleService.this.putTask(topic, mqDivided);
 break;
 default:
 break;
 }
 }
 }

 class PullTaskImpl implements Runnable {
 private final MessageQueue messageQueue;
 private volatile boolean cancelled = false;

 public PullTaskImpl(final MessageQueue messageQueue) {
 this.messageQueue = messageQueue;
 }

 @Override
 public void run() {
 String topic = this.messageQueue.getTopic();
 if (!this.isCancelled()) {
 // 获取回调任务
 PullTaskCallback pullTaskCallback =
 MQPullConsumerScheduleService.this.callbackTable.get(topic);
 if (pullTaskCallback != null) {
 final PullTaskContext context = new PullTaskContext();
 context.setPullConsumer(MQPullConsumerScheduleService.this.defaultMQPullConsumer);
 try {
 // 执行客户端实现的回调任务
 pullTaskCallback.doPullTask(this.messageQueue, context);
 } catch (Throwable e) {
 context.setPullNextDelayTimeMillis(1000);
 log.error("doPullTask Exception", e);
 }

 if (!this.isCancelled()) {
 MQPullConsumerScheduleService.this.scheduledThreadPoolExecutor.schedule(this,
 context.getPullNextDelayTimeMillis(), TimeUnit.MILLISECONDS);
 } else {
 log.warn("The Pull Task is cancelled after doPullTask, {}", messageQueue);
 }
 } else {
 log.warn("Pull Task Callback not exist , {}", topic);
 }
 } else {
 log.warn("The Pull Task is cancelled, {}", messageQueue);
 }
 }
 }
}

RocketMQ学习笔记3——生产者核心

Wed, 23 Dec 2020 19:09:00 +0800

配置参数

producerGroup：组名
createTopicKey：创建Topic（不允许使用，应当由架构设计定义）
defaultTopicQueueNums：创建topic时默认的队列数量（默认为4）
sendMsgTimeout：发送消息时的超时时间（默认值：3000，单位：毫秒）
compressMsgBodyOverHowmuch ：消息压缩字节（默认4096）
retryTimesWhenSendFailed ：同步模式下，在返回发送失败之前，内部尝试重新发送消息的最大次数。（默认值：2）
retryTimesWhenSendAsyncFailed：异步模式下，在发送失败之前，内部尝试重新发送消息的最大次数。（默认值：2）
retryAnotherBrokerWhenNotStoreOK：同步模式下，消息保存失败时是否重试其他broker 。（默认值：false ）
maxMessageSize：最大消息限制。（默认值：1024 * 1024 * 4，单位：字节）

主从同步机制

主从同步
同步信息：数据内容+元数据信息
元数据（配置信息）同步：Broker角色识别，为Slave则启动同步任务
- 通过Netty传输
- 实质为定时任务
消息同步（通过Socket）：HAService、HAconnection、WaitNotifyObject
- 通过原生Socket传输，性能高，数据包支持容量大

元数据同步源码分析

元信息同步工程源码位于broker中

发起定时任务源码BrokerController

package org.apache.rocketmq.broker;

public class BrokerController {
 public boolean initialize() throws CloneNotSupportedException {
 ...
 if (result) {
 ...
 // 进行角色判断
 if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) {
 if (this.messageStoreConfig.getHaMasterAddress() != null && this.messageStoreConfig.getHaMasterAddress().length() >= 6) {
 this.messageStore.updateHaMasterAddress(this.messageStoreConfig.getHaMasterAddress());
 this.updateMasterHAServerAddrPeriodically = false;
 } else {
 this.updateMasterHAServerAddrPeriodically = true;
 }
 // 启动定时任务进行元信息同步
 this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

 @Override
 public void run() {
 try {
 BrokerController.this.slaveSynchronize.syncAll();
 } catch (Throwable e) {
 log.error("ScheduledTask syncAll slave exception", e);
 }
 }
 }, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);
 } else {
 this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

 @Override
 public void run() {
 try {
 BrokerController.this.printMasterAndSlaveDiff();
 } catch (Throwable e) {
 log.error("schedule printMasterAndSlaveDiff error.", e);
 }
 }
 }, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);
 }
 }
 ...
 }
}

syncAll方法实现

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 public void syncAll() {
 // 同步Topic配置信息
 this.syncTopicConfig();
 // 同步消费者偏移量
 this.syncConsumerOffset();
 // 同步延迟偏移量
 this.syncDelayOffset();
 // 同步订阅组配置信息
 this.syncSubscriptionGroupConfig();
 }
 ...
}

同步Topic配置信息源码

syncTopicConfig方法实现如下

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 private void syncTopicConfig() {
 String masterAddrBak = this.masterAddr;
 if (masterAddrBak != null) {
 try {
 TopicConfigSerializeWrapper topicWrapper =
 this.brokerController.getBrokerOuterAPI().getAllTopicConfig(masterAddrBak);
 if (!this.brokerController.getTopicConfigManager().getDataVersion()
 .equals(topicWrapper.getDataVersion())) {

 this.brokerController.getTopicConfigManager().getDataVersion()
 .assignNewOne(topicWrapper.getDataVersion());
 this.brokerController.getTopicConfigManager().getTopicConfigTable().clear();
 this.brokerController.getTopicConfigManager().getTopicConfigTable()
 .putAll(topicWrapper.getTopicConfigTable());
 this.brokerController.getTopicConfigManager().persist();

 log.info("Update slave topic config from master, {}", masterAddrBak);
 }
 } catch (Exception e) {
 log.error("SyncTopicConfig Exception, {}", masterAddrBak, e);
 }
 }
 }
 ...
}

其中getAllTopicConfig方法实现如下

package org.apache.rocketmq.broker.out;

public class BrokerOuterAPI {
 ...
 public TopicConfigSerializeWrapper getAllTopicConfig(
 final String addr) throws RemotingConnectException, RemotingSendRequestException,
 RemotingTimeoutException, InterruptedException, MQBrokerException {
 RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_TOPIC_CONFIG, null);
 // 发起远程同步调用
 RemotingCommand response = this.remotingClient.invokeSync(MixAll.brokerVIPChannel(true, addr), request, 3000);
 assert response != null;
 switch (response.getCode()) {
 case ResponseCode.SUCCESS: {
 return TopicConfigSerializeWrapper.decode(response.getBody(), TopicConfigSerializeWrapper.class);
 }
 default:
 break;
 }

 throw new MQBrokerException(response.getCode(), response.getRemark());
 }
 ...
}

invokeSync通过调用封装过的Netty客户端实现

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingClient extends NettyRemotingAbstract implements RemotingClient {
 ...
 /**
 * 同步请求
 * @param addr 请求地址
 * @param request 请求命令
 * @param timeoutMillis 超时时间
 */
 @Override
 public RemotingCommand invokeSync(String addr, final RemotingCommand request, long timeoutMillis)
 throws InterruptedException, RemotingConnectException, RemotingSendRequestException, RemotingTimeoutException {
 long beginStartTime = System.currentTimeMillis();
 final Channel channel = this.getAndCreateChannel(addr);
 if (channel != null && channel.isActive()) {
 try {
 doBeforeRpcHooks(addr, request);
 // 算出极限耗时代价
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTime) {
 // 如果同步等待时长 < 极限耗时 说明请求不成立，抛出异常
 throw new RemotingTimeoutException("invokeSync call timeout");
 }
 // 同步请求实现
 RemotingCommand response = this.invokeSyncImpl(channel, request, timeoutMillis - costTime);
 doAfterRpcHooks(RemotingHelper.parseChannelRemoteAddr(channel), request, response);
 return response;
 } catch (RemotingSendRequestException e) {
 log.warn("invokeSync: send request exception, so close the channel[{}]", addr);
 this.closeChannel(addr, channel);
 throw e;
 } catch (RemotingTimeoutException e) {
 if (nettyClientConfig.isClientCloseSocketIfTimeout()) {
 this.closeChannel(addr, channel);
 log.warn("invokeSync: close socket because of timeout, {}ms, {}", timeoutMillis, addr);
 }
 log.warn("invokeSync: wait response timeout exception, the channel[{}]", addr);
 throw e;
 }
 } else {
 this.closeChannel(addr, channel);
 throw new RemotingConnectException(addr);
 }
 }
 ...
}

invokeSyncImpl实现逻辑

package org.apache.rocketmq.remoting.netty;

public abstract class NettyRemotingAbstract {
 ...
 public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
 final long timeoutMillis)
 throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
 final int opaque = request.getOpaque();

 try {
 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
 this.responseTable.put(opaque, responseFuture);
 final SocketAddress addr = channel.remoteAddress();
 // 使用netty channel
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 // 轮训监听
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 } else {
 responseFuture.setSendRequestOK(false);
 }
 // 异常情况
 responseTable.remove(opaque);
 responseFuture.setCause(f.cause());
 responseFuture.putResponse(null);
 log.warn("send a request command to channel <" + addr + "> failed.");
 }
 });

 RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
 if (null == responseCommand) {
 if (responseFuture.isSendRequestOK()) {
 throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
 responseFuture.getCause());
 } else {
 throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
 }
 }

 return responseCommand;
 } finally {
 this.responseTable.remove(opaque);
 }
 }
	...
}

同步消费者偏移量源码

其中syncConsumerOffset方法实现如下

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 private void syncConsumerOffset() {
 String masterAddrBak = this.masterAddr;
 if (masterAddrBak != null) {
 try {
 ConsumerOffsetSerializeWrapper offsetWrapper =
 this.brokerController.getBrokerOuterAPI().getAllConsumerOffset(masterAddrBak);
 this.brokerController.getConsumerOffsetManager().getOffsetTable()
 .putAll(offsetWrapper.getOffsetTable());
 this.brokerController.getConsumerOffsetManager().persist();
 log.info("Update slave consumer offset from master, {}", masterAddrBak);
 } catch (Exception e) {
 log.error("SyncConsumerOffset Exception, {}", masterAddrBak, e);
 }
 }
 }
 ...
}

getAllConsumerOffset实现逻辑

package org.apache.rocketmq.broker.out;

public class BrokerOuterAPI {
 ...
 public ConsumerOffsetSerializeWrapper getAllConsumerOffset(
 final String addr) throws InterruptedException, RemotingTimeoutException,
 RemotingSendRequestException, RemotingConnectException, MQBrokerException {
 // 封装CMD指令向主节点发起同步offset信息请求，同步时间上限为3000ms
 RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_CONSUMER_OFFSET, null);
 RemotingCommand response = this.remotingClient.invokeSync(addr, request, 3000);
 assert response != null;
 switch (response.getCode()) {
 case ResponseCode.SUCCESS: {
 return ConsumerOffsetSerializeWrapper.decode(response.getBody(), ConsumerOffsetSerializeWrapper.class);
 }
 default:
 break;
 }

 throw new MQBrokerException(response.getCode(), response.getRemark());
 }
 ...
}

封装的CMD指令

package org.apache.rocketmq.common.protocol;

public class RequestCode {
 ...
 public static final int NOTIFY_CONSUMER_IDS_CHANGED = 40;

 public static final int LOCK_BATCH_MQ = 41;

 public static final int UNLOCK_BATCH_MQ = 42;
 public static final int GET_ALL_CONSUMER_OFFSET = 43;

 public static final int GET_ALL_DELAY_OFFSET = 45;
 ...
}

远程命令对象构建，构建后通过remotingClient发送即可

package org.apache.rocketmq.remoting.protocol;

public class RemotingCommand {
 ...
	public static RemotingCommand createRequestCommand(int code, CommandCustomHeader customHeader) {
 RemotingCommand cmd = new RemotingCommand();
 cmd.setCode(code);
 cmd.customHeader = customHeader;
 setCmdVersion(cmd);
 return cmd;
 }
 ...
}

消息同步源码

消息同步工程源码位于store中

HAConnection实现消息读写

HAservice包含节点连接实现HAClient

逻辑

Master节点：

AcceptSocketService：接收Slave节点连接
HAConnection：
- ReadSocketService：读取来自Slave节点的数据
- WriteSocketService：写入Slave节点的数据

Slave节点：

HAService：
- HAClient：对Master节点连接，读写数据

通信协议：Master节点与Slave节点通信协议较简单，主要如下

对象	用途	第几位	字段	数据类型	字节数	说明
`Slave`=>`Master`	上报`CommitLog`已经同步的物理位置	0	`maxPhyOffset`	Long	8	`CommitLog`最大物理位置
`Master`=>`Slave`	传输新的`CommitLog`数据	0	`fromPhyOffset`	Long	8	传输`CommitLog`开始的物理位置
		1	`size`	Int	4	传输`CommitLog`数据长度
		2	`body`	Bytes		传输`CommitLog`数据

连接Master源码

package org.apache.rocketmq.store.ha;

public class HAService {
 ...
 class HAClient extends ServiceThread {
 /**
 * 客户端连接Master，使用NIO函数，提高效率
 * @return
 * @throws ClosedChannelException
 */
 private boolean connectMaster() throws ClosedChannelException {
 if (null == socketChannel) {
 String addr = this.masterAddress.get();
 if (addr != null) {

 SocketAddress socketAddress = RemotingUtil.string2SocketAddress(addr);
 if (socketAddress != null) {
 this.socketChannel = RemotingUtil.connect(socketAddress);
 if (this.socketChannel != null) {
 this.socketChannel.register(this.selector, SelectionKey.OP_READ);
 }
 }
 }

 this.currentReportedOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();

 this.lastWriteTimestamp = System.currentTimeMillis();
 }

 return this.socketChannel != null;
 }
 
 /**
 * Client(Slave)主循环，实现不断从Master传输CommitLog数据
 * 上传Master自己本地的CommitLog已经同步物理位置
 */
 @Override
 public void run() {
 log.info(this.getServiceName() + " service started");

 while (!this.isStopped()) {
 try {
 if (this.connectMaster()) {

 if (this.isTimeToReportOffset()) {
 // 满足上报间隔，上报到Master进度
 // 固定间隔（默认为5s）向Master上报Slave本地CommitLog已经同步到的物理位置
 boolean result = this.reportSlaveMaxOffset(this.currentReportedOffset);
 if (!result) {
 this.closeMaster();
 }
 }

 this.selector.select(1000);
 // 处理Master传输给Slave的CommitLog数据
 boolean ok = this.processReadEvent();
 if (!ok) {
 this.closeMaster();
 }
 // 若进度有变化，上报到Master进度
 if (!reportSlaveMaxOffsetPlus()) {
 continue;
 }
 // Master超时未返回，关闭连接
 long interval =
 HAService.this.getDefaultMessageStore().getSystemClock().now()
 - this.lastWriteTimestamp;
 if (interval > HAService.this.getDefaultMessageStore().getMessageStoreConfig()
 .getHaHousekeepingInterval()) {
 log.warn("HAClient, housekeeping, found this connection[" + this.masterAddress
 + "] expired, " + interval);
 this.closeMaster();
 log.warn("HAClient, master not response some time, so close connection");
 }
 } else {
 this.waitForRunning(1000 * 5);
 }
 } catch (Exception e) {
 log.warn(this.getServiceName() + " service has exception. ", e);
 this.waitForRunning(1000 * 5);
 }
 }

 log.info(this.getServiceName() + " service end");
 }
 ...
 }
 ...
}

RemotingUtil使用原生Socket实现

package org.apache.rocketmq.remoting.common;

public class RemotingUtil {
 public static SocketAddress string2SocketAddress(final String addr) {
 String[] s = addr.split(":");
 InetSocketAddress isa = new InetSocketAddress(s[0], Integer.parseInt(s[1]));
 return isa;
 }
 
 public static SocketChannel connect(SocketAddress remote, final int timeoutMillis) {
 SocketChannel sc = null;
 try {
 // 打开channel
 sc = SocketChannel.open();
 // 设置同步阻塞，可能存在bug
 sc.configureBlocking(true);
 // 设置关闭Socket的延迟事件，那么当线程执行Socket的close方法时，会进入阻塞状态
 // 直到Socket发送完所有剩余数据，或者超过了方法设置的延迟时间，才从close方法返回
 sc.socket().setSoLinger(false, -1);
 // 禁止使用Nagle算法，使用小数据传输
 // Nagle算法通过将缓冲区的小封包自动相连，组成较大的包
 sc.socket().setTcpNoDelay(true);
 // 设置接收缓冲区
 sc.socket().setReceiveBufferSize(1024 * 64);
 // 设置发送缓冲区
 sc.socket().setSendBufferSize(1024 * 64);
 // 连接
 sc.socket().connect(remote, timeoutMillis);
 // bug？又设置非同步阻塞了
 sc.configureBlocking(false);
 return sc;
 } catch (Exception e) {
 if (sc != null) {
 try {
 sc.close();
 } catch (IOException e1) {
 e1.printStackTrace();
 }
 }
 }

 return null;
 }
 ...
}

核心读请求实现

package org.apache.rocketmq.store.ha;

public class HAService {
 ...
 class HAClient extends ServiceThread {
 private boolean processReadEvent() {
 int readSizeZeroTimes = 0;
 while (this.byteBufferRead.hasRemaining()) {
 try {
 int readSize = this.socketChannel.read(this.byteBufferRead);
 if (readSize > 0) {
 lastWriteTimestamp = HAService.this.defaultMessageStore.getSystemClock().now();
 readSizeZeroTimes = 0;
 boolean result = this.dispatchReadRequest();
 if (!result) {
 log.error("HAClient, dispatchReadRequest error");
 return false;
 }
 } else if (readSize == 0) {
 if (++readSizeZeroTimes >= 3) {
 break;
 }
 } else {
 log.info("HAClient, processReadEvent read socket < 0");
 return false;
 }
 } catch (IOException e) {
 log.info("HAClient, processReadEvent read socket exception", e);
 return false;
 }
 }

 return true;
 }
 
 /**
 * 核心读请求实现
 * 
 * 读取Master传输的CommitLog数据，并返回是否正确
 * 如果读取到数据，写入CommitLog
 * 异常原因：
 * 1. Master传输来的数据开始位置offset(masterPhyOffset)不等于Slave的CommitLog数最大offset(slavePhyOffset)
 * 2. 上报到Master进度失败
 * @return 是否正确
 */
 private boolean dispatchReadRequest() {
 final int msgHeaderSize = 8 + 4; // phyoffset + size
 int readSocketPos = this.byteBufferRead.position();

 while (true) {
 // 读取到请求数据
 int diff = this.byteBufferRead.position() - this.dispatchPostion;
 if (diff >= msgHeaderSize) {
 // 读取masterPhyOffset、bodySize
 long masterPhyOffset = this.byteBufferRead.getLong(this.dispatchPostion);
 int bodySize = this.byteBufferRead.getInt(this.dispatchPostion + 8);
 // 设置Slave节点CommitLog文件最大的offset位置
 long slavePhyOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();
 // 校验Master传输来的数据开始位置offset和slave的commitLog数最大offset是否相同
 if (slavePhyOffset != 0) {
 if (slavePhyOffset != masterPhyOffset) {
 log.error("master pushed offset not equal the max phy offset in slave, SLAVE: "
 + slavePhyOffset + " MASTER: " + masterPhyOffset);
 return false;
 }
 }
 // 读取消息
 if (diff >= (msgHeaderSize + bodySize)) {
 // 写入CommitLog
 byte[] bodyData = new byte[bodySize];
 this.byteBufferRead.position(this.dispatchPostion + msgHeaderSize);
 this.byteBufferRead.get(bodyData);

 HAService.this.defaultMessageStore.appendToCommitLog(masterPhyOffset, bodyData);
 // 设置处理到的位置
 this.byteBufferRead.position(readSocketPos);
 this.dispatchPostion += msgHeaderSize + bodySize;
 // 上报到Master进度
 if (!reportSlaveMaxOffsetPlus()) {
 return false;
 }
 // 继续读数据
 continue;
 }
 }
 // 空间写满，重新分配空间
 if (!this.byteBufferRead.hasRemaining()) {
 this.reallocateByteBuffer();
 }

 break;
 }

 return true;
 }
 ...
 }
 ...
}

同步\异步消息发送

实现分类

同步消息发送
- 发送：producer.send(Message msg)
- 核心实现：DefaultMQProducerImpl
异步消息发送
- 发送：producer.send(Message msg,SendCallback sendCallback)
- 核心实现：DefaultMQProducerImpl

实例

public class Producer {

	public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
		
		DefaultMQProducer producer = new DefaultMQProducer("test_quick_producer_name");
		
	producer.setNamesrvAddr(Const.NAMESRV_ADDR_MASTER_SLAVE);
		
		producer.start();
		
		for(int i = 0 ; i <5; i ++) {
			//	1.	创建消息
			Message message = new Message("test_quick_topic",	//	主题
					"TagA", //	标签
					"key" + i,	// 	用户自定义的key ,唯一的标识
					("Hello RocketMQ" + i).getBytes());	//	消息内容实体（byte[]）
			// 异步发送消息
			producer.send(message, new SendCallback() {
				@Override
				public void onSuccess(SendResult sendResult) {
					System.err.println("msgId: " + sendResult.getMsgId() + ", status: " + sendResult.getSendStatus());
					producer.shutdown();
				}
				@Override
				public void onException(Throwable e) {
					e.printStackTrace();
					System.err.println("------发送失败");
				}
			});
		}	
	}
}

同步消息发送源码分析

发送消息方法

package org.apache.rocketmq.client.producer;

public class DefaultMQProducer extends ClientConfig implements MQProducer {
 ...
 @Override
 public SendResult send(Message msg,
 long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 return this.defaultMQProducerImpl.send(msg, timeout);
 }
}

模式枚举

package org.apache.rocketmq.client.impl;

public enum CommunicationMode {
 SYNC,
 ASYNC,
 // 单向，不需要响应
 ONEWAY,
}

消息发送实现类

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public SendResult send(Message msg,
 long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 return this.sendDefaultImpl(msg, CommunicationMode.SYNC, null, timeout);
 }
 
 private SendResult sendDefaultImpl(
 Message msg,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final long timeout
 ) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 // 校验运行状态
 this.makeSureStateOK();
 // 检查消息格式
 Validators.checkMessage(msg, this.defaultMQProducer);
 // 回调ID，用户标记是否同一次消息
 final long invokeID = random.nextLong();
 // 开始发送消息的时间
 long beginTimestampFirst = System.currentTimeMillis();
 long beginTimestampPrev = beginTimestampFirst;
 long endTimestamp = beginTimestampFirst;
 // 查找topic
 TopicPublishInfo topicPublishInfo = this.tryToFindTopicPublishInfo(msg.getTopic());
 if (topicPublishInfo != null && topicPublishInfo.ok()) {
 boolean callTimeout = false;
 // 要发送的消息队列
 MessageQueue mq = null;
 Exception exception = null;
 SendResult sendResult = null;
 // 重试次数，从配置参数中读取
 int timesTotal = communicationMode == CommunicationMode.SYNC ? 1 + this.defaultMQProducer.getRetryTimesWhenSendFailed() : 1;
 int times = 0;
 // 存储每次发送消息选择的broker名称
 String[] brokersSent = new String[timesTotal];
 for (; times < timesTotal; times++) {
 // 最后一次消息发送的brokerName进行存储
 String lastBrokerName = null == mq ? null : mq.getBrokerName();
 // 消息队列选择策略
 MessageQueue mqSelected = this.selectOneMessageQueue(topicPublishInfo, lastBrokerName);
 if (mqSelected != null) {
 mq = mqSelected;
 brokersSent[times] = mq.getBrokerName();
 try {
 beginTimestampPrev = System.currentTimeMillis();
 // 计算代价
 long costTime = beginTimestampPrev - beginTimestampFirst;
 if (timeout < costTime) {
 callTimeout = true;
 break;
 }
 // 消息发送实现
 sendResult = this.sendKernelImpl(msg, mq, communicationMode, sendCallback, topicPublishInfo, timeout - costTime);
 endTimestamp = System.currentTimeMillis();
 // 更新broker可用信息（容错）
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
 switch (communicationMode) {
 case ASYNC:
 return null;
 case ONEWAY:
 return null;
 case SYNC:
 // 同步发送等待返回结果
 if (sendResult.getSendStatus() != SendStatus.SEND_OK) {
 if (this.defaultMQProducer.isRetryAnotherBrokerWhenNotStoreOK()) {
 continue;
 }
 }

 return sendResult;
 default:
 break;
 }
 } catch (RemotingException e) {
 endTimestamp = System.currentTimeMillis();
 // 抛异常则更新broker容错信息
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 continue;
 } catch (MQClientException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 continue;
 } catch (MQBrokerException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 // 抛出如下异常则进行重试
 switch (e.getResponseCode()) {
 case ResponseCode.TOPIC_NOT_EXIST:
 case ResponseCode.SERVICE_NOT_AVAILABLE:
 case ResponseCode.SYSTEM_ERROR:
 case ResponseCode.NO_PERMISSION:
 case ResponseCode.NO_BUYER_ID:
 case ResponseCode.NOT_IN_CURRENT_UNIT:
 continue;
 default:
 if (sendResult != null) {
 return sendResult;
 }

 throw e;
 }
 } catch (InterruptedException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
 log.warn(String.format("sendKernelImpl exception, throw exception, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());

 log.warn("sendKernelImpl exception", e);
 log.warn(msg.toString());
 throw e;
 }
 } else {
 break;
 }
 }

 if (sendResult != null) {
 return sendResult;
 }

 String info = String.format("Send [%d] times, still failed, cost [%d]ms, Topic: %s, BrokersSent: %s",
 times,
 System.currentTimeMillis() - beginTimestampFirst,
 msg.getTopic(),
 Arrays.toString(brokersSent));

 info += FAQUrl.suggestTodo(FAQUrl.SEND_MSG_FAILED);

 MQClientException mqClientException = new MQClientException(info, exception);
 if (callTimeout) {
 throw new RemotingTooMuchRequestException("sendDefaultImpl call timeout");
 }

 if (exception instanceof MQBrokerException) {
 mqClientException.setResponseCode(((MQBrokerException) exception).getResponseCode());
 } else if (exception instanceof RemotingConnectException) {
 mqClientException.setResponseCode(ClientErrorCode.CONNECT_BROKER_EXCEPTION);
 } else if (exception instanceof RemotingTimeoutException) {
 mqClientException.setResponseCode(ClientErrorCode.ACCESS_BROKER_TIMEOUT);
 } else if (exception instanceof MQClientException) {
 mqClientException.setResponseCode(ClientErrorCode.BROKER_NOT_EXIST_EXCEPTION);
 }

 throw mqClientException;
 }

 List<String> nsList = this.getmQClientFactory().getMQClientAPIImpl().getNameServerAddressList();
 if (null == nsList || nsList.isEmpty()) {
 throw new MQClientException(
 "No name server address, please set it." + FAQUrl.suggestTodo(FAQUrl.NAME_SERVER_ADDR_NOT_EXIST_URL), null).setResponseCode(ClientErrorCode.NO_NAME_SERVER_EXCEPTION);
 }

 throw new MQClientException("No route info of this topic, " + msg.getTopic() + FAQUrl.suggestTodo(FAQUrl.NO_TOPIC_ROUTE_INFO),
 null).setResponseCode(ClientErrorCode.NOT_FOUND_TOPIC_EXCEPTION);
 }
}

查找topic方法实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 private TopicPublishInfo tryToFindTopicPublishInfo(final String topic) {
 // 使用Map缓存
 TopicPublishInfo topicPublishInfo = this.topicPublishInfoTable.get(topic);
 if (null == topicPublishInfo || !topicPublishInfo.ok()) {
 // 当无可用topic发布信息时，主动向NameServer拉取一次
 this.topicPublishInfoTable.putIfAbsent(topic, new TopicPublishInfo());
 this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
 topicPublishInfo = this.topicPublishInfoTable.get(topic);
 }
 // 若可用
 if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) {
 return topicPublishInfo;
 } else {
 // 否则根据设置判断是否新建Topic
 this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, true, this.defaultMQProducer);
 topicPublishInfo = this.topicPublishInfoTable.get(topic);
 return topicPublishInfo;
 }
 }
}

客户端获取路由信息实现

package org.apache.rocketmq.client.impl.factory;

public class MQClientInstance {
 public boolean updateTopicRouteInfoFromNameServer(final String topic) {
 return updateTopicRouteInfoFromNameServer(topic, false, null);
 }
 
 public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault,
 DefaultMQProducer defaultMQProducer) {
 try {
 if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
 try {
 TopicRouteData topicRouteData;
 if (isDefault && defaultMQProducer != null) {
 topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(),
 1000 * 3);
 if (topicRouteData != null) {
 for (QueueData data : topicRouteData.getQueueDatas()) {
 int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
 data.setReadQueueNums(queueNums);
 data.setWriteQueueNums(queueNums);
 }
 }
 } else {
 topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, 1000 * 3);
 }
 if (topicRouteData != null) {
 TopicRouteData old = this.topicRouteTable.get(topic);
 boolean changed = topicRouteDataIsChange(old, topicRouteData);
 if (!changed) {
 changed = this.isNeedUpdateTopicRouteInfo(topic);
 } else {
 log.info("the topic[{}] route info changed, old[{}] ,new[{}]", topic, old, topicRouteData);
 }

 if (changed) {
 TopicRouteData cloneTopicRouteData = topicRouteData.cloneTopicRouteData();

 for (BrokerData bd : topicRouteData.getBrokerDatas()) {
 this.brokerAddrTable.put(bd.getBrokerName(), bd.getBrokerAddrs());
 }

 // Update Pub info
 {
 TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData);
 publishInfo.setHaveTopicRouterInfo(true);
 Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<String, MQProducerInner> entry = it.next();
 MQProducerInner impl = entry.getValue();
 if (impl != null) {
 impl.updateTopicPublishInfo(topic, publishInfo);
 }
 }
 }

 // Update sub info
 {
 Set<MessageQueue> subscribeInfo = topicRouteData2TopicSubscribeInfo(topic, topicRouteData);
 Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<String, MQConsumerInner> entry = it.next();
 MQConsumerInner impl = entry.getValue();
 if (impl != null) {
 impl.updateTopicSubscribeInfo(topic, subscribeInfo);
 }
 }
 }
 log.info("topicRouteTable.put. Topic = {}, TopicRouteData[{}]", topic, cloneTopicRouteData);
 this.topicRouteTable.put(topic, cloneTopicRouteData);
 return true;
 }
 } else {
 log.warn("updateTopicRouteInfoFromNameServer, getTopicRouteInfoFromNameServer return null, Topic: {}", topic);
 }
 } catch (Exception e) {
 if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX) && !topic.equals(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC)) {
 log.warn("updateTopicRouteInfoFromNameServer Exception", e);
 }
 } finally {
 this.lockNamesrv.unlock();
 }
 } else {
 log.warn("updateTopicRouteInfoFromNameServer tryLock timeout {}ms", LOCK_TIMEOUT_MILLIS);
 }
 } catch (InterruptedException e) {
 log.warn("updateTopicRouteInfoFromNameServer Exception", e);
 }

 return false;
 }
}

消息队列选择策略实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
 return this.mqFaultStrategy.selectOneMessageQueue(tpInfo, lastBrokerName);
 }
}

负载均衡消息选择策略实现

package org.apache.rocketmq.client.latency;

public class MQFaultStrategy {
 private final static InternalLogger log = ClientLogger.getLog();
 private final LatencyFaultTolerance<String> latencyFaultTolerance = new LatencyFaultToleranceImpl();
 // 发送消息延迟容错开关
 private boolean sendLatencyFaultEnable = false;
 // 消息发送时长的延迟级别数组
 private long[] latencyMax = {50L, 100L, 550L, 1000L, 2000L, 3000L, 15000L};
 // broker预估不可用时长级别数组
 private long[] notAvailableDuration = {0L, 0L, 30000L, 60000L, 120000L, 180000L, 600000L};
 /**
 * 负载均衡消息选择策略
 */
 public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
 if (this.sendLatencyFaultEnable) {
 try {
 // 获取可用队列
 int index = tpInfo.getSendWhichQueue().getAndIncrement();
 for (int i = 0; i < tpInfo.getMessageQueueList().size(); i++) {
 int pos = Math.abs(index++) % tpInfo.getMessageQueueList().size();
 if (pos < 0)
 pos = 0;
 MessageQueue mq = tpInfo.getMessageQueueList().get(pos);
 if (latencyFaultTolerance.isAvailable(mq.getBrokerName())) {
 if (null == lastBrokerName || mq.getBrokerName().equals(lastBrokerName))
 return mq;
 }
 }
 // 选择最优队列
 final String notBestBroker = latencyFaultTolerance.pickOneAtLeast();
 int writeQueueNums = tpInfo.getQueueIdByBroker(notBestBroker);
 if (writeQueueNums > 0) {
 final MessageQueue mq = tpInfo.selectOneMessageQueue();
 if (notBestBroker != null) {
 mq.setBrokerName(notBestBroker);
 mq.setQueueId(tpInfo.getSendWhichQueue().getAndIncrement() % writeQueueNums);
 }
 return mq;
 } else {
 latencyFaultTolerance.remove(notBestBroker);
 }
 } catch (Exception e) {
 log.error("Error occurred when selecting message queue", e);
 }

 return tpInfo.selectOneMessageQueue();
 }

 return tpInfo.selectOneMessageQueue(lastBrokerName);
 }
}

更新broker可用信息（容错）实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) {
 this.mqFaultStrategy.updateFaultItem(brokerName, currentLatency, isolation);
 }
}

与负载均衡消息选择策略位于相同类中

package org.apache.rocketmq.client.latency;

public class MQFaultStrategy {
 	 public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) {
 if (this.sendLatencyFaultEnable) {
 long duration = computeNotAvailableDuration(isolation ? 30000 : currentLatency);
 this.latencyFaultTolerance.updateFaultItem(brokerName, currentLatency, duration);
 }
 } 
}

消息发送实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 private SendResult sendKernelImpl(final Message msg,
 final MessageQueue mq,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final TopicPublishInfo topicPublishInfo,
 final long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 long beginStartTime = System.currentTimeMillis();
 // 获得地址
 String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
 if (null == brokerAddr) {
 tryToFindTopicPublishInfo(mq.getTopic());
 brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
 }

 SendMessageContext context = null;
 if (brokerAddr != null) {
 brokerAddr = MixAll.brokerVIPChannel(this.defaultMQProducer.isSendMessageWithVIPChannel(), brokerAddr);
 // 记录消息内容，因为下面逻辑可能改变消息内容，例如消息压缩
 byte[] prevBody = msg.getBody();
 try {
 //for MessageBatch,ID has been set in the generating process
 if (!(msg instanceof MessageBatch)) {
 MessageClientIDSetter.setUniqID(msg);
 }

 int sysFlag = 0;
 boolean msgBodyCompressed = false;
 if (this.tryToCompressMessage(msg)) {
 sysFlag |= MessageSysFlag.COMPRESSED_FLAG;
 msgBodyCompressed = true;
 }

 final String tranMsg = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
 if (tranMsg != null && Boolean.parseBoolean(tranMsg)) {
 sysFlag |= MessageSysFlag.TRANSACTION_PREPARED_TYPE;
 }
 // 消息发送前校验
 if (hasCheckForbiddenHook()) {
 CheckForbiddenContext checkForbiddenContext = new CheckForbiddenContext();
 checkForbiddenContext.setNameSrvAddr(this.defaultMQProducer.getNamesrvAddr());
 checkForbiddenContext.setGroup(this.defaultMQProducer.getProducerGroup());
 checkForbiddenContext.setCommunicationMode(communicationMode);
 checkForbiddenContext.setBrokerAddr(brokerAddr);
 checkForbiddenContext.setMessage(msg);
 checkForbiddenContext.setMq(mq);
 checkForbiddenContext.setUnitMode(this.isUnitMode());
 this.executeCheckForbiddenHook(checkForbiddenContext);
 }
 // 消息发送前回调处理
 if (this.hasSendMessageHook()) {
 context = new SendMessageContext();
 context.setProducer(this);
 context.setProducerGroup(this.defaultMQProducer.getProducerGroup());
 context.setCommunicationMode(communicationMode);
 context.setBornHost(this.defaultMQProducer.getClientIP());
 context.setBrokerAddr(brokerAddr);
 context.setMessage(msg);
 context.setMq(mq);
 String isTrans = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
 if (isTrans != null && isTrans.equals("true")) {
 context.setMsgType(MessageType.Trans_Msg_Half);
 }

 if (msg.getProperty("__STARTDELIVERTIME") != null || msg.getProperty(MessageConst.PROPERTY_DELAY_TIME_LEVEL) != null) {
 context.setMsgType(MessageType.Delay_Msg);
 }
 this.executeSendMessageHookBefore(context);
 }
 // 构建消息发送请求头
 SendMessageRequestHeader requestHeader = new SendMessageRequestHeader();
 requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup());
 requestHeader.setTopic(msg.getTopic());
 requestHeader.setDefaultTopic(this.defaultMQProducer.getCreateTopicKey());
 requestHeader.setDefaultTopicQueueNums(this.defaultMQProducer.getDefaultTopicQueueNums());
 requestHeader.setQueueId(mq.getQueueId());
 requestHeader.setSysFlag(sysFlag);
 requestHeader.setBornTimestamp(System.currentTimeMillis());
 requestHeader.setFlag(msg.getFlag());
 requestHeader.setProperties(MessageDecoder.messageProperties2String(msg.getProperties()));
 requestHeader.setReconsumeTimes(0);
 requestHeader.setUnitMode(this.isUnitMode());
 requestHeader.setBatch(msg instanceof MessageBatch);
 if (requestHeader.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
 String reconsumeTimes = MessageAccessor.getReconsumeTime(msg);
 if (reconsumeTimes != null) {
 requestHeader.setReconsumeTimes(Integer.valueOf(reconsumeTimes));
 MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_RECONSUME_TIME);
 }

 String maxReconsumeTimes = MessageAccessor.getMaxReconsumeTimes(msg);
 if (maxReconsumeTimes != null) {
 requestHeader.setMaxReconsumeTimes(Integer.valueOf(maxReconsumeTimes));
 MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_MAX_RECONSUME_TIMES);
 }
 }

 SendResult sendResult = null;
 // 消息发送逻辑
 switch (communicationMode) {
 // 异步发送
 case ASYNC:
 Message tmpMessage = msg;
 if (msgBodyCompressed) {
 //If msg body was compressed, msgbody should be reset using prevBody.
 //Clone new message using commpressed message body and recover origin massage.
 //Fix bug:https://github.com/apache/rocketmq-externals/issues/66
 tmpMessage = MessageAccessor.cloneMessage(msg);
 msg.setBody(prevBody);
 }
 long costTimeAsync = System.currentTimeMillis() - beginStartTime;
 if (timeout < costTimeAsync) {
 throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
 }
 sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
 brokerAddr,
 mq.getBrokerName(),
 tmpMessage,
 requestHeader,
 timeout - costTimeAsync,
 communicationMode,
 sendCallback,
 topicPublishInfo,
 this.mQClientFactory,
 this.defaultMQProducer.getRetryTimesWhenSendAsyncFailed(),
 context,
 this);
 break;
 case ONEWAY:
 // 同步发送
 case SYNC:
 long costTimeSync = System.currentTimeMillis() - beginStartTime;
 if (timeout < costTimeSync) {
 throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
 }
 sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
 brokerAddr,
 mq.getBrokerName(),
 msg,
 requestHeader,
 timeout - costTimeSync,
 communicationMode,
 context,
 this);
 break;
 default:
 assert false;
 break;
 }

 if (this.hasSendMessageHook()) {
 // 发送结果放入上下文后返回
 context.setSendResult(sendResult);
 this.executeSendMessageHookAfter(context);
 }

 return sendResult;
 } catch (RemotingException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } catch (MQBrokerException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } catch (InterruptedException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } finally {
 msg.setBody(prevBody);
 }
 }

 throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
 }
}

使用MQClientAPIImpl真正发起Netty网络请求

package org.apache.rocketmq.client.impl;

public class MQClientAPIImpl {
 ...
 public SendResult sendMessage(
 final String addr,
 final String brokerName,
 final Message msg,
 final SendMessageRequestHeader requestHeader,
 final long timeoutMillis,
 final CommunicationMode communicationMode,
 final SendMessageContext context,
 final DefaultMQProducerImpl producer
 ) throws RemotingException, MQBrokerException, InterruptedException {
 return sendMessage(addr, brokerName, msg, requestHeader, timeoutMillis, communicationMode, null, null, null, 0, context, producer);
 }

 /**
 * 封装 RemotingCommand 对象后真正发起网络请求
 */
 public SendResult sendMessage(
 final String addr,
 final String brokerName,
 final Message msg,
 final SendMessageRequestHeader requestHeader,
 final long timeoutMillis,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final TopicPublishInfo topicPublishInfo,
 final MQClientInstance instance,
 final int retryTimesWhenSendFailed,
 final SendMessageContext context,
 final DefaultMQProducerImpl producer
 ) throws RemotingException, MQBrokerException, InterruptedException {
 long beginStartTime = System.currentTimeMillis();
 RemotingCommand request = null;
 if (sendSmartMsg || msg instanceof MessageBatch) {
 SendMessageRequestHeaderV2 requestHeaderV2 = SendMessageRequestHeaderV2.createSendMessageRequestHeaderV2(requestHeader);
 request = RemotingCommand.createRequestCommand(msg instanceof MessageBatch ? RequestCode.SEND_BATCH_MESSAGE : RequestCode.SEND_MESSAGE_V2, requestHeaderV2);
 } else {
 request = RemotingCommand.createRequestCommand(RequestCode.SEND_MESSAGE, requestHeader);
 }
 // 设置请求体
 request.setBody(msg.getBody());

 switch (communicationMode) {
 case ONEWAY:
 this.remotingClient.invokeOneway(addr, request, timeoutMillis);
 return null;
 case ASYNC:
 final AtomicInteger times = new AtomicInteger();
 long costTimeAsync = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTimeAsync) {
 throw new RemotingTooMuchRequestException("sendMessage call timeout");
 }
 this.sendMessageAsync(addr, brokerName, msg, timeoutMillis - costTimeAsync, request, sendCallback, topicPublishInfo, instance,
 retryTimesWhenSendFailed, times, context, producer);
 return null;
 case SYNC:
 long costTimeSync = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTimeSync) {
 throw new RemotingTooMuchRequestException("sendMessage call timeout");
 }
 return this.sendMessageSync(addr, brokerName, msg, timeoutMillis - costTimeSync, request);
 default:
 assert false;
 break;
 }

 return null;
 }

 private SendResult sendMessageSync(
 final String addr,
 final String brokerName,
 final Message msg,
 final long timeoutMillis,
 final RemotingCommand request
 ) throws RemotingException, MQBrokerException, InterruptedException {
 // 同步发网络请求，返回响应结果
 RemotingCommand response = this.remotingClient.invokeSync(addr, request, timeoutMillis);
 assert response != null;
 return this.processSendResponse(brokerName, msg, response);
 }
}

底层Netty通信

类图

`RemotingClient`源码

package org.apache.rocketmq.remoting;

public interface RemotingClient extends RemotingService {

 /**
 * 更新协调服务地址列表
 * @param addrs
 */
 void updateNameServerAddressList(final List<String> addrs);

 /**
 * 更新协调服务地址列表
 * @return
 */
 List<String> getNameServerAddressList();

 /**
 * 同步消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 RemotingCommand invokeSync(final String addr, final RemotingCommand request,
 final long timeoutMillis) throws InterruptedException, RemotingConnectException,
 RemotingSendRequestException, RemotingTimeoutException;

 /**
 * 异步消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeAsync(final String addr, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback) throws InterruptedException, RemotingConnectException,
 RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException;

 /**
 * 单次消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeOneway(final String addr, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingConnectException, RemotingTooMuchRequestException,
 RemotingTimeoutException, RemotingSendRequestException;

 /**
 *用于注册一些底层通信服务，比如进行元数据信息的同步工作，commitlog的同步等
 * @param requestCode 底层业务通信规则码
 * @param processor 注册器（单线程）
 * @param executor 线程池，用于执行注册器的业务逻辑
 */
 void registerProcessor(final int requestCode, final NettyRequestProcessor processor,
 final ExecutorService executor);

 /**
 * 回调函数执行线程设置，用于生产者发送消息后的回调函数
 * @param callbackExecutor
 */
 void setCallbackExecutor(final ExecutorService callbackExecutor);

 /**
 * 获取回调池：NameServer地址
 * @return
 */
 ExecutorService getCallbackExecutor();
 
 /**
 * 用于远程服务地址判断是否可以进行通信
 * @param addr
 * @return
 */
 boolean isChannelWritable(final String addr);
}

registerProcessor方法在实现类中用于初始化时进行注册逻辑

package org.apache.rocketmq.client.impl;

public class MQClientAPIImpl {
 /**
 * Netty 服务启动时初始化工作
 * @param nettyClientConfig 配置信息
 * @param clientRemotingProcessor
 * @param rpcHook
 * @param clientConfig
 */
 public MQClientAPIImpl(final NettyClientConfig nettyClientConfig,
 final ClientRemotingProcessor clientRemotingProcessor,
 RPCHook rpcHook, final ClientConfig clientConfig) {
 this.clientConfig = clientConfig;
 topAddressing = new TopAddressing(MixAll.getWSAddr(), clientConfig.getUnitName());
 this.remotingClient = new NettyRemotingClient(nettyClientConfig, null);
 this.clientRemotingProcessor = clientRemotingProcessor;

 this.remotingClient.registerRPCHook(rpcHook);
 this.remotingClient.registerProcessor(RequestCode.CHECK_TRANSACTION_STATE, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.NOTIFY_CONSUMER_IDS_CHANGED, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.RESET_CONSUMER_CLIENT_OFFSET, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.GET_CONSUMER_STATUS_FROM_CLIENT, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.GET_CONSUMER_RUNNING_INFO, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.CONSUME_MESSAGE_DIRECTLY, this.clientRemotingProcessor, null);
 }
 ...
}

setCallbackExecutor方法用于回调函数执行线程设置，用于生产者发送消息后的回调线程池，例如DefaultMQProducerImpl中的使用方式

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public void setCallbackExecutor(final ExecutorService callbackExecutor) {
	 this.mQClientFactory.getMQClientAPIImpl().getRemotingClient().setCallbackExecutor(callbackExecutor);
 }
 
 /**
 * It will be removed at 4.4.0 cause for exception handling and the wrong Semantics of timeout.
 * A new one will be provided in next version
 * @param msg
 * @param sendCallback
 * @param timeout the <code>sendCallback</code> will be invoked at most time
 * @throws RejectedExecutionException
 */
 @Deprecated
 public void send(final Message msg, final SendCallback sendCallback, final long timeout)
 throws MQClientException, RemotingException, InterruptedException {
 final long beginStartTime = System.currentTimeMillis();
 // 获取回调线程池
 ExecutorService executor = this.getCallbackExecutor();
 try {
 executor.submit(new Runnable() {
 @Override
 public void run() {
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeout > costTime) {
 try {
 sendDefaultImpl(msg, CommunicationMode.ASYNC, sendCallback, timeout - costTime);
 } catch (Exception e) {
 sendCallback.onException(e);
 }
 } else {
 sendCallback.onException(
 new RemotingTooMuchRequestException("DEFAULT ASYNC send call timeout"));
 }
 }

 });
 } catch (RejectedExecutionException e) {
 throw new MQClientException("executor rejected ", e);
 }

 }
	...
}

`RemotingServer`源码

package org.apache.rocketmq.remoting;

public interface RemotingServer extends RemotingService {

 /**
 * 用于注册一些底层通信服务，比如进行元数据信息的同步工作，commitlog的同步等
 * @param requestCode 底层业务通信规则码
 * @param processor 注册器（单线程）
 * @param executor 线程池，用于执行注册器的业务逻辑
 */
 void registerProcessor(final int requestCode, final NettyRequestProcessor processor,
 final ExecutorService executor);

 /**
 * 默认注册方法：比如注册：{@link org.apache.rocketmq.broker.processor.AdminBrokerProcessor}管理Broker工作线程或者namesrv有关的流程器，
 * 比如同步地址、路由topic等信息
 * @param processor
 * @param executor
 */
 void registerDefaultProcessor(final NettyRequestProcessor processor, final ExecutorService executor);

 /**
 * 获取端口
 * @return
 */
 int localListenPort();

 /**
 * 根据业务规则获取Pair<执行器，线程池>
 * @param requestCode 业务规则码
 * @return
 */
 Pair<NettyRequestProcessor, ExecutorService> getProcessorPair(final int requestCode);

 /**
 * 同步消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 RemotingCommand invokeSync(final Channel channel, final RemotingCommand request,
 final long timeoutMillis) throws InterruptedException, RemotingSendRequestException,
 RemotingTimeoutException;

 /**
 * 异步消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeAsync(final Channel channel, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback) throws InterruptedException,
 RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException;

 /**
 * 单向消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeOneway(final Channel channel, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException,
 RemotingSendRequestException;

}

`NettyRemotingServer`源码

其唯一实现类NettyRemotingServer，主要实现Netty编程模型

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingServer extends NettyRemotingAbstract implements RemotingServer {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);
 /**
 * NettyServer辅助类
 */
 private final ServerBootstrap serverBootstrap;
 /**
 * selector组（用于注册并监听event）
 */
 private final EventLoopGroup eventLoopGroupSelector;
 /**
 * 主线程组（用于接收TCP连接）
 */
 private final EventLoopGroup eventLoopGroupBoss;
 /**
 * NettyServer配置
 */
 private final NettyServerConfig nettyServerConfig;

 /**
 * 主要使用在registerProcessor方法中
 */
 private final ExecutorService publicExecutor;

 /**
 * channel事件监听
 */
 private final ChannelEventListener channelEventListener;

 public NettyRemotingServer(final NettyServerConfig nettyServerConfig,
 final ChannelEventListener channelEventListener) {
 super(nettyServerConfig.getServerOnewaySemaphoreValue(), nettyServerConfig.getServerAsyncSemaphoreValue());
 this.serverBootstrap = new ServerBootstrap();
 this.nettyServerConfig = nettyServerConfig;
 this.channelEventListener = channelEventListener;

 int publicThreadNums = nettyServerConfig.getServerCallbackExecutorThreads();
 if (publicThreadNums <= 0) {
 publicThreadNums = 4;
 }

 this.publicExecutor = Executors.newFixedThreadPool(publicThreadNums, new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyServerPublicExecutor_" + this.threadIndex.incrementAndGet());
 }
 });

 this.eventLoopGroupBoss = new NioEventLoopGroup(1, new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyBoss_%d", this.threadIndex.incrementAndGet()));
 }
 });

 // 如果是Linux平台，并且开启了native epoll，就用EpollEventLoopGroup，这个也就是用JNI，调用c写的epoll；否则就用java NIO的NioEventLoopGroup
 if (useEpoll()) {
 this.eventLoopGroupSelector = new EpollEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);
 private int threadTotal = nettyServerConfig.getServerSelectorThreads();

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyServerEPOLLSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
 }
 });
 } else {
 this.eventLoopGroupSelector = new NioEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);
 private int threadTotal = nettyServerConfig.getServerSelectorThreads();

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyServerNIOSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
 }
 });
 }

 loadSslContext();
 }
 
 @Override
 public void start() {
 // 根据配置创建工作线程组
 this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
 nettyServerConfig.getServerWorkerThreads(),
 new ThreadFactory() {

 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyServerCodecThread_" + this.threadIndex.incrementAndGet());
 }
 });
 // Netty编程模型
 ServerBootstrap childHandler =
 this.serverBootstrap.group(this.eventLoopGroupBoss, this.eventLoopGroupSelector)
 .channel(useEpoll() ? EpollServerSocketChannel.class : NioServerSocketChannel.class)
 // sync + accept = backlog
 .option(ChannelOption.SO_BACKLOG, 1024)
 // 允许重复使用本地地址和端口
 .option(ChannelOption.SO_REUSEADDR, true)
 .option(ChannelOption.SO_KEEPALIVE, false)
 .childOption(ChannelOption.TCP_NODELAY, true)
 .childOption(ChannelOption.SO_SNDBUF, nettyServerConfig.getServerSocketSndBufSize())
 .childOption(ChannelOption.SO_RCVBUF, nettyServerConfig.getServerSocketRcvBufSize())
 .localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort()))
 .childHandler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) throws Exception {
 ch.pipeline()
 .addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME,
 new HandshakeHandler(TlsSystemConfig.tlsMode))
 .addLast(defaultEventExecutorGroup,
 new NettyEncoder(),
 new NettyDecoder(),
 new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()),
 new NettyConnectManageHandler(),
 new NettyServerHandler()
 );
 }
 });

 if (nettyServerConfig.isServerPooledByteBufAllocatorEnable()) {
 childHandler.childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
 }

 try {
 ChannelFuture sync = this.serverBootstrap.bind().sync();
 InetSocketAddress addr = (InetSocketAddress) sync.channel().localAddress();
 this.port = addr.getPort();
 } catch (InterruptedException e1) {
 throw new RuntimeException("this.serverBootstrap.bind().sync() InterruptedException", e1);
 }

 if (this.channelEventListener != null) {
 this.nettyEventExecutor.start();
 }

 this.timer.scheduleAtFixedRate(new TimerTask() {

 @Override
 public void run() {
 try {
 NettyRemotingServer.this.scanResponseTable();
 } catch (Throwable e) {
 log.error("scanResponseTable exception", e);
 }
 }
 }, 1000 * 3, 1000);
 }
 ...
}

`NettyRemotingAbstract`源码

NettyRemotingAbstract具体实现三种发送方法

package org.apache.rocketmq.remoting.netty;

public abstract class NettyRemotingAbstract {
 /**
 * 异步调用具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
 long beginStartTime = System.currentTimeMillis();
 final int opaque = request.getOpaque();
 boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
 if (acquired) {
 final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync);
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTime) {
 throw new RemotingTooMuchRequestException("invokeAsyncImpl call timeout");
 }

 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis - costTime, invokeCallback, once);
 this.responseTable.put(opaque, responseFuture);
 try {
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 }
 requestFail(opaque);
 log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel));
 }
 });
 } catch (Exception e) {
 responseFuture.release();
 log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e);
 throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
 }
 } else {
 if (timeoutMillis <= 0) {
 throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast");
 } else {
 String info =
 String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
 timeoutMillis,
 this.semaphoreAsync.getQueueLength(),
 this.semaphoreAsync.availablePermits()
 );
 log.warn(info);
 throw new RemotingTimeoutException(info);
 }
 }
 }
 /**
 * 单向发送具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 public void invokeOnewayImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
 request.markOnewayRPC();
 boolean acquired = this.semaphoreOneway.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
 if (acquired) {
 final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreOneway);
 try {
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 once.release();
 if (!f.isSuccess()) {
 log.warn("send a request command to channel <" + channel.remoteAddress() + "> failed.");
 }
 }
 });
 } catch (Exception e) {
 once.release();
 log.warn("write send a request command to channel <" + channel.remoteAddress() + "> failed.");
 throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
 }
 } else {
 if (timeoutMillis <= 0) {
 throw new RemotingTooMuchRequestException("invokeOnewayImpl invoke too fast");
 } else {
 String info = String.format(
 "invokeOnewayImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
 timeoutMillis,
 this.semaphoreOneway.getQueueLength(),
 this.semaphoreOneway.availablePermits()
 );
 log.warn(info);
 throw new RemotingTimeoutException(info);
 }
 }
 }
 /**
 * 同步调用具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
 final long timeoutMillis)
 throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
 final int opaque = request.getOpaque();

 try {
 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
 this.responseTable.put(opaque, responseFuture);
 final SocketAddress addr = channel.remoteAddress();
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 } else {
 responseFuture.setSendRequestOK(false);
 }

 responseTable.remove(opaque);
 responseFuture.setCause(f.cause());
 responseFuture.putResponse(null);
 log.warn("send a request command to channel <" + addr + "> failed.");
 }
 });

 RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
 if (null == responseCommand) {
 if (responseFuture.isSendRequestOK()) {
 throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
 responseFuture.getCause());
 } else {
 throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
 }
 }

 return responseCommand;
 } finally {
 this.responseTable.remove(opaque);
 }
 }
 ...
}

Netty编码解码源码解析——`RemotingCommand`源码

netty消息数据包结构

RemotingCommand中编码、解码方法体现了其数据包格式和用法

package org.apache.rocketmq.remoting.protocol;

public class RemotingCommand {
 
 public ByteBuffer encode() {
 // 1> header length size
 int length = 4;

 // 2> header data length
 byte[] headerData = this.headerEncode();
 length += headerData.length;

 // 3> body data length
 if (this.body != null) {
 length += body.length;
 }

 ByteBuffer result = ByteBuffer.allocate(4 + length);

 // length
 result.putInt(length);

 // header length
 result.put(markProtocolType(headerData.length, serializeTypeCurrentRPC));

 // header data
 result.put(headerData);

 // body data;
 if (this.body != null) {
 result.put(this.body);
 }

 result.flip();

 return result;
 }
 public static RemotingCommand decode(final ByteBuffer byteBuffer) {
 int length = byteBuffer.limit();
 int oriHeaderLen = byteBuffer.getInt();
 int headerLength = getHeaderLength(oriHeaderLen);

 byte[] headerData = new byte[headerLength];
 // 此处使用的是java nio buffer
 byteBuffer.get(headerData);
 // header封装，长度+类型 对应数据包结构第二位
 RemotingCommand cmd = headerDecode(headerData, getProtocolType(oriHeaderLen));

 int bodyLength = length - 4 - headerLength;
 byte[] bodyData = null;
 if (bodyLength > 0) {
 bodyData = new byte[bodyLength];
 byteBuffer.get(bodyData);
 }
 cmd.body = bodyData;

 return cmd;
 }
 ...
}

其支持的序列化类型有两个

SerializeType序列化类型枚举

package org.apache.rocketmq.remoting.protocol;

public enum SerializeType {
 JSON((byte) 0),
 ROCKETMQ((byte) 1);
 ...
}

在NettyRemotingClient中的Netty编程模型也有编码、解码配置，其使用的是netty buffer

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingClient extends NettyRemotingAbstract implements RemotingClient {
 @Override
 public void start() {
 this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
 nettyClientConfig.getClientWorkerThreads(),
 new ThreadFactory() {

 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyClientWorkerThread_" + this.threadIndex.incrementAndGet());
 }
 });

 Bootstrap handler = this.bootstrap.group(this.eventLoopGroupWorker).channel(NioSocketChannel.class)
 .option(ChannelOption.TCP_NODELAY, true)
 .option(ChannelOption.SO_KEEPALIVE, false)
 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, nettyClientConfig.getConnectTimeoutMillis())
 .option(ChannelOption.SO_SNDBUF, nettyClientConfig.getClientSocketSndBufSize())
 .option(ChannelOption.SO_RCVBUF, nettyClientConfig.getClientSocketRcvBufSize())
 .handler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) throws Exception {
 ChannelPipeline pipeline = ch.pipeline();
 if (nettyClientConfig.isUseTLS()) {
 if (null != sslContext) {
 pipeline.addFirst(defaultEventExecutorGroup, "sslHandler", sslContext.newHandler(ch.alloc()));
 log.info("Prepend SSL handler");
 } else {
 log.warn("Connections are insecure as SSLContext is null!");
 }
 }
 pipeline.addLast(
 // 使用工作线程
 defaultEventExecutorGroup,
 // 编码
 new NettyEncoder(),
 // 解码
 new NettyDecoder(),
 new IdleStateHandler(0, 0, nettyClientConfig.getClientChannelMaxIdleTimeSeconds()),
 // 连接管理器
 new NettyConnectManageHandler(),
 new NettyClientHandler());
 }
 });

 this.timer.scheduleAtFixedRate(new TimerTask() {
 @Override
 public void run() {
 try {
 NettyRemotingClient.this.scanResponseTable();
 } catch (Throwable e) {
 log.error("scanResponseTable exception", e);
 }
 }
 }, 1000 * 3, 1000);

 if (this.channelEventListener != null) {
 this.nettyEventExecutor.start();
 }
 }
}

NettyEncoder源码

package org.apache.rocketmq.remoting.netty;

import io.netty.buffer.ByteBuf;
import java.nio.ByteBuffer;

public class NettyEncoder extends MessageToByteEncoder<RemotingCommand> {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);

 @Override
 public void encode(ChannelHandlerContext ctx, RemotingCommand remotingCommand, ByteBuf out)
 throws Exception {
 try {
 ByteBuffer header = remotingCommand.encodeHeader();
 out.writeBytes(header);
 byte[] body = remotingCommand.getBody();
 if (body != null) {
 out.writeBytes(body);
 }
 } catch (Exception e) {
 log.error("encode exception, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e);
 if (remotingCommand != null) {
 log.error(remotingCommand.toString());
 }
 RemotingUtil.closeChannel(ctx.channel());
 }
 }
}

NettyDecoder源码

其继承LengthFieldBasedFrameDecoder，支持灵活配置编解码参数

package org.apache.rocketmq.remoting.netty;

import io.netty.buffer.ByteBuf;
import io.netty.handler.codec.LengthFieldBasedFrameDecoder;
import java.nio.ByteBuffer;

public class NettyDecoder extends LengthFieldBasedFrameDecoder {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);

 private static final int FRAME_MAX_LENGTH =
 Integer.parseInt(System.getProperty("com.rocketmq.remoting.frameMaxLength", "16777216"));

 public NettyDecoder() {
 super(FRAME_MAX_LENGTH, 0, 4, 0, 4);
 }

 @Override
 public Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception {
 ByteBuf frame = null;
 try {
 frame = (ByteBuf) super.decode(ctx, in);
 if (null == frame) {
 return null;
 }

 ByteBuffer byteBuffer = frame.nioBuffer();

 return RemotingCommand.decode(byteBuffer);
 } catch (Exception e) {
 log.error("decode exception, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e);
 RemotingUtil.closeChannel(ctx.channel());
 } finally {
 if (null != frame) {
 frame.release();
 }
 }

 return null;
 }
}

消息返回状态详解

SEND_OK：消息发送成功。
FLUSH_DISK_TIMEOUT：消息发送成功，但消息刷盘失败，此时服务器宕机则消息丢失
FLUSH_SLAVE_TIMEOUT：主从同步消息时超时，从节点已经收到消息，若此时从节点宕机，则从节点消息丢失
SLAVE_NOT_AVAILABLE：从节点不可用，此时主节点宕机，则消息丢失

对于可靠消息投递，后三种情况均应该考虑补偿。

延迟消息投递

场景：消息发送到Broker后，要特定的时间才会被Consumer消费

特点：目前只支持固定精度的定时消息

固定精度消息原因：支持任意精度消息时，对消息持久化、排序产生较大性能开销

使用方法：对message设置setDelayTimeLevel级别即可

核心类：MessageStoreConfig和ScheduleMessageService

MessageStoreConfig源码

package org.apache.rocketmq.store.config;

public class MessageStoreConfig {
 // 延时投递级别，固定精度
 private String messageDelayLevel = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h";
 ...
}

自定义消息发送规则

场景：把消息发送到指定的队列（Message Queue）

处理类：MessageQueueSelector

用法：调用send(Message msg, MessageQueueSelector selector, Object arg)方法即可

实现代码片段，指定发送到id为1的队列中

SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
 // 外部方法的arg会传递到Object中
 @Override
 public MessageQueue select(List<MessageQueue> list, Message message, Object o) {
 Integer queueNumber = (Integer) o;
 return list.get(queueNumber);
 }
},1);

RocketMQ学习笔记2——入门

Wed, 23 Dec 2020 19:06:00 +0800

生产者模型使用

步骤

创建生产者对象DefaultMQProducer
设置NamesrvAddr
启动生产者服务
创建消息并发送

简单消息发送

创建测试主题

常量定义

public class Const {

	public static final String NAMESRV_ADDR_SINGLE = "10.139.12.52:9876";
}

public class Producer {

	public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
		
		DefaultMQProducer producer = new DefaultMQProducer("test_quick_producer_name");
		
		producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
		producer.start();
		
		for(int i = 0 ; i <5; i ++) {
			//	1.	创建消息
			Message message = new Message("test_quick_topic",	//	主题
					"TagA", //	标签
					"key" + i,	// 	用户自定义的key ,唯一的标识
					("Hello RocketMQ" + i).getBytes());	//	消息内容实体（byte[]）
			SendResult sr = producer.send(message, new MessageQueueSelector() {
				@Override
				public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
					Integer queueNumber = (Integer)arg;
					return mqs.get(queueNumber);
				}
			}, 2);
			System.err.println(sr);
			System.err.println("消息发出: " + sr);
		}
		producer.shutdown();
	}
}

运行结果

SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4]
10:45:15.129 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10911] result: true
10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10909] result: true
10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:9876] result: true

消费者模型使用

步骤

创建消费者对象DefaultMQPushConsumer
设置NamesrvAddr及消费位置ConsumeFromWhere
进行主题订阅Subscribe
注册监听并消费registerMessageListener

简单消息消费

public class Consumer {

	public static void main(String[] args) throws MQClientException {
		// 采用Push方式
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test_quick_consumer_name");
		
		consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
		// 匹配所有Tag
		consumer.subscribe("test_quick_topic", "*");
		
		consumer.registerMessageListener(new MessageListenerConcurrently() {
			
			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
				MessageExt me = msgs.get(0);
				try {
					String topic = me.getTopic();
					String tags = me.getTags();
					String keys = me.getKeys();
					
					String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET);
					System.err.println("topic: " + topic + ",tags: " + tags + ", keys: " + keys + ",body: " + msgBody);
				} catch (Exception e) {
					e.printStackTrace();
 // 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
					return ConsumeConcurrentlyStatus.RECONSUME_LATER;
				}
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		consumer.start();
		System.err.println("consumer start...");
	}
}

模拟异常重试

public class Consumer {

	public static void main(String[] args) throws MQClientException {
		// 采用Push方式
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test_quick_consumer_name");
		
		consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
		consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
		// 匹配所有Tag
		consumer.subscribe("test_quick_topic", "*");
		
		consumer.registerMessageListener(new MessageListenerConcurrently() {
			
			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
				MessageExt me = msgs.get(0);
				try {
					String topic = me.getTopic();
					String tags = me.getTags();
					String keys = me.getKeys();
					// 模拟异常
					if(keys.equals("key1")) {
						System.err.println("消息消费失败..");
						int a = 1/0;
					}
					
					String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET);
					System.err.println("topic: " + topic + ",tags: " + tags + ", keys: " + keys + ",body: " + msgBody);
				} catch (Exception e) {
					e.printStackTrace();
					// 重发次数
					int recousumeTimes = me.getReconsumeTimes();
					System.err.println("recousumeTimes: " + recousumeTimes);
					// 超过3次不再重发
					if(recousumeTimes == 3) {
						//		记录日志....
						// 	做补偿处理
						return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
					}
					// 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
					return ConsumeConcurrentlyStatus.RECONSUME_LATER;
				}
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		consumer.start();
		System.err.println("consumer start...");
	}
}

日志输出

consumer start...
topic: test_quick_topic,tags: TagA, keys: key0,body: Hello RocketMQ0
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 0
topic: test_quick_topic,tags: TagA, keys: key2,body: Hello RocketMQ2
topic: test_quick_topic,tags: TagA, keys: key3,body: Hello RocketMQ3
topic: test_quick_topic,tags: TagA, keys: key4,body: Hello RocketMQ4
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 1
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 2
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 3

四种集群环境

集群环境拓扑图

分类

单点模式（不推荐）
主从模式
双主模式（不推荐）
双主双从、多主多从模式

主从集群环境构建

优势：

主从模式可以保障消息的即时性与可靠性
当投递一条消息后关闭主节点，从节点可以继续提供消费者进行消费，但不能接受消息
主节点上线后会进行消费进度offset同步

基础环境

选择2台服务器进行部署验证

IP	IP
GY11-SyOCM33-101	10.139.12.52	主
GY11-SyOCM33-201	10.139.12.53	从

53节点按照单点部署时操作好对应目录和日志配置

修改主节点配置

52节点修改主节点配置文件broker-a.properties如下

修改了namesrvAddr

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876;10.139.12.53:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

修改从节点配置

修改从节点配置文件broker-a-s.properties如下

拷贝主节点配置后修改id和brokerRole

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876;10.139.12.53:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

主从配置同步到53节点即可

启动NameSrv

启动Namesrv（主从节点均执行）

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin
$ nohup sh mqnamesrv &

启动Broker

主节点启动

$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &

从节点启动

$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a-s.properties >/dev/null 2>&1 &

启动控制台

修改控制台配置地址参数为

rocketmq.config.namesrvAddr=10.139.12.52:9876;10.139.12.53:9876

数据高可用机制故障演练

投递一条消息

关闭主节点，剩余一个slave

启动消费者，消费正常

主节点上线，同步消息位点（offset）使消费端不会重复消费

RocketMQ学习笔记1——初探

Wed, 23 Dec 2020 19:04:00 +0800

RocketMQ整体认知

优势

支持集群模型、负载均衡、水平扩展能力
亿级别的消息堆积能力
采用零拷贝的原理、顺序写盘、随机读
丰富的API使用
代码优秀，底层通信框架采用Netty NIO框架
NameServer代替Zookeeper
强调集群无单点，可扩展，任意一点高可用，水平可扩展
消息失败重试机制，消息可查询
开源社区活跃，成熟度高（经过双十一考验）

核心概念模型

基本概念

消息生产者（Producer）：负责生产消息，一般由业务系统负责生产消息。
消息消费者（Consumer）：负责消费消息，一般是后台系统负责异步消费。
推动式消费（Push Consumer）： Consumer消费的一种类型，该模式下Broker收到数据后会主动推送给消费端。
拉取式消费（Pull Consumer）： Consumer消费的一种类型，应用通常主动调用Consumer的拉消息方法从Broker服务器拉消息。
生产者组（Producer Group）：同一类Producer的集合，这类Producer发送同一类消息且发送逻辑一致。
消费者组（Consumer Group）：同一类Consumer的集合，这类Consumer通常消费同一类消息且消费逻辑一致。
Broker：MQ消息服务（中转角色，用于消息存储于生产消息转发）

源码结构

源码分析以4.4.0为例

源码地址： https://github.com/apache/rocketmq

结构：

结构解析：

rocketmq-common：公用数据结构
rocketmq-distribution：编译模块，编译输出
rocketmq-filter：进行Broker过滤的不感兴趣的消息传输，减小带宽压力
rocketmq-logappender、rocketmq-logging：日志服务
rocketmq-namesrv：Namesrv服务，用于服务协调
rocketmq-openmessaging：对外提供服务
rocketmq-remoting：远程调用接口，封装Netty底层通信
rocketmq-srvutil：提供一些公用的工具方法，比如命令行解析参数
rocketmq-store：消息存储
rocketmq-test、rocketmq-example
rocketmq-tools：管理工具，比如有名的mqadmin工具

编译部署&环境搭建

选择1台服务器进行部署验证

IP	IP
GY11-SyOCM33-101	10.139.12.52

编译部署

在源码目录执行一下命令编译工程

$ mvn -Prelease-all -DskipTests clean package

编译成功后在rocketmq\distribution\target目录下生成apache-rocketmq.tar.gz包，使用该包解压与从官网下载一致，其他步骤参考二进制部署。

二进制包部署（单点）

此处从官网下载4.4.0版本并上传至服务器解压

下载地址

解压到~/rocketmq-test/rocketmq-all-4.4.0-bin-release/如下

$ ll
total 36
drwxr-xr-x 2 ocmuser gpocm 83 Jan 17 2019 benchmark
drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 bin
drwxr-xr-x 5 ocmuser gpocm 196 Jan 17 2019 conf
drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 lib
-rw-r--r-- 1 ocmuser gpocm 17336 Jan 17 2019 LICENSE
-rw-r--r-- 1 ocmuser gpocm 1337 Jan 17 2019 NOTICE
-rw-r--r-- 1 ocmuser gpocm 2481 Jan 17 2019 README.md

创建存储目录

$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index

RocketMQ 配置文件

进入提供好的样例配置文件目录，其中包含双主双从异步复制、双主双从同步复制和双主集群配置文件，此处选择双主双从异步复制作为配置模板

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf]$ll
total 36
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync
drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave
-rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf
-rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml
-rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml
-rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml
-rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml
-rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml

选择broker-a配置文件

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async]$ll
total 16
-rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-a.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties
-rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-b.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties

删除原有配置，加入如下配置（单点）

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

修改日志配置文件

$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs 
$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf && sed -i 's#${user.home}#/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release#g' *.xml

修改启动脚本参数

将默认8g的内存配置改成1g

$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runbroker.sh

runbroker.sh

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"
...

同理

$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runserver.sh

runserver.sh

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"
...

启动 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ nohup sh mqnamesrv &

检查进程

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin]$jps
65322 NamesrvStartup
65534 Jps

启动 BrokerServer A

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &

$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/namesrv.log

停止与数据清理

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ sh mqshutdown broker 
$ sh mqshutdown namesrv 
# --等待停止 
$ rm -rf /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index 
# --按照上面步骤重启 NameServer 与 BrokerServer

SpringBoot 控制台讲解

克隆工程：https://github.com/apache/rocketmq-externals.git

进入控制台配置文件rocketmq-externals\rocketmq-console\src\main\resources\application.properties修改配置

server.contextPath=
server.port=9090

## SSL setting
#server.ssl.key-store=classpath:rmqcngkeystore.jks
#server.ssl.key-store-password=rocketmq
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=rmqcngkey

#spring.application.index=true
spring.application.name=rocketmq-console
spring.http.encoding.charset=UTF-8
spring.http.encoding.enabled=true
spring.http.encoding.force=true
logging.config=classpath:logback.xml
#if this value is empty,use env value rocketmq.config.namesrvAddr NAMESRV_ADDR | now, you can set it in ops page.default localhost:9876
rocketmq.config.namesrvAddr=10.139.12.52:9876
#if you use rocketmq version < 3.5.8, rocketmq.config.isVIPChannel should be false.default true
rocketmq.config.isVIPChannel=
#rocketmq-console's data path:dashboard/monitor
rocketmq.config.dataPath=./data
#set it false if you don't want use dashboard.default true
rocketmq.config.enableDashBoardCollect=true
#set the message track trace topic if you don't want use the default one

RocketMQ on F嘉阳

RocketMQ学习笔记6——集群实战

组建高性能MQ服务

集群拓扑图

配置文件最优设置解析

修改host文件映射

创建存储目录

建立软连接

修改配置文件

broker-a.properties 配置

broker-b.properties 配置

broker-a-s.properties 配置

broker-b-s.properties 配置

修改日志配置文件

修改脚本启动参数

启动 NameServer

启动 Master1:BrokerServerA

启动 Master2:BrokerServerB

启动 Master1-Slave:BrokerServerC

启动 Master2-Slave:BrokerServerD

控制台监控验证

服务停止

双主双从环境构建与调优

RocketMQ学习笔记5——原理解析

消息存储结构

同步\异步刷盘

异步刷盘

同步刷盘

同步\异步复制

同步复制

异步复制

同步双写

高可用机制

最佳实践

NameServer协调者

为什么不用zookeeper

RocketMQ学习笔记4——消费者核心

配置参数详解

PushConsumer核心参数

集群与广播模式

集群模式

样例

广播模式

消费存储核心-OffSet

集群模式-RemoteBrokerOffsetStore

广播模式-LocalFileOffsetStore

PushConsumer长轮询模式分析

长轮询过程

源码分析

DefaultMQPullConsumer消息拉取模式

生产者样例

使用普通Pull轮询模式

Pull定时任务实现样例

实现原理

RocketMQ学习笔记3——生产者核心

配置参数

主从同步机制

元数据同步源码分析

同步Topic配置信息源码

同步消费者偏移量源码

消息同步源码

逻辑

连接Master源码

同步\异步消息发送

实现分类

实例

同步消息发送源码分析

底层Netty通信

RemotingClient源码

RemotingServer源码

NettyRemotingServer源码

NettyRemotingAbstract源码

Netty编码解码源码解析——RemotingCommand源码

消息返回状态详解

延迟消息投递

自定义消息发送规则

RocketMQ学习笔记2——入门

生产者模型使用

消费者模型使用

四种集群环境

`broker-a.properties` 配置

`broker-b.properties` 配置

`broker-a-s.properties` 配置

`broker-b-s.properties` 配置

集群模式-`RemoteBrokerOffsetStore`

广播模式-`LocalFileOffsetStore`

`DefaultMQPullConsumer`消息拉取模式

`RemotingClient`源码

`RemotingServer`源码

`NettyRemotingServer`源码

`NettyRemotingAbstract`源码

Netty编码解码源码解析——`RemotingCommand`源码