源码学习 on F嘉阳

RocketMQ学习笔记5——原理解析

Wed, 23 Dec 2020 19:13:00 +0800

消息存储结构

CommitLog：消息主体以及元数据的存储主体，存储Producer端写入的消息主体内容,消息内容不是定长的。消息主要是顺序写入日志文件，当文件满了，写入下一个文件；其日志信息科用于数据恢复，作为MQ数据保障兜底策略。
ConsumerQueue：消息消费队列，存储消息在commit log 中所在的这个位置，也可以说是一个偏移量.引入的目的主要是提高消息消费的性能。
IndexFile：IndexFile（索引文件）提供了一种可以通过key或时间区间来查询消息的方法。
每一个topic下的每一个message queue都对应一个ConsumerQueue文件，具体存储位置通过配置文件配置。
RocketMQ采用的是混合型的存储结构，即为Broker单个实例下所有的队列（ConsumerQueue）共用一个日志数据文件（即为CommitLog）来存储。
刷盘采用顺序写随机读的模式。
消费端采用“0拷贝”模式提升性能。

同步\异步刷盘

消息存储：内存+磁盘存储，两种刷盘方式

异步刷盘

能够充分利用OS的PageCache的优势，只要消息写入PageCache（内存）即可将成功的ACK（成功信息）返回给Producer端。
消息堆积到一定量才触发刷盘。
消息刷盘采用后台异步线程提交的方式进行，降低了读写延迟，提高了MQ的性能和吞吐量。

同步刷盘

只有在消息真正持久化至磁盘后RocketMQ的Broker端才会真正返回给Producer端一个成功的ACK响应。
同步刷盘对MQ消息可靠性来说是一种不错的保障，但是性能上会有较大影响，一般适用于金融业务应用该模式较多。

同步\异步复制

针对集群模式设计，同一组Broker有Master-Slave角色
推荐采用同步双写，异步复制的模式保持性能与可靠性的平衡

同步复制

Master与Slave都成功才返回ACK（成功信息）给Producer

异步复制

Master节点刷盘成功后立即返回ACK（成功信息）给Producer
再通过异步线程将信息同步给其他Slave节点
高性能，但存在数据丢失风险

同步双写

每个Master配置一个Slave，有多对Master-Slave，HA采用同步双写方式，即只有主备都写成功，才向应用返回成功，这种模式的优缺点如下：

优点：数据与服务都无单点故障，Master宕机情况下，消息无延迟，服务可用性与数据可用性都非常高；
缺点：性能比异步复制模式略低（大约低10%左右），发送单个消息的RT会略高，且目前版本在主节点宕机后，备机不能自动切换为主机。

高可用机制

Master-Slave搭配实现高可用机制
通过BrokerId区分节点
Master支持读写，Slave只读
当Master繁忙或者不可用时，可以自动切换到Slave读取消息

最佳实践

创建Topic时，将Topic的Message Queue创建到多个Broker组中，使得逻辑上高可用
由于RocketMQ不支持主从切换，故可以考虑使用监控程序监听主节点的有效性，若连接超时，则将Slave节点的配置改为Master并重启，升级为主节点，实现高可用。

NameServer协调者

NameServer是一个非常简单的Topic路由注册中心，其角色类似Dubbo中的zookeeper，支持Broker的动态注册与发现，是整个集群的状态服务器。
主要包括两个功能：
- Broker管理，NameServer接受Broker集群的注册信息并且保存下来作为路由信息的基本数据。然后提供心跳检测机制，检查Broker是否还存活；
- 路由信息管理，每个NameServer将保存关于Broker集群的整个路由信息和用于客户端查询的队列信息。然后Producer和Conumser通过NameServer就可以知道整个Broker集群的路由信息，从而进行消息的投递和消费。
NameServer通常也是集群的方式部署，各实例间相互不进行信息通讯。Broker是向每一台NameServer注册自己的路由信息，所以每一个NameServer实例上面都保存一份完整的路由信息。
当某个NameServer因某种原因下线了，Broker仍然可以向其它NameServer同步其路由信息，Producer,Consumer仍然可以动态感知Broker的路由的信息。

为什么不用zookeeper

RocketMQ 3.x以前使用zookeeper实现，后来改为nameserver实现
zookeeper提供主从选举，广播原子性等特性在MQ中用不上，功能冗余
zk使用主从自动切换提供高可用保障，nameserver使用多副本节点保障高可用
nameserver只提供路由等基本状态信息维护，代码量小，更加轻量，便于维护。

NameServer信息维护源码

package org.apache.rocketmq.namesrv.routeinfo;

public class RouteInfoManager {
 private final ReadWriteLock lock = new ReentrantReadWriteLock();
 private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;
 private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
 private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
 private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
 private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;
}

RocketMQ学习笔记4——消费者核心

Wed, 23 Dec 2020 19:11:00 +0800

配置参数详解

PushConsumer核心参数

ConsumeFromWhere源码，支持三种参数配置，默认是CONSUME_FROM_LAST_OFFSET

package org.apache.rocketmq.common.consumer;

public enum ConsumeFromWhere {
 /**
 * 第一次启动时从队列最后开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_LAST_OFFSET,

 @Deprecated
 CONSUME_FROM_LAST_OFFSET_AND_FROM_MIN_WHEN_BOOT_FIRST,
 @Deprecated
 CONSUME_FROM_MIN_OFFSET,
 @Deprecated
 CONSUME_FROM_MAX_OFFSET,
 /**
 * 从队列最开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_FIRST_OFFSET,
 /**
 * 指定具体时间点开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_TIMESTAMP,
}

AllocateMessageQueueStrategy接口用于集群模式下消息策略，默认配置为平均消息分配器AllocateMessageQueueAveragely

默认配置源码

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Constructor specifying consumer group and enabled msg trace flag.
 *
 * @param consumerGroup Consumer group.
 * @param enableMsgTrace Switch flag instance for message trace.
 */
 public DefaultMQPushConsumer(final String consumerGroup, boolean enableMsgTrace) {
 this(null, consumerGroup, null, new AllocateMessageQueueAveragely(), enableMsgTrace, null);
 }
}

subscription订阅参数，传递两个值，一个是主题，一个是表达式tag过滤

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Subscription relationship
 */
 private Map<String /* topic */, String /* sub expression */> subscription = new HashMap<String, String>();
}

样例：

consumer.subscribe("test_model_topic2", "TagA||TagB");

consumer.subscribe("test_model_topic2", "TagA");

consumer.subscribe("test_model_topic2", "*");

offsetStore配置核心偏移量，有两个实现，用于支持远程存储和本地存储，即支持集群和单机模式

consumeThreadMin与consumeThreadMax线程池自动调整参数，默认为20

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Minimum consumer thread number
 */
 private int consumeThreadMin = 20;

 /**
 * Max consumer thread number
 */
 private int consumeThreadMax = 20;
}

流控参数consumeConcurrentlyMaxSpan与pullThresholdForQueue

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Concurrently max span offset.it has no effect on sequential consumption
 * 单个队列并行消费的最大跨度
 */
 private int consumeConcurrentlyMaxSpan = 2000;

 /**
 * Flow control threshold on queue level, each message queue will cache at most 1000 messages by default,
 * Consider the {@code pullBatchSize}, the instantaneous value may exceed the limit
 * 一个队列最大的消息消费个数
 */
 private int pullThresholdForQueue = 1000;
}

消息拉取参数pullInterval与pullBatchSize

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Message pull Interval
 * 消息拉取的时间间隔
 */
 private long pullInterval = 0;

 /**
 * Batch pull size
 * 一次拉取的数据量
 */
 private int pullBatchSize = 32;
}

consumeMessageBatchMaxSize一次消息最大拉取的数据数量配置

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Batch consumption size
 * 一次消息最大拉取的数据数量
 */
 private int consumeMessageBatchMaxSize = 1;
}

MessageListener用于消息监听配置，包括并发、顺序读取等

MessageModel配置消息模式，广播或者集群，默认为集群模式

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Message model defines the way how messages are delivered to each consumer clients.
 * </p>
 *
 * RocketMQ supports two message models: clustering and broadcasting. If clustering is set, consumer clients with
 * the same {@link #consumerGroup} would only consume shards of the messages subscribed, which achieves load
 * balances; Conversely, if the broadcasting is set, each consumer client will consume all subscribed messages
 * separately.
 * </p>
 *
 * This field defaults to clustering.
 */
 private MessageModel messageModel = MessageModel.CLUSTERING;
}

集群与广播模式

集群模式

MessageModel源码支持两种模式，集群和广播

package org.apache.rocketmq.common.protocol.heartbeat;

/**
 * Message model
 */
public enum MessageModel {
 /**
 * broadcast
 */
 BROADCASTING("BROADCASTING"),
 /**
 * clustering（默认）
 */
 CLUSTERING("CLUSTERING");

 private String modeCN;

 MessageModel(String modeCN) {
 this.modeCN = modeCN;
 }

 public String getModeCN() {
 return modeCN;
 }
}

集群模式下GroupName用于把多个Consumer组织到一起，所以同一个集群，GroupName必须相同
相同的GroupName的Consumer只消费所订阅消息的一部分，即支持负载均衡
优势：集群模式下增加节点能动态增加负载均衡
一条消息只会被一个Consumer消费

样例

消息消费与奇偶性无关，随机负载均衡，并且负载均衡存在一定的差值，并非100%平分
为提高附在均衡度，可以用2个consume监听4个队列，或者3个consume监听6个队列
Producer只能保证平均分配到各个队列，不能保证被平均的消费

生产者

public class Producer {
	
 public static void main(String[] args) throws MQClientException, InterruptedException {
 	String group_name = "test_model_producer_name";
 DefaultMQProducer producer = new DefaultMQProducer(group_name);
 producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 producer.start();
 
 for (int i = 0; i < 10; i++) {
 try {
 	String tag = (i % 2 ==0) ? "TagA":"TagB";
 Message msg = new Message("test_model_topic2",// topic
 		tag,// tag
 ("信息内容" + i).getBytes()// body
 );
 SendResult sendResult = producer.send(msg);
 System.out.println(sendResult);
 }
 catch (Exception e) {
 e.printStackTrace();
 Thread.sleep(1000);
 }
 }
 
 producer.shutdown();
 }
}

消费者1

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.CLUSTERING);
//			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					//if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					//}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
		
	}
}

消费者2

public class Consumer2 {

	public Consumer2() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.CLUSTERING);
//			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					//if(tags.equals("TagB")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					//}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer2 c2 = new Consumer2();
		System.out.println("c2 start..");
		
	}
}

发送10条消息结果

消费者1结果

消费者2结果

注意，若使用集群模式，则tag表达式应该为*号，才能支持多consumer消费，若集群模式下指定tag表达式，则会导致消费结果不确定，原因如下

consumerA监听4条队列中的2条，consumerB监听4条队列中的另外2条。
Producer将包含TagA和TagB的消息均分到4条队列中。
consumerA消费TagA的消息，那么此时consumerA监听的2条队列中的TagB消息则永远不会被消费

样例

consumer1改成监听TagA，其他内容不变

consumer.subscribe("test_model_topic2", "TagA");

此时发送10条消息，consumer1监听的结果

此时将consumer2改成监听TagB并启动

consumer.subscribe("test_model_topic2", "TagB");

此时发送10条消息，consumer1监听的结果

consumer2监听结果为空，大量消息丢失

经过实测发现，只要集群模式下指定了tag，那么只有一个consumer能正常进行消费，其他consumer都为空

广播模式

同一个ConsumeGroup里的Consumer都消费订阅Topic全部信息
也就是一条消息会被每个Consume消费

样例

将consumer1和2改成广播模式，并监听所有tag

consumer.setMessageModel(MessageModel.BROADCASTING);
consumer.subscribe("test_model_topic2", "*");

发送10条消息consumer1消费结果

consumer2消费结果

注意广播模式下使用服务端tag过滤仍然有bug

将consumer1设置为只监听tagA消息

consumer.subscribe("test_model_topic2", "TagA");

consumer2设置为只监听tag2消息

consumer.subscribe("test_model_topic2", "TagB");

发送10条消息consumer1消费结果

consumer2消费结果

结果只有其中一个consumer能正常消费，与集群模式一样的问题

目前（4.4.0）若想要实现同一个topic的tag过滤，则只能让consumer监听所有消息并主动判断

样例：

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
	}
}

发送10条消息结果

消费者1结果

消费者2结果

对于consumer消费指定队列消息，推荐做法是将消息投递到专属的队列中，由同一个consumer监听，即相当于用groupname+topic唯一定位一个consumer，该方法集群与广播模式均适用

样例

此时consumer1变为test_model_consumer_name1

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name1";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "TagA");
//			consumer.setMessageModel(MessageModel.CLUSTERING);
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
	}
}

consumer2变为test_model_consumer_name2

public class Consumer2 {

	public Consumer2() {
		try {
			String group_name = "test_model_consumer_name2";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "TagB");
//			consumer.setMessageModel(MessageModel.CLUSTERING);
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagB")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer2 c2 = new Consumer2();
		System.out.println("c2 start..");
	}
}

发送10条消息结果

消费者1结果

消费者2结果

消费存储核心-OffSet

Offset是消费者消费进度的核心
Offset指某个topic下的一条消息在某个MessageQueue里的位置
通过Offset可以进行定位到这条消息
Offset的存储实现分为远程文件类型和本地文件类型两种

集群模式-`RemoteBrokerOffsetStore`

默认集群模式clustering，采用远程文件存储Offset
本质上因为多消费模式，每个Consumer消费所订阅主题的一部分

广播模式-`LocalFileOffsetStore`

广播模式下，由于每个Consumer都会收到消息且消费
各个Consumer之间没有任何干扰，独立线程消费
所以使用LocalFileOffsetStore，也就是把Offset存储到本地

PushConsumer长轮询模式分析

DefaultMQPushConsumer是使用长轮询模式进行实现的
通常主流的消息获取模式：Push消息推送模式与Pull消息拉取模式
Push模式消耗broker端性能，若消费端消费速度不一致，可能导致消息堆积
pull模式消耗消费端性能，消费端不知道消息达到时间
RocketMQ采用长轮询机制（消费端主动获取）

长轮询过程

客户端发起获取消息请求，假设一次请求超时时间为15s
服务端判断是否有消息，有则直接返回
服务端无消息，则等待5s（阈值），5s后再次检查是否有消息，有则返回，无则继续等待
15s后循环检查结束，若仍然无消息，则返回给客户端

优势：

主动权在客户端，当服务端消息堆积，不会全推送到客户端

缺点：

连接返回时间较长，占用客户端资源（请求阻塞，等待服务端返回）

源码分析

源码实现位置

PullRequestHoldService继承自Runnable接口

核心类PullRequestHoldService源码全貌

package org.apache.rocketmq.broker.longpolling;

public class PullRequestHoldService extends ServiceThread {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
 private static final String TOPIC_QUEUEID_SEPARATOR = "@";
 private final BrokerController brokerController;
 private final SystemClock systemClock = new SystemClock();
 private ConcurrentMap<String/* topic@queueId */, ManyPullRequest> pullRequestTable =
 new ConcurrentHashMap<String, ManyPullRequest>(1024);

 public PullRequestHoldService(final BrokerController brokerController) {
 this.brokerController = brokerController;
 }

 public void suspendPullRequest(final String topic, final int queueId, final PullRequest pullRequest) {
 String key = this.buildKey(topic, queueId);
 ManyPullRequest mpr = this.pullRequestTable.get(key);
 if (null == mpr) {
 mpr = new ManyPullRequest();
 ManyPullRequest prev = this.pullRequestTable.putIfAbsent(key, mpr);
 if (prev != null) {
 mpr = prev;
 }
 }

 mpr.addPullRequest(pullRequest);
 }

 private String buildKey(final String topic, final int queueId) {
 StringBuilder sb = new StringBuilder();
 sb.append(topic);
 sb.append(TOPIC_QUEUEID_SEPARATOR);
 sb.append(queueId);
 return sb.toString();
 }

 @Override
 public void run() {
 log.info("{} service started", this.getServiceName());
 while (!this.isStopped()) {
 try {
 // 判断是否启用长轮询
 if (this.brokerController.getBrokerConfig().isLongPollingEnable()) {
 // 等待5s
 this.waitForRunning(5 * 1000);
 } else {
 this.waitForRunning(this.brokerController.getBrokerConfig().getShortPollingTimeMills());
 }

 long beginLockTimestamp = this.systemClock.now();
 // 检查请求
 this.checkHoldRequest();
 long costTime = this.systemClock.now() - beginLockTimestamp;
 if (costTime > 5 * 1000) {
 log.info("[NOTIFYME] check hold request cost {} ms.", costTime);
 }
 } catch (Throwable e) {
 log.warn(this.getServiceName() + " service has exception. ", e);
 }
 }

 log.info("{} service end", this.getServiceName());
 }

 @Override
 public String getServiceName() {
 return PullRequestHoldService.class.getSimpleName();
 }

 private void checkHoldRequest() {
 // 遍历所有请求的channel
 for (String key : this.pullRequestTable.keySet()) {
 String[] kArray = key.split(TOPIC_QUEUEID_SEPARATOR);
 if (2 == kArray.length) {
 String topic = kArray[0];
 int queueId = Integer.parseInt(kArray[1]);
 final long offset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
 try {
 // 唤醒
 this.notifyMessageArriving(topic, queueId, offset);
 } catch (Throwable e) {
 log.error("check hold request failed. topic={}, queueId={}", topic, queueId, e);
 }
 }
 }
 }

 public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset) {
 notifyMessageArriving(topic, queueId, maxOffset, null, 0, null, null);
 }

 public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset, final Long tagsCode,
 long msgStoreTime, byte[] filterBitMap, Map<String, String> properties) {
 String key = this.buildKey(topic, queueId);
 // 取出等待的channel（请求）
 ManyPullRequest mpr = this.pullRequestTable.get(key);
 if (mpr != null) {
 List<PullRequest> requestList = mpr.cloneListAndClear();
 if (requestList != null) {
 List<PullRequest> replayList = new ArrayList<PullRequest>();
 // 循环所有的请求
 for (PullRequest request : requestList) {
 long newestOffset = maxOffset;
 if (newestOffset <= request.getPullFromThisOffset()) {
 newestOffset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
 }

 if (newestOffset > request.getPullFromThisOffset()) {
 boolean match = request.getMessageFilter().isMatchedByConsumeQueue(tagsCode,
 new ConsumeQueueExt.CqExtUnit(tagsCode, msgStoreTime, filterBitMap));
 // match by bit map, need eval again when properties is not null.
 if (match && properties != null) {
 match = request.getMessageFilter().isMatchedByCommitLog(null, properties);
 }
 // 如果匹配上
 if (match) {
 try {
 // 唤醒request
 this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
 request.getRequestCommand());
 } catch (Throwable e) {
 log.error("execute request when wakeup failed.", e);
 }
 continue;
 }
 }

 if (System.currentTimeMillis() >= (request.getSuspendTimestamp() + request.getTimeoutMillis())) {
 try {
 this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
 request.getRequestCommand());
 } catch (Throwable e) {
 log.error("execute request when wakeup failed.", e);
 }
 continue;
 }

 replayList.add(request);
 }

 if (!replayList.isEmpty()) {
 mpr.addPullRequest(replayList);
 }
 }
 }
 }
}

ManyPullRequest里存储的实际为PullRequest数组

public class ManyPullRequest {
 private final ArrayList<PullRequest> pullRequestList = new ArrayList<>();
	...
}

PullRequest实际存储NIO请求

public class PullRequest {
 private final RemotingCommand requestCommand;
 /** NIO channel */
 private final Channel clientChannel;
 private final long timeoutMillis;
 private final long suspendTimestamp;
 private final long pullFromThisOffset;
 private final SubscriptionData subscriptionData;
 private final MessageFilter messageFilter;
	...
}

`DefaultMQPullConsumer`消息拉取模式

实现类DefaultMQPullConsumer
主要逻辑
- 获取Message Queue并遍历
- 维护OffsetStore
- 根据不同消息状态做不同处理

生产者样例

public class Producer {
 public static void main(String[] args) throws MQClientException, InterruptedException {
 	
 	String group_name = "test_pull_producer_name";
 DefaultMQProducer producer = new DefaultMQProducer(group_name);
 producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 producer.start();
 
 for (int i = 0; i < 10; i++) {
 try {
 Message msg = new Message("test_pull_topic",// topic
 		"TagA",// tag
 		("Hello RocketMQ " + i).getBytes()// body
 );
 // 配置延迟
 msg.setDelayTimeLevel(2);
 
 SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
 // 外部方法的arg会传递到Object中
 @Override
 public MessageQueue select(List<MessageQueue> list, Message message, Object o) {
 Integer queueNumber = (Integer) o;
 return list.get(queueNumber);
 }
 },1);

 SendStatus status = sendResult.getSendStatus();
 System.out.println(status);

 Thread.sleep(1000);
 }
 catch (Exception e) {
 e.printStackTrace();
 Thread.sleep(3000);
 }
 }
 producer.shutdown();
 }
}

发送消息后的结果，在队列1有45条消息

使用普通Pull轮询模式

public class PullConsumer {
	//Map<key, value> key为指定的队列，value为这个队列拉取数据的最后位置
 private static final Map<MessageQueue, Long> offseTable = new HashMap<MessageQueue, Long>();
 
 public static void main(String[] args) throws MQClientException {
 	
 	String group_name = "test_pull_consumer_name";
 DefaultMQPullConsumer consumer = new DefaultMQPullConsumer(group_name);
 consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 consumer.start();
 System.err.println("consumer start");
 //	从TopicTest这个主题去获取所有的队列（默认会有4个队列）
 Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("test_pull_topic");
 //	遍历每一个队列，进行拉取数据
 for (MessageQueue mq : mqs) {
 System.out.println("Consume from the queue: " + mq);
 
 SINGLE_MQ: while (true) {
 try {
 	//	从queue中获取数据，从什么位置开始拉取数据 单次对多拉取32条记录
 PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
 System.out.println(pullResult);
 System.out.println(pullResult.getPullStatus());
 System.out.println();
 // 客户端自行维护Offset
 putMessageQueueOffset(mq, pullResult.getNextBeginOffset());
 switch (pullResult.getPullStatus()) {
	 case FOUND:
	 	List<MessageExt> list = pullResult.getMsgFoundList();
	 	for(MessageExt msg : list){
	 		System.out.println(new String(msg.getBody()));
	 	}
	 break;
	 case NO_MATCHED_MSG:
	 break;
	 case NO_NEW_MSG:
	 	System.out.println("没有新的数据啦...");
	 break SINGLE_MQ;
	 case OFFSET_ILLEGAL:
	 break;
	 default:
	 break;
 }
 }
 catch (Exception e) {
 e.printStackTrace();
 }
 }
 }
 consumer.shutdown();
 }
 
 
 private static void putMessageQueueOffset(MessageQueue mq, long offset) {
 offseTable.put(mq, offset);
 }
 
 
 private static long getMessageQueueOffset(MessageQueue mq) {
 Long offset = offseTable.get(mq);
 if (offset != null)
 return offset;
 return 0;
 }
}

消费结果，其会随机选择队列读取，导致服务端积压的消息得不到有效的消费

而且该模式需要客户端编码实现Offset存储

Pull定时任务实现样例

public class PullScheduleService {

 public static void main(String[] args) throws MQClientException {
 	
 	String group_name = "test_pull_consumer_name";
 	// 使用定时任务
 final MQPullConsumerScheduleService scheduleService = new MQPullConsumerScheduleService(group_name);
 
 scheduleService.getDefaultMQPullConsumer().setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 
 scheduleService.setMessageModel(MessageModel.CLUSTERING);
 
 scheduleService.registerPullTaskCallback("test_pull_topic", new PullTaskCallback() {
 
 @Override
 public void doPullTask(MessageQueue mq, PullTaskContext context) {
 MQPullConsumer consumer = context.getPullConsumer();
 System.err.println("-------------- queueId: " + mq.getQueueId() + "-------------");
 try {
 // 获取从哪里拉取
 long offset = consumer.fetchConsumeOffset(mq, false);
 if (offset < 0)
 offset = 0;
 
 PullResult pullResult = consumer.pull(mq, "*", offset, 32);
 switch (pullResult.getPullStatus()) {
 case FOUND:
 	List<MessageExt> list = pullResult.getMsgFoundList();
 	for(MessageExt msg : list){
 		//消费数据...
 		System.out.println(new String(msg.getBody()));
 	}
 break;
 case NO_MATCHED_MSG:
 break;
 case NO_NEW_MSG:
 case OFFSET_ILLEGAL:
 break;
 default:
 break;
 }
 // 客户端自动更新Offset，无需本地持久化维护
 consumer.updateConsumeOffset(mq, pullResult.getNextBeginOffset());
 // 设置再过3000ms后重新拉取
 context.setPullNextDelayTimeMillis(3000);
 
 }
 catch (Exception e) {
 e.printStackTrace();
 }
 }
 });
 
 scheduleService.start();
 }
}

运行结果，由于采用多线程读取所有队列，所以当服务端有消息时可以及时获取

实现原理

MQPullConsumerScheduleService中将PullTaskCallback方法作为Map存储
并使用ScheduledThreadPoolExecutor作为线程池，配置线程数为10
线程通过调度taskTable中的PullTaskImpl任务，实现方法回调

package org.apache.rocketmq.client.consumer;

public class MQPullConsumerScheduleService {
 private final InternalLogger log = ClientLogger.getLog();
 private final MessageQueueListener messageQueueListener = new MessageQueueListenerImpl();
 private final ConcurrentMap<MessageQueue, PullTaskImpl> taskTable =
 new ConcurrentHashMap<MessageQueue, PullTaskImpl>();
 private DefaultMQPullConsumer defaultMQPullConsumer;
 private int pullThreadNums = 20;
 private ConcurrentMap<String /* topic */, PullTaskCallback> callbackTable =
 new ConcurrentHashMap<String, PullTaskCallback>();
 private ScheduledThreadPoolExecutor scheduledThreadPoolExecutor;

 public MQPullConsumerScheduleService(final String consumerGroup) {
 this.defaultMQPullConsumer = new DefaultMQPullConsumer(consumerGroup);
 this.defaultMQPullConsumer.setMessageModel(MessageModel.CLUSTERING);
 }

 public void putTask(String topic, Set<MessageQueue> mqNewSet) {
 Iterator<Entry<MessageQueue, PullTaskImpl>> it = this.taskTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<MessageQueue, PullTaskImpl> next = it.next();
 if (next.getKey().getTopic().equals(topic)) {
 if (!mqNewSet.contains(next.getKey())) {
 next.getValue().setCancelled(true);
 it.remove();
 }
 }
 }

 for (MessageQueue mq : mqNewSet) {
 if (!this.taskTable.containsKey(mq)) {
 // 需要调度的任务
 PullTaskImpl command = new PullTaskImpl(mq);
 this.taskTable.put(mq, command);
 // 使用多线程调度
 this.scheduledThreadPoolExecutor.schedule(command, 0, TimeUnit.MILLISECONDS);

 }
 }
 }

 public void start() throws MQClientException {
 final String group = this.defaultMQPullConsumer.getConsumerGroup();
 this.scheduledThreadPoolExecutor = new ScheduledThreadPoolExecutor(
 this.pullThreadNums,
 new ThreadFactoryImpl("PullMsgThread-" + group)
 );

 this.defaultMQPullConsumer.setMessageQueueListener(this.messageQueueListener);

 this.defaultMQPullConsumer.start();

 log.info("MQPullConsumerScheduleService start OK, {} {}",
 this.defaultMQPullConsumer.getConsumerGroup(), this.callbackTable);
 }

 public void registerPullTaskCallback(final String topic, final PullTaskCallback callback) {
 // 放入回调任务集合
 this.callbackTable.put(topic, callback);
 this.defaultMQPullConsumer.registerMessageQueueListener(topic, null);
 }

 class MessageQueueListenerImpl implements MessageQueueListener {
 @Override
 public void messageQueueChanged(String topic, Set<MessageQueue> mqAll, Set<MessageQueue> mqDivided) {
 MessageModel messageModel =
 MQPullConsumerScheduleService.this.defaultMQPullConsumer.getMessageModel();
 switch (messageModel) {
 case BROADCASTING:
 MQPullConsumerScheduleService.this.putTask(topic, mqAll);
 break;
 case CLUSTERING:
 MQPullConsumerScheduleService.this.putTask(topic, mqDivided);
 break;
 default:
 break;
 }
 }
 }

 class PullTaskImpl implements Runnable {
 private final MessageQueue messageQueue;
 private volatile boolean cancelled = false;

 public PullTaskImpl(final MessageQueue messageQueue) {
 this.messageQueue = messageQueue;
 }

 @Override
 public void run() {
 String topic = this.messageQueue.getTopic();
 if (!this.isCancelled()) {
 // 获取回调任务
 PullTaskCallback pullTaskCallback =
 MQPullConsumerScheduleService.this.callbackTable.get(topic);
 if (pullTaskCallback != null) {
 final PullTaskContext context = new PullTaskContext();
 context.setPullConsumer(MQPullConsumerScheduleService.this.defaultMQPullConsumer);
 try {
 // 执行客户端实现的回调任务
 pullTaskCallback.doPullTask(this.messageQueue, context);
 } catch (Throwable e) {
 context.setPullNextDelayTimeMillis(1000);
 log.error("doPullTask Exception", e);
 }

 if (!this.isCancelled()) {
 MQPullConsumerScheduleService.this.scheduledThreadPoolExecutor.schedule(this,
 context.getPullNextDelayTimeMillis(), TimeUnit.MILLISECONDS);
 } else {
 log.warn("The Pull Task is cancelled after doPullTask, {}", messageQueue);
 }
 } else {
 log.warn("Pull Task Callback not exist , {}", topic);
 }
 } else {
 log.warn("The Pull Task is cancelled, {}", messageQueue);
 }
 }
 }
}

RocketMQ学习笔记3——生产者核心

Wed, 23 Dec 2020 19:09:00 +0800

配置参数

producerGroup：组名
createTopicKey：创建Topic（不允许使用，应当由架构设计定义）
defaultTopicQueueNums：创建topic时默认的队列数量（默认为4）
sendMsgTimeout：发送消息时的超时时间（默认值：3000，单位：毫秒）
compressMsgBodyOverHowmuch ：消息压缩字节（默认4096）
retryTimesWhenSendFailed ：同步模式下，在返回发送失败之前，内部尝试重新发送消息的最大次数。（默认值：2）
retryTimesWhenSendAsyncFailed：异步模式下，在发送失败之前，内部尝试重新发送消息的最大次数。（默认值：2）
retryAnotherBrokerWhenNotStoreOK：同步模式下，消息保存失败时是否重试其他broker 。（默认值：false ）
maxMessageSize：最大消息限制。（默认值：1024 * 1024 * 4，单位：字节）

主从同步机制

主从同步
同步信息：数据内容+元数据信息
元数据（配置信息）同步：Broker角色识别，为Slave则启动同步任务
- 通过Netty传输
- 实质为定时任务
消息同步（通过Socket）：HAService、HAconnection、WaitNotifyObject
- 通过原生Socket传输，性能高，数据包支持容量大

元数据同步源码分析

元信息同步工程源码位于broker中

发起定时任务源码BrokerController

package org.apache.rocketmq.broker;

public class BrokerController {
 public boolean initialize() throws CloneNotSupportedException {
 ...
 if (result) {
 ...
 // 进行角色判断
 if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) {
 if (this.messageStoreConfig.getHaMasterAddress() != null && this.messageStoreConfig.getHaMasterAddress().length() >= 6) {
 this.messageStore.updateHaMasterAddress(this.messageStoreConfig.getHaMasterAddress());
 this.updateMasterHAServerAddrPeriodically = false;
 } else {
 this.updateMasterHAServerAddrPeriodically = true;
 }
 // 启动定时任务进行元信息同步
 this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

 @Override
 public void run() {
 try {
 BrokerController.this.slaveSynchronize.syncAll();
 } catch (Throwable e) {
 log.error("ScheduledTask syncAll slave exception", e);
 }
 }
 }, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);
 } else {
 this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

 @Override
 public void run() {
 try {
 BrokerController.this.printMasterAndSlaveDiff();
 } catch (Throwable e) {
 log.error("schedule printMasterAndSlaveDiff error.", e);
 }
 }
 }, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);
 }
 }
 ...
 }
}

syncAll方法实现

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 public void syncAll() {
 // 同步Topic配置信息
 this.syncTopicConfig();
 // 同步消费者偏移量
 this.syncConsumerOffset();
 // 同步延迟偏移量
 this.syncDelayOffset();
 // 同步订阅组配置信息
 this.syncSubscriptionGroupConfig();
 }
 ...
}

同步Topic配置信息源码

syncTopicConfig方法实现如下

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 private void syncTopicConfig() {
 String masterAddrBak = this.masterAddr;
 if (masterAddrBak != null) {
 try {
 TopicConfigSerializeWrapper topicWrapper =
 this.brokerController.getBrokerOuterAPI().getAllTopicConfig(masterAddrBak);
 if (!this.brokerController.getTopicConfigManager().getDataVersion()
 .equals(topicWrapper.getDataVersion())) {

 this.brokerController.getTopicConfigManager().getDataVersion()
 .assignNewOne(topicWrapper.getDataVersion());
 this.brokerController.getTopicConfigManager().getTopicConfigTable().clear();
 this.brokerController.getTopicConfigManager().getTopicConfigTable()
 .putAll(topicWrapper.getTopicConfigTable());
 this.brokerController.getTopicConfigManager().persist();

 log.info("Update slave topic config from master, {}", masterAddrBak);
 }
 } catch (Exception e) {
 log.error("SyncTopicConfig Exception, {}", masterAddrBak, e);
 }
 }
 }
 ...
}

其中getAllTopicConfig方法实现如下

package org.apache.rocketmq.broker.out;

public class BrokerOuterAPI {
 ...
 public TopicConfigSerializeWrapper getAllTopicConfig(
 final String addr) throws RemotingConnectException, RemotingSendRequestException,
 RemotingTimeoutException, InterruptedException, MQBrokerException {
 RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_TOPIC_CONFIG, null);
 // 发起远程同步调用
 RemotingCommand response = this.remotingClient.invokeSync(MixAll.brokerVIPChannel(true, addr), request, 3000);
 assert response != null;
 switch (response.getCode()) {
 case ResponseCode.SUCCESS: {
 return TopicConfigSerializeWrapper.decode(response.getBody(), TopicConfigSerializeWrapper.class);
 }
 default:
 break;
 }

 throw new MQBrokerException(response.getCode(), response.getRemark());
 }
 ...
}

invokeSync通过调用封装过的Netty客户端实现

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingClient extends NettyRemotingAbstract implements RemotingClient {
 ...
 /**
 * 同步请求
 * @param addr 请求地址
 * @param request 请求命令
 * @param timeoutMillis 超时时间
 */
 @Override
 public RemotingCommand invokeSync(String addr, final RemotingCommand request, long timeoutMillis)
 throws InterruptedException, RemotingConnectException, RemotingSendRequestException, RemotingTimeoutException {
 long beginStartTime = System.currentTimeMillis();
 final Channel channel = this.getAndCreateChannel(addr);
 if (channel != null && channel.isActive()) {
 try {
 doBeforeRpcHooks(addr, request);
 // 算出极限耗时代价
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTime) {
 // 如果同步等待时长 < 极限耗时 说明请求不成立，抛出异常
 throw new RemotingTimeoutException("invokeSync call timeout");
 }
 // 同步请求实现
 RemotingCommand response = this.invokeSyncImpl(channel, request, timeoutMillis - costTime);
 doAfterRpcHooks(RemotingHelper.parseChannelRemoteAddr(channel), request, response);
 return response;
 } catch (RemotingSendRequestException e) {
 log.warn("invokeSync: send request exception, so close the channel[{}]", addr);
 this.closeChannel(addr, channel);
 throw e;
 } catch (RemotingTimeoutException e) {
 if (nettyClientConfig.isClientCloseSocketIfTimeout()) {
 this.closeChannel(addr, channel);
 log.warn("invokeSync: close socket because of timeout, {}ms, {}", timeoutMillis, addr);
 }
 log.warn("invokeSync: wait response timeout exception, the channel[{}]", addr);
 throw e;
 }
 } else {
 this.closeChannel(addr, channel);
 throw new RemotingConnectException(addr);
 }
 }
 ...
}

invokeSyncImpl实现逻辑

package org.apache.rocketmq.remoting.netty;

public abstract class NettyRemotingAbstract {
 ...
 public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
 final long timeoutMillis)
 throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
 final int opaque = request.getOpaque();

 try {
 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
 this.responseTable.put(opaque, responseFuture);
 final SocketAddress addr = channel.remoteAddress();
 // 使用netty channel
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 // 轮训监听
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 } else {
 responseFuture.setSendRequestOK(false);
 }
 // 异常情况
 responseTable.remove(opaque);
 responseFuture.setCause(f.cause());
 responseFuture.putResponse(null);
 log.warn("send a request command to channel <" + addr + "> failed.");
 }
 });

 RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
 if (null == responseCommand) {
 if (responseFuture.isSendRequestOK()) {
 throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
 responseFuture.getCause());
 } else {
 throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
 }
 }

 return responseCommand;
 } finally {
 this.responseTable.remove(opaque);
 }
 }
	...
}

同步消费者偏移量源码

其中syncConsumerOffset方法实现如下

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 private void syncConsumerOffset() {
 String masterAddrBak = this.masterAddr;
 if (masterAddrBak != null) {
 try {
 ConsumerOffsetSerializeWrapper offsetWrapper =
 this.brokerController.getBrokerOuterAPI().getAllConsumerOffset(masterAddrBak);
 this.brokerController.getConsumerOffsetManager().getOffsetTable()
 .putAll(offsetWrapper.getOffsetTable());
 this.brokerController.getConsumerOffsetManager().persist();
 log.info("Update slave consumer offset from master, {}", masterAddrBak);
 } catch (Exception e) {
 log.error("SyncConsumerOffset Exception, {}", masterAddrBak, e);
 }
 }
 }
 ...
}

getAllConsumerOffset实现逻辑

package org.apache.rocketmq.broker.out;

public class BrokerOuterAPI {
 ...
 public ConsumerOffsetSerializeWrapper getAllConsumerOffset(
 final String addr) throws InterruptedException, RemotingTimeoutException,
 RemotingSendRequestException, RemotingConnectException, MQBrokerException {
 // 封装CMD指令向主节点发起同步offset信息请求，同步时间上限为3000ms
 RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_CONSUMER_OFFSET, null);
 RemotingCommand response = this.remotingClient.invokeSync(addr, request, 3000);
 assert response != null;
 switch (response.getCode()) {
 case ResponseCode.SUCCESS: {
 return ConsumerOffsetSerializeWrapper.decode(response.getBody(), ConsumerOffsetSerializeWrapper.class);
 }
 default:
 break;
 }

 throw new MQBrokerException(response.getCode(), response.getRemark());
 }
 ...
}

封装的CMD指令

package org.apache.rocketmq.common.protocol;

public class RequestCode {
 ...
 public static final int NOTIFY_CONSUMER_IDS_CHANGED = 40;

 public static final int LOCK_BATCH_MQ = 41;

 public static final int UNLOCK_BATCH_MQ = 42;
 public static final int GET_ALL_CONSUMER_OFFSET = 43;

 public static final int GET_ALL_DELAY_OFFSET = 45;
 ...
}

远程命令对象构建，构建后通过remotingClient发送即可

package org.apache.rocketmq.remoting.protocol;

public class RemotingCommand {
 ...
	public static RemotingCommand createRequestCommand(int code, CommandCustomHeader customHeader) {
 RemotingCommand cmd = new RemotingCommand();
 cmd.setCode(code);
 cmd.customHeader = customHeader;
 setCmdVersion(cmd);
 return cmd;
 }
 ...
}

消息同步源码

消息同步工程源码位于store中

HAConnection实现消息读写

HAservice包含节点连接实现HAClient

逻辑

Master节点：

AcceptSocketService：接收Slave节点连接
HAConnection：
- ReadSocketService：读取来自Slave节点的数据
- WriteSocketService：写入Slave节点的数据

Slave节点：

HAService：
- HAClient：对Master节点连接，读写数据

通信协议：Master节点与Slave节点通信协议较简单，主要如下

对象	用途	第几位	字段	数据类型	字节数	说明
`Slave`=>`Master`	上报`CommitLog`已经同步的物理位置	0	`maxPhyOffset`	Long	8	`CommitLog`最大物理位置
`Master`=>`Slave`	传输新的`CommitLog`数据	0	`fromPhyOffset`	Long	8	传输`CommitLog`开始的物理位置
		1	`size`	Int	4	传输`CommitLog`数据长度
		2	`body`	Bytes		传输`CommitLog`数据

连接Master源码

package org.apache.rocketmq.store.ha;

public class HAService {
 ...
 class HAClient extends ServiceThread {
 /**
 * 客户端连接Master，使用NIO函数，提高效率
 * @return
 * @throws ClosedChannelException
 */
 private boolean connectMaster() throws ClosedChannelException {
 if (null == socketChannel) {
 String addr = this.masterAddress.get();
 if (addr != null) {

 SocketAddress socketAddress = RemotingUtil.string2SocketAddress(addr);
 if (socketAddress != null) {
 this.socketChannel = RemotingUtil.connect(socketAddress);
 if (this.socketChannel != null) {
 this.socketChannel.register(this.selector, SelectionKey.OP_READ);
 }
 }
 }

 this.currentReportedOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();

 this.lastWriteTimestamp = System.currentTimeMillis();
 }

 return this.socketChannel != null;
 }
 
 /**
 * Client(Slave)主循环，实现不断从Master传输CommitLog数据
 * 上传Master自己本地的CommitLog已经同步物理位置
 */
 @Override
 public void run() {
 log.info(this.getServiceName() + " service started");

 while (!this.isStopped()) {
 try {
 if (this.connectMaster()) {

 if (this.isTimeToReportOffset()) {
 // 满足上报间隔，上报到Master进度
 // 固定间隔（默认为5s）向Master上报Slave本地CommitLog已经同步到的物理位置
 boolean result = this.reportSlaveMaxOffset(this.currentReportedOffset);
 if (!result) {
 this.closeMaster();
 }
 }

 this.selector.select(1000);
 // 处理Master传输给Slave的CommitLog数据
 boolean ok = this.processReadEvent();
 if (!ok) {
 this.closeMaster();
 }
 // 若进度有变化，上报到Master进度
 if (!reportSlaveMaxOffsetPlus()) {
 continue;
 }
 // Master超时未返回，关闭连接
 long interval =
 HAService.this.getDefaultMessageStore().getSystemClock().now()
 - this.lastWriteTimestamp;
 if (interval > HAService.this.getDefaultMessageStore().getMessageStoreConfig()
 .getHaHousekeepingInterval()) {
 log.warn("HAClient, housekeeping, found this connection[" + this.masterAddress
 + "] expired, " + interval);
 this.closeMaster();
 log.warn("HAClient, master not response some time, so close connection");
 }
 } else {
 this.waitForRunning(1000 * 5);
 }
 } catch (Exception e) {
 log.warn(this.getServiceName() + " service has exception. ", e);
 this.waitForRunning(1000 * 5);
 }
 }

 log.info(this.getServiceName() + " service end");
 }
 ...
 }
 ...
}

RemotingUtil使用原生Socket实现

package org.apache.rocketmq.remoting.common;

public class RemotingUtil {
 public static SocketAddress string2SocketAddress(final String addr) {
 String[] s = addr.split(":");
 InetSocketAddress isa = new InetSocketAddress(s[0], Integer.parseInt(s[1]));
 return isa;
 }
 
 public static SocketChannel connect(SocketAddress remote, final int timeoutMillis) {
 SocketChannel sc = null;
 try {
 // 打开channel
 sc = SocketChannel.open();
 // 设置同步阻塞，可能存在bug
 sc.configureBlocking(true);
 // 设置关闭Socket的延迟事件，那么当线程执行Socket的close方法时，会进入阻塞状态
 // 直到Socket发送完所有剩余数据，或者超过了方法设置的延迟时间，才从close方法返回
 sc.socket().setSoLinger(false, -1);
 // 禁止使用Nagle算法，使用小数据传输
 // Nagle算法通过将缓冲区的小封包自动相连，组成较大的包
 sc.socket().setTcpNoDelay(true);
 // 设置接收缓冲区
 sc.socket().setReceiveBufferSize(1024 * 64);
 // 设置发送缓冲区
 sc.socket().setSendBufferSize(1024 * 64);
 // 连接
 sc.socket().connect(remote, timeoutMillis);
 // bug？又设置非同步阻塞了
 sc.configureBlocking(false);
 return sc;
 } catch (Exception e) {
 if (sc != null) {
 try {
 sc.close();
 } catch (IOException e1) {
 e1.printStackTrace();
 }
 }
 }

 return null;
 }
 ...
}

核心读请求实现

package org.apache.rocketmq.store.ha;

public class HAService {
 ...
 class HAClient extends ServiceThread {
 private boolean processReadEvent() {
 int readSizeZeroTimes = 0;
 while (this.byteBufferRead.hasRemaining()) {
 try {
 int readSize = this.socketChannel.read(this.byteBufferRead);
 if (readSize > 0) {
 lastWriteTimestamp = HAService.this.defaultMessageStore.getSystemClock().now();
 readSizeZeroTimes = 0;
 boolean result = this.dispatchReadRequest();
 if (!result) {
 log.error("HAClient, dispatchReadRequest error");
 return false;
 }
 } else if (readSize == 0) {
 if (++readSizeZeroTimes >= 3) {
 break;
 }
 } else {
 log.info("HAClient, processReadEvent read socket < 0");
 return false;
 }
 } catch (IOException e) {
 log.info("HAClient, processReadEvent read socket exception", e);
 return false;
 }
 }

 return true;
 }
 
 /**
 * 核心读请求实现
 * 
 * 读取Master传输的CommitLog数据，并返回是否正确
 * 如果读取到数据，写入CommitLog
 * 异常原因：
 * 1. Master传输来的数据开始位置offset(masterPhyOffset)不等于Slave的CommitLog数最大offset(slavePhyOffset)
 * 2. 上报到Master进度失败
 * @return 是否正确
 */
 private boolean dispatchReadRequest() {
 final int msgHeaderSize = 8 + 4; // phyoffset + size
 int readSocketPos = this.byteBufferRead.position();

 while (true) {
 // 读取到请求数据
 int diff = this.byteBufferRead.position() - this.dispatchPostion;
 if (diff >= msgHeaderSize) {
 // 读取masterPhyOffset、bodySize
 long masterPhyOffset = this.byteBufferRead.getLong(this.dispatchPostion);
 int bodySize = this.byteBufferRead.getInt(this.dispatchPostion + 8);
 // 设置Slave节点CommitLog文件最大的offset位置
 long slavePhyOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();
 // 校验Master传输来的数据开始位置offset和slave的commitLog数最大offset是否相同
 if (slavePhyOffset != 0) {
 if (slavePhyOffset != masterPhyOffset) {
 log.error("master pushed offset not equal the max phy offset in slave, SLAVE: "
 + slavePhyOffset + " MASTER: " + masterPhyOffset);
 return false;
 }
 }
 // 读取消息
 if (diff >= (msgHeaderSize + bodySize)) {
 // 写入CommitLog
 byte[] bodyData = new byte[bodySize];
 this.byteBufferRead.position(this.dispatchPostion + msgHeaderSize);
 this.byteBufferRead.get(bodyData);

 HAService.this.defaultMessageStore.appendToCommitLog(masterPhyOffset, bodyData);
 // 设置处理到的位置
 this.byteBufferRead.position(readSocketPos);
 this.dispatchPostion += msgHeaderSize + bodySize;
 // 上报到Master进度
 if (!reportSlaveMaxOffsetPlus()) {
 return false;
 }
 // 继续读数据
 continue;
 }
 }
 // 空间写满，重新分配空间
 if (!this.byteBufferRead.hasRemaining()) {
 this.reallocateByteBuffer();
 }

 break;
 }

 return true;
 }
 ...
 }
 ...
}

同步\异步消息发送

实现分类

同步消息发送
- 发送：producer.send(Message msg)
- 核心实现：DefaultMQProducerImpl
异步消息发送
- 发送：producer.send(Message msg,SendCallback sendCallback)
- 核心实现：DefaultMQProducerImpl

实例

public class Producer {

	public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
		
		DefaultMQProducer producer = new DefaultMQProducer("test_quick_producer_name");
		
	producer.setNamesrvAddr(Const.NAMESRV_ADDR_MASTER_SLAVE);
		
		producer.start();
		
		for(int i = 0 ; i <5; i ++) {
			//	1.	创建消息
			Message message = new Message("test_quick_topic",	//	主题
					"TagA", //	标签
					"key" + i,	// 	用户自定义的key ,唯一的标识
					("Hello RocketMQ" + i).getBytes());	//	消息内容实体（byte[]）
			// 异步发送消息
			producer.send(message, new SendCallback() {
				@Override
				public void onSuccess(SendResult sendResult) {
					System.err.println("msgId: " + sendResult.getMsgId() + ", status: " + sendResult.getSendStatus());
					producer.shutdown();
				}
				@Override
				public void onException(Throwable e) {
					e.printStackTrace();
					System.err.println("------发送失败");
				}
			});
		}	
	}
}

同步消息发送源码分析

发送消息方法

package org.apache.rocketmq.client.producer;

public class DefaultMQProducer extends ClientConfig implements MQProducer {
 ...
 @Override
 public SendResult send(Message msg,
 long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 return this.defaultMQProducerImpl.send(msg, timeout);
 }
}

模式枚举

package org.apache.rocketmq.client.impl;

public enum CommunicationMode {
 SYNC,
 ASYNC,
 // 单向，不需要响应
 ONEWAY,
}

消息发送实现类

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public SendResult send(Message msg,
 long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 return this.sendDefaultImpl(msg, CommunicationMode.SYNC, null, timeout);
 }
 
 private SendResult sendDefaultImpl(
 Message msg,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final long timeout
 ) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 // 校验运行状态
 this.makeSureStateOK();
 // 检查消息格式
 Validators.checkMessage(msg, this.defaultMQProducer);
 // 回调ID，用户标记是否同一次消息
 final long invokeID = random.nextLong();
 // 开始发送消息的时间
 long beginTimestampFirst = System.currentTimeMillis();
 long beginTimestampPrev = beginTimestampFirst;
 long endTimestamp = beginTimestampFirst;
 // 查找topic
 TopicPublishInfo topicPublishInfo = this.tryToFindTopicPublishInfo(msg.getTopic());
 if (topicPublishInfo != null && topicPublishInfo.ok()) {
 boolean callTimeout = false;
 // 要发送的消息队列
 MessageQueue mq = null;
 Exception exception = null;
 SendResult sendResult = null;
 // 重试次数，从配置参数中读取
 int timesTotal = communicationMode == CommunicationMode.SYNC ? 1 + this.defaultMQProducer.getRetryTimesWhenSendFailed() : 1;
 int times = 0;
 // 存储每次发送消息选择的broker名称
 String[] brokersSent = new String[timesTotal];
 for (; times < timesTotal; times++) {
 // 最后一次消息发送的brokerName进行存储
 String lastBrokerName = null == mq ? null : mq.getBrokerName();
 // 消息队列选择策略
 MessageQueue mqSelected = this.selectOneMessageQueue(topicPublishInfo, lastBrokerName);
 if (mqSelected != null) {
 mq = mqSelected;
 brokersSent[times] = mq.getBrokerName();
 try {
 beginTimestampPrev = System.currentTimeMillis();
 // 计算代价
 long costTime = beginTimestampPrev - beginTimestampFirst;
 if (timeout < costTime) {
 callTimeout = true;
 break;
 }
 // 消息发送实现
 sendResult = this.sendKernelImpl(msg, mq, communicationMode, sendCallback, topicPublishInfo, timeout - costTime);
 endTimestamp = System.currentTimeMillis();
 // 更新broker可用信息（容错）
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
 switch (communicationMode) {
 case ASYNC:
 return null;
 case ONEWAY:
 return null;
 case SYNC:
 // 同步发送等待返回结果
 if (sendResult.getSendStatus() != SendStatus.SEND_OK) {
 if (this.defaultMQProducer.isRetryAnotherBrokerWhenNotStoreOK()) {
 continue;
 }
 }

 return sendResult;
 default:
 break;
 }
 } catch (RemotingException e) {
 endTimestamp = System.currentTimeMillis();
 // 抛异常则更新broker容错信息
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 continue;
 } catch (MQClientException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 continue;
 } catch (MQBrokerException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 // 抛出如下异常则进行重试
 switch (e.getResponseCode()) {
 case ResponseCode.TOPIC_NOT_EXIST:
 case ResponseCode.SERVICE_NOT_AVAILABLE:
 case ResponseCode.SYSTEM_ERROR:
 case ResponseCode.NO_PERMISSION:
 case ResponseCode.NO_BUYER_ID:
 case ResponseCode.NOT_IN_CURRENT_UNIT:
 continue;
 default:
 if (sendResult != null) {
 return sendResult;
 }

 throw e;
 }
 } catch (InterruptedException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
 log.warn(String.format("sendKernelImpl exception, throw exception, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());

 log.warn("sendKernelImpl exception", e);
 log.warn(msg.toString());
 throw e;
 }
 } else {
 break;
 }
 }

 if (sendResult != null) {
 return sendResult;
 }

 String info = String.format("Send [%d] times, still failed, cost [%d]ms, Topic: %s, BrokersSent: %s",
 times,
 System.currentTimeMillis() - beginTimestampFirst,
 msg.getTopic(),
 Arrays.toString(brokersSent));

 info += FAQUrl.suggestTodo(FAQUrl.SEND_MSG_FAILED);

 MQClientException mqClientException = new MQClientException(info, exception);
 if (callTimeout) {
 throw new RemotingTooMuchRequestException("sendDefaultImpl call timeout");
 }

 if (exception instanceof MQBrokerException) {
 mqClientException.setResponseCode(((MQBrokerException) exception).getResponseCode());
 } else if (exception instanceof RemotingConnectException) {
 mqClientException.setResponseCode(ClientErrorCode.CONNECT_BROKER_EXCEPTION);
 } else if (exception instanceof RemotingTimeoutException) {
 mqClientException.setResponseCode(ClientErrorCode.ACCESS_BROKER_TIMEOUT);
 } else if (exception instanceof MQClientException) {
 mqClientException.setResponseCode(ClientErrorCode.BROKER_NOT_EXIST_EXCEPTION);
 }

 throw mqClientException;
 }

 List<String> nsList = this.getmQClientFactory().getMQClientAPIImpl().getNameServerAddressList();
 if (null == nsList || nsList.isEmpty()) {
 throw new MQClientException(
 "No name server address, please set it." + FAQUrl.suggestTodo(FAQUrl.NAME_SERVER_ADDR_NOT_EXIST_URL), null).setResponseCode(ClientErrorCode.NO_NAME_SERVER_EXCEPTION);
 }

 throw new MQClientException("No route info of this topic, " + msg.getTopic() + FAQUrl.suggestTodo(FAQUrl.NO_TOPIC_ROUTE_INFO),
 null).setResponseCode(ClientErrorCode.NOT_FOUND_TOPIC_EXCEPTION);
 }
}

查找topic方法实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 private TopicPublishInfo tryToFindTopicPublishInfo(final String topic) {
 // 使用Map缓存
 TopicPublishInfo topicPublishInfo = this.topicPublishInfoTable.get(topic);
 if (null == topicPublishInfo || !topicPublishInfo.ok()) {
 // 当无可用topic发布信息时，主动向NameServer拉取一次
 this.topicPublishInfoTable.putIfAbsent(topic, new TopicPublishInfo());
 this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
 topicPublishInfo = this.topicPublishInfoTable.get(topic);
 }
 // 若可用
 if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) {
 return topicPublishInfo;
 } else {
 // 否则根据设置判断是否新建Topic
 this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, true, this.defaultMQProducer);
 topicPublishInfo = this.topicPublishInfoTable.get(topic);
 return topicPublishInfo;
 }
 }
}

客户端获取路由信息实现

package org.apache.rocketmq.client.impl.factory;

public class MQClientInstance {
 public boolean updateTopicRouteInfoFromNameServer(final String topic) {
 return updateTopicRouteInfoFromNameServer(topic, false, null);
 }
 
 public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault,
 DefaultMQProducer defaultMQProducer) {
 try {
 if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
 try {
 TopicRouteData topicRouteData;
 if (isDefault && defaultMQProducer != null) {
 topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(),
 1000 * 3);
 if (topicRouteData != null) {
 for (QueueData data : topicRouteData.getQueueDatas()) {
 int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
 data.setReadQueueNums(queueNums);
 data.setWriteQueueNums(queueNums);
 }
 }
 } else {
 topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, 1000 * 3);
 }
 if (topicRouteData != null) {
 TopicRouteData old = this.topicRouteTable.get(topic);
 boolean changed = topicRouteDataIsChange(old, topicRouteData);
 if (!changed) {
 changed = this.isNeedUpdateTopicRouteInfo(topic);
 } else {
 log.info("the topic[{}] route info changed, old[{}] ,new[{}]", topic, old, topicRouteData);
 }

 if (changed) {
 TopicRouteData cloneTopicRouteData = topicRouteData.cloneTopicRouteData();

 for (BrokerData bd : topicRouteData.getBrokerDatas()) {
 this.brokerAddrTable.put(bd.getBrokerName(), bd.getBrokerAddrs());
 }

 // Update Pub info
 {
 TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData);
 publishInfo.setHaveTopicRouterInfo(true);
 Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<String, MQProducerInner> entry = it.next();
 MQProducerInner impl = entry.getValue();
 if (impl != null) {
 impl.updateTopicPublishInfo(topic, publishInfo);
 }
 }
 }

 // Update sub info
 {
 Set<MessageQueue> subscribeInfo = topicRouteData2TopicSubscribeInfo(topic, topicRouteData);
 Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<String, MQConsumerInner> entry = it.next();
 MQConsumerInner impl = entry.getValue();
 if (impl != null) {
 impl.updateTopicSubscribeInfo(topic, subscribeInfo);
 }
 }
 }
 log.info("topicRouteTable.put. Topic = {}, TopicRouteData[{}]", topic, cloneTopicRouteData);
 this.topicRouteTable.put(topic, cloneTopicRouteData);
 return true;
 }
 } else {
 log.warn("updateTopicRouteInfoFromNameServer, getTopicRouteInfoFromNameServer return null, Topic: {}", topic);
 }
 } catch (Exception e) {
 if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX) && !topic.equals(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC)) {
 log.warn("updateTopicRouteInfoFromNameServer Exception", e);
 }
 } finally {
 this.lockNamesrv.unlock();
 }
 } else {
 log.warn("updateTopicRouteInfoFromNameServer tryLock timeout {}ms", LOCK_TIMEOUT_MILLIS);
 }
 } catch (InterruptedException e) {
 log.warn("updateTopicRouteInfoFromNameServer Exception", e);
 }

 return false;
 }
}

消息队列选择策略实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
 return this.mqFaultStrategy.selectOneMessageQueue(tpInfo, lastBrokerName);
 }
}

负载均衡消息选择策略实现

package org.apache.rocketmq.client.latency;

public class MQFaultStrategy {
 private final static InternalLogger log = ClientLogger.getLog();
 private final LatencyFaultTolerance<String> latencyFaultTolerance = new LatencyFaultToleranceImpl();
 // 发送消息延迟容错开关
 private boolean sendLatencyFaultEnable = false;
 // 消息发送时长的延迟级别数组
 private long[] latencyMax = {50L, 100L, 550L, 1000L, 2000L, 3000L, 15000L};
 // broker预估不可用时长级别数组
 private long[] notAvailableDuration = {0L, 0L, 30000L, 60000L, 120000L, 180000L, 600000L};
 /**
 * 负载均衡消息选择策略
 */
 public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
 if (this.sendLatencyFaultEnable) {
 try {
 // 获取可用队列
 int index = tpInfo.getSendWhichQueue().getAndIncrement();
 for (int i = 0; i < tpInfo.getMessageQueueList().size(); i++) {
 int pos = Math.abs(index++) % tpInfo.getMessageQueueList().size();
 if (pos < 0)
 pos = 0;
 MessageQueue mq = tpInfo.getMessageQueueList().get(pos);
 if (latencyFaultTolerance.isAvailable(mq.getBrokerName())) {
 if (null == lastBrokerName || mq.getBrokerName().equals(lastBrokerName))
 return mq;
 }
 }
 // 选择最优队列
 final String notBestBroker = latencyFaultTolerance.pickOneAtLeast();
 int writeQueueNums = tpInfo.getQueueIdByBroker(notBestBroker);
 if (writeQueueNums > 0) {
 final MessageQueue mq = tpInfo.selectOneMessageQueue();
 if (notBestBroker != null) {
 mq.setBrokerName(notBestBroker);
 mq.setQueueId(tpInfo.getSendWhichQueue().getAndIncrement() % writeQueueNums);
 }
 return mq;
 } else {
 latencyFaultTolerance.remove(notBestBroker);
 }
 } catch (Exception e) {
 log.error("Error occurred when selecting message queue", e);
 }

 return tpInfo.selectOneMessageQueue();
 }

 return tpInfo.selectOneMessageQueue(lastBrokerName);
 }
}

更新broker可用信息（容错）实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) {
 this.mqFaultStrategy.updateFaultItem(brokerName, currentLatency, isolation);
 }
}

与负载均衡消息选择策略位于相同类中

package org.apache.rocketmq.client.latency;

public class MQFaultStrategy {
 	 public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) {
 if (this.sendLatencyFaultEnable) {
 long duration = computeNotAvailableDuration(isolation ? 30000 : currentLatency);
 this.latencyFaultTolerance.updateFaultItem(brokerName, currentLatency, duration);
 }
 } 
}

消息发送实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 private SendResult sendKernelImpl(final Message msg,
 final MessageQueue mq,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final TopicPublishInfo topicPublishInfo,
 final long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 long beginStartTime = System.currentTimeMillis();
 // 获得地址
 String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
 if (null == brokerAddr) {
 tryToFindTopicPublishInfo(mq.getTopic());
 brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
 }

 SendMessageContext context = null;
 if (brokerAddr != null) {
 brokerAddr = MixAll.brokerVIPChannel(this.defaultMQProducer.isSendMessageWithVIPChannel(), brokerAddr);
 // 记录消息内容，因为下面逻辑可能改变消息内容，例如消息压缩
 byte[] prevBody = msg.getBody();
 try {
 //for MessageBatch,ID has been set in the generating process
 if (!(msg instanceof MessageBatch)) {
 MessageClientIDSetter.setUniqID(msg);
 }

 int sysFlag = 0;
 boolean msgBodyCompressed = false;
 if (this.tryToCompressMessage(msg)) {
 sysFlag |= MessageSysFlag.COMPRESSED_FLAG;
 msgBodyCompressed = true;
 }

 final String tranMsg = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
 if (tranMsg != null && Boolean.parseBoolean(tranMsg)) {
 sysFlag |= MessageSysFlag.TRANSACTION_PREPARED_TYPE;
 }
 // 消息发送前校验
 if (hasCheckForbiddenHook()) {
 CheckForbiddenContext checkForbiddenContext = new CheckForbiddenContext();
 checkForbiddenContext.setNameSrvAddr(this.defaultMQProducer.getNamesrvAddr());
 checkForbiddenContext.setGroup(this.defaultMQProducer.getProducerGroup());
 checkForbiddenContext.setCommunicationMode(communicationMode);
 checkForbiddenContext.setBrokerAddr(brokerAddr);
 checkForbiddenContext.setMessage(msg);
 checkForbiddenContext.setMq(mq);
 checkForbiddenContext.setUnitMode(this.isUnitMode());
 this.executeCheckForbiddenHook(checkForbiddenContext);
 }
 // 消息发送前回调处理
 if (this.hasSendMessageHook()) {
 context = new SendMessageContext();
 context.setProducer(this);
 context.setProducerGroup(this.defaultMQProducer.getProducerGroup());
 context.setCommunicationMode(communicationMode);
 context.setBornHost(this.defaultMQProducer.getClientIP());
 context.setBrokerAddr(brokerAddr);
 context.setMessage(msg);
 context.setMq(mq);
 String isTrans = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
 if (isTrans != null && isTrans.equals("true")) {
 context.setMsgType(MessageType.Trans_Msg_Half);
 }

 if (msg.getProperty("__STARTDELIVERTIME") != null || msg.getProperty(MessageConst.PROPERTY_DELAY_TIME_LEVEL) != null) {
 context.setMsgType(MessageType.Delay_Msg);
 }
 this.executeSendMessageHookBefore(context);
 }
 // 构建消息发送请求头
 SendMessageRequestHeader requestHeader = new SendMessageRequestHeader();
 requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup());
 requestHeader.setTopic(msg.getTopic());
 requestHeader.setDefaultTopic(this.defaultMQProducer.getCreateTopicKey());
 requestHeader.setDefaultTopicQueueNums(this.defaultMQProducer.getDefaultTopicQueueNums());
 requestHeader.setQueueId(mq.getQueueId());
 requestHeader.setSysFlag(sysFlag);
 requestHeader.setBornTimestamp(System.currentTimeMillis());
 requestHeader.setFlag(msg.getFlag());
 requestHeader.setProperties(MessageDecoder.messageProperties2String(msg.getProperties()));
 requestHeader.setReconsumeTimes(0);
 requestHeader.setUnitMode(this.isUnitMode());
 requestHeader.setBatch(msg instanceof MessageBatch);
 if (requestHeader.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
 String reconsumeTimes = MessageAccessor.getReconsumeTime(msg);
 if (reconsumeTimes != null) {
 requestHeader.setReconsumeTimes(Integer.valueOf(reconsumeTimes));
 MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_RECONSUME_TIME);
 }

 String maxReconsumeTimes = MessageAccessor.getMaxReconsumeTimes(msg);
 if (maxReconsumeTimes != null) {
 requestHeader.setMaxReconsumeTimes(Integer.valueOf(maxReconsumeTimes));
 MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_MAX_RECONSUME_TIMES);
 }
 }

 SendResult sendResult = null;
 // 消息发送逻辑
 switch (communicationMode) {
 // 异步发送
 case ASYNC:
 Message tmpMessage = msg;
 if (msgBodyCompressed) {
 //If msg body was compressed, msgbody should be reset using prevBody.
 //Clone new message using commpressed message body and recover origin massage.
 //Fix bug:https://github.com/apache/rocketmq-externals/issues/66
 tmpMessage = MessageAccessor.cloneMessage(msg);
 msg.setBody(prevBody);
 }
 long costTimeAsync = System.currentTimeMillis() - beginStartTime;
 if (timeout < costTimeAsync) {
 throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
 }
 sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
 brokerAddr,
 mq.getBrokerName(),
 tmpMessage,
 requestHeader,
 timeout - costTimeAsync,
 communicationMode,
 sendCallback,
 topicPublishInfo,
 this.mQClientFactory,
 this.defaultMQProducer.getRetryTimesWhenSendAsyncFailed(),
 context,
 this);
 break;
 case ONEWAY:
 // 同步发送
 case SYNC:
 long costTimeSync = System.currentTimeMillis() - beginStartTime;
 if (timeout < costTimeSync) {
 throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
 }
 sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
 brokerAddr,
 mq.getBrokerName(),
 msg,
 requestHeader,
 timeout - costTimeSync,
 communicationMode,
 context,
 this);
 break;
 default:
 assert false;
 break;
 }

 if (this.hasSendMessageHook()) {
 // 发送结果放入上下文后返回
 context.setSendResult(sendResult);
 this.executeSendMessageHookAfter(context);
 }

 return sendResult;
 } catch (RemotingException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } catch (MQBrokerException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } catch (InterruptedException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } finally {
 msg.setBody(prevBody);
 }
 }

 throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
 }
}

使用MQClientAPIImpl真正发起Netty网络请求

package org.apache.rocketmq.client.impl;

public class MQClientAPIImpl {
 ...
 public SendResult sendMessage(
 final String addr,
 final String brokerName,
 final Message msg,
 final SendMessageRequestHeader requestHeader,
 final long timeoutMillis,
 final CommunicationMode communicationMode,
 final SendMessageContext context,
 final DefaultMQProducerImpl producer
 ) throws RemotingException, MQBrokerException, InterruptedException {
 return sendMessage(addr, brokerName, msg, requestHeader, timeoutMillis, communicationMode, null, null, null, 0, context, producer);
 }

 /**
 * 封装 RemotingCommand 对象后真正发起网络请求
 */
 public SendResult sendMessage(
 final String addr,
 final String brokerName,
 final Message msg,
 final SendMessageRequestHeader requestHeader,
 final long timeoutMillis,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final TopicPublishInfo topicPublishInfo,
 final MQClientInstance instance,
 final int retryTimesWhenSendFailed,
 final SendMessageContext context,
 final DefaultMQProducerImpl producer
 ) throws RemotingException, MQBrokerException, InterruptedException {
 long beginStartTime = System.currentTimeMillis();
 RemotingCommand request = null;
 if (sendSmartMsg || msg instanceof MessageBatch) {
 SendMessageRequestHeaderV2 requestHeaderV2 = SendMessageRequestHeaderV2.createSendMessageRequestHeaderV2(requestHeader);
 request = RemotingCommand.createRequestCommand(msg instanceof MessageBatch ? RequestCode.SEND_BATCH_MESSAGE : RequestCode.SEND_MESSAGE_V2, requestHeaderV2);
 } else {
 request = RemotingCommand.createRequestCommand(RequestCode.SEND_MESSAGE, requestHeader);
 }
 // 设置请求体
 request.setBody(msg.getBody());

 switch (communicationMode) {
 case ONEWAY:
 this.remotingClient.invokeOneway(addr, request, timeoutMillis);
 return null;
 case ASYNC:
 final AtomicInteger times = new AtomicInteger();
 long costTimeAsync = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTimeAsync) {
 throw new RemotingTooMuchRequestException("sendMessage call timeout");
 }
 this.sendMessageAsync(addr, brokerName, msg, timeoutMillis - costTimeAsync, request, sendCallback, topicPublishInfo, instance,
 retryTimesWhenSendFailed, times, context, producer);
 return null;
 case SYNC:
 long costTimeSync = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTimeSync) {
 throw new RemotingTooMuchRequestException("sendMessage call timeout");
 }
 return this.sendMessageSync(addr, brokerName, msg, timeoutMillis - costTimeSync, request);
 default:
 assert false;
 break;
 }

 return null;
 }

 private SendResult sendMessageSync(
 final String addr,
 final String brokerName,
 final Message msg,
 final long timeoutMillis,
 final RemotingCommand request
 ) throws RemotingException, MQBrokerException, InterruptedException {
 // 同步发网络请求，返回响应结果
 RemotingCommand response = this.remotingClient.invokeSync(addr, request, timeoutMillis);
 assert response != null;
 return this.processSendResponse(brokerName, msg, response);
 }
}

底层Netty通信

类图

`RemotingClient`源码

package org.apache.rocketmq.remoting;

public interface RemotingClient extends RemotingService {

 /**
 * 更新协调服务地址列表
 * @param addrs
 */
 void updateNameServerAddressList(final List<String> addrs);

 /**
 * 更新协调服务地址列表
 * @return
 */
 List<String> getNameServerAddressList();

 /**
 * 同步消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 RemotingCommand invokeSync(final String addr, final RemotingCommand request,
 final long timeoutMillis) throws InterruptedException, RemotingConnectException,
 RemotingSendRequestException, RemotingTimeoutException;

 /**
 * 异步消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeAsync(final String addr, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback) throws InterruptedException, RemotingConnectException,
 RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException;

 /**
 * 单次消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeOneway(final String addr, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingConnectException, RemotingTooMuchRequestException,
 RemotingTimeoutException, RemotingSendRequestException;

 /**
 *用于注册一些底层通信服务，比如进行元数据信息的同步工作，commitlog的同步等
 * @param requestCode 底层业务通信规则码
 * @param processor 注册器（单线程）
 * @param executor 线程池，用于执行注册器的业务逻辑
 */
 void registerProcessor(final int requestCode, final NettyRequestProcessor processor,
 final ExecutorService executor);

 /**
 * 回调函数执行线程设置，用于生产者发送消息后的回调函数
 * @param callbackExecutor
 */
 void setCallbackExecutor(final ExecutorService callbackExecutor);

 /**
 * 获取回调池：NameServer地址
 * @return
 */
 ExecutorService getCallbackExecutor();
 
 /**
 * 用于远程服务地址判断是否可以进行通信
 * @param addr
 * @return
 */
 boolean isChannelWritable(final String addr);
}

registerProcessor方法在实现类中用于初始化时进行注册逻辑

package org.apache.rocketmq.client.impl;

public class MQClientAPIImpl {
 /**
 * Netty 服务启动时初始化工作
 * @param nettyClientConfig 配置信息
 * @param clientRemotingProcessor
 * @param rpcHook
 * @param clientConfig
 */
 public MQClientAPIImpl(final NettyClientConfig nettyClientConfig,
 final ClientRemotingProcessor clientRemotingProcessor,
 RPCHook rpcHook, final ClientConfig clientConfig) {
 this.clientConfig = clientConfig;
 topAddressing = new TopAddressing(MixAll.getWSAddr(), clientConfig.getUnitName());
 this.remotingClient = new NettyRemotingClient(nettyClientConfig, null);
 this.clientRemotingProcessor = clientRemotingProcessor;

 this.remotingClient.registerRPCHook(rpcHook);
 this.remotingClient.registerProcessor(RequestCode.CHECK_TRANSACTION_STATE, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.NOTIFY_CONSUMER_IDS_CHANGED, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.RESET_CONSUMER_CLIENT_OFFSET, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.GET_CONSUMER_STATUS_FROM_CLIENT, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.GET_CONSUMER_RUNNING_INFO, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.CONSUME_MESSAGE_DIRECTLY, this.clientRemotingProcessor, null);
 }
 ...
}

setCallbackExecutor方法用于回调函数执行线程设置，用于生产者发送消息后的回调线程池，例如DefaultMQProducerImpl中的使用方式

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public void setCallbackExecutor(final ExecutorService callbackExecutor) {
	 this.mQClientFactory.getMQClientAPIImpl().getRemotingClient().setCallbackExecutor(callbackExecutor);
 }
 
 /**
 * It will be removed at 4.4.0 cause for exception handling and the wrong Semantics of timeout.
 * A new one will be provided in next version
 * @param msg
 * @param sendCallback
 * @param timeout the <code>sendCallback</code> will be invoked at most time
 * @throws RejectedExecutionException
 */
 @Deprecated
 public void send(final Message msg, final SendCallback sendCallback, final long timeout)
 throws MQClientException, RemotingException, InterruptedException {
 final long beginStartTime = System.currentTimeMillis();
 // 获取回调线程池
 ExecutorService executor = this.getCallbackExecutor();
 try {
 executor.submit(new Runnable() {
 @Override
 public void run() {
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeout > costTime) {
 try {
 sendDefaultImpl(msg, CommunicationMode.ASYNC, sendCallback, timeout - costTime);
 } catch (Exception e) {
 sendCallback.onException(e);
 }
 } else {
 sendCallback.onException(
 new RemotingTooMuchRequestException("DEFAULT ASYNC send call timeout"));
 }
 }

 });
 } catch (RejectedExecutionException e) {
 throw new MQClientException("executor rejected ", e);
 }

 }
	...
}

`RemotingServer`源码

package org.apache.rocketmq.remoting;

public interface RemotingServer extends RemotingService {

 /**
 * 用于注册一些底层通信服务，比如进行元数据信息的同步工作，commitlog的同步等
 * @param requestCode 底层业务通信规则码
 * @param processor 注册器（单线程）
 * @param executor 线程池，用于执行注册器的业务逻辑
 */
 void registerProcessor(final int requestCode, final NettyRequestProcessor processor,
 final ExecutorService executor);

 /**
 * 默认注册方法：比如注册：{@link org.apache.rocketmq.broker.processor.AdminBrokerProcessor}管理Broker工作线程或者namesrv有关的流程器，
 * 比如同步地址、路由topic等信息
 * @param processor
 * @param executor
 */
 void registerDefaultProcessor(final NettyRequestProcessor processor, final ExecutorService executor);

 /**
 * 获取端口
 * @return
 */
 int localListenPort();

 /**
 * 根据业务规则获取Pair<执行器，线程池>
 * @param requestCode 业务规则码
 * @return
 */
 Pair<NettyRequestProcessor, ExecutorService> getProcessorPair(final int requestCode);

 /**
 * 同步消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 RemotingCommand invokeSync(final Channel channel, final RemotingCommand request,
 final long timeoutMillis) throws InterruptedException, RemotingSendRequestException,
 RemotingTimeoutException;

 /**
 * 异步消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeAsync(final Channel channel, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback) throws InterruptedException,
 RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException;

 /**
 * 单向消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeOneway(final Channel channel, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException,
 RemotingSendRequestException;

}

`NettyRemotingServer`源码

其唯一实现类NettyRemotingServer，主要实现Netty编程模型

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingServer extends NettyRemotingAbstract implements RemotingServer {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);
 /**
 * NettyServer辅助类
 */
 private final ServerBootstrap serverBootstrap;
 /**
 * selector组（用于注册并监听event）
 */
 private final EventLoopGroup eventLoopGroupSelector;
 /**
 * 主线程组（用于接收TCP连接）
 */
 private final EventLoopGroup eventLoopGroupBoss;
 /**
 * NettyServer配置
 */
 private final NettyServerConfig nettyServerConfig;

 /**
 * 主要使用在registerProcessor方法中
 */
 private final ExecutorService publicExecutor;

 /**
 * channel事件监听
 */
 private final ChannelEventListener channelEventListener;

 public NettyRemotingServer(final NettyServerConfig nettyServerConfig,
 final ChannelEventListener channelEventListener) {
 super(nettyServerConfig.getServerOnewaySemaphoreValue(), nettyServerConfig.getServerAsyncSemaphoreValue());
 this.serverBootstrap = new ServerBootstrap();
 this.nettyServerConfig = nettyServerConfig;
 this.channelEventListener = channelEventListener;

 int publicThreadNums = nettyServerConfig.getServerCallbackExecutorThreads();
 if (publicThreadNums <= 0) {
 publicThreadNums = 4;
 }

 this.publicExecutor = Executors.newFixedThreadPool(publicThreadNums, new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyServerPublicExecutor_" + this.threadIndex.incrementAndGet());
 }
 });

 this.eventLoopGroupBoss = new NioEventLoopGroup(1, new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyBoss_%d", this.threadIndex.incrementAndGet()));
 }
 });

 // 如果是Linux平台，并且开启了native epoll，就用EpollEventLoopGroup，这个也就是用JNI，调用c写的epoll；否则就用java NIO的NioEventLoopGroup
 if (useEpoll()) {
 this.eventLoopGroupSelector = new EpollEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);
 private int threadTotal = nettyServerConfig.getServerSelectorThreads();

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyServerEPOLLSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
 }
 });
 } else {
 this.eventLoopGroupSelector = new NioEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);
 private int threadTotal = nettyServerConfig.getServerSelectorThreads();

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyServerNIOSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
 }
 });
 }

 loadSslContext();
 }
 
 @Override
 public void start() {
 // 根据配置创建工作线程组
 this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
 nettyServerConfig.getServerWorkerThreads(),
 new ThreadFactory() {

 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyServerCodecThread_" + this.threadIndex.incrementAndGet());
 }
 });
 // Netty编程模型
 ServerBootstrap childHandler =
 this.serverBootstrap.group(this.eventLoopGroupBoss, this.eventLoopGroupSelector)
 .channel(useEpoll() ? EpollServerSocketChannel.class : NioServerSocketChannel.class)
 // sync + accept = backlog
 .option(ChannelOption.SO_BACKLOG, 1024)
 // 允许重复使用本地地址和端口
 .option(ChannelOption.SO_REUSEADDR, true)
 .option(ChannelOption.SO_KEEPALIVE, false)
 .childOption(ChannelOption.TCP_NODELAY, true)
 .childOption(ChannelOption.SO_SNDBUF, nettyServerConfig.getServerSocketSndBufSize())
 .childOption(ChannelOption.SO_RCVBUF, nettyServerConfig.getServerSocketRcvBufSize())
 .localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort()))
 .childHandler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) throws Exception {
 ch.pipeline()
 .addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME,
 new HandshakeHandler(TlsSystemConfig.tlsMode))
 .addLast(defaultEventExecutorGroup,
 new NettyEncoder(),
 new NettyDecoder(),
 new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()),
 new NettyConnectManageHandler(),
 new NettyServerHandler()
 );
 }
 });

 if (nettyServerConfig.isServerPooledByteBufAllocatorEnable()) {
 childHandler.childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
 }

 try {
 ChannelFuture sync = this.serverBootstrap.bind().sync();
 InetSocketAddress addr = (InetSocketAddress) sync.channel().localAddress();
 this.port = addr.getPort();
 } catch (InterruptedException e1) {
 throw new RuntimeException("this.serverBootstrap.bind().sync() InterruptedException", e1);
 }

 if (this.channelEventListener != null) {
 this.nettyEventExecutor.start();
 }

 this.timer.scheduleAtFixedRate(new TimerTask() {

 @Override
 public void run() {
 try {
 NettyRemotingServer.this.scanResponseTable();
 } catch (Throwable e) {
 log.error("scanResponseTable exception", e);
 }
 }
 }, 1000 * 3, 1000);
 }
 ...
}

`NettyRemotingAbstract`源码

NettyRemotingAbstract具体实现三种发送方法

package org.apache.rocketmq.remoting.netty;

public abstract class NettyRemotingAbstract {
 /**
 * 异步调用具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
 long beginStartTime = System.currentTimeMillis();
 final int opaque = request.getOpaque();
 boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
 if (acquired) {
 final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync);
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTime) {
 throw new RemotingTooMuchRequestException("invokeAsyncImpl call timeout");
 }

 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis - costTime, invokeCallback, once);
 this.responseTable.put(opaque, responseFuture);
 try {
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 }
 requestFail(opaque);
 log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel));
 }
 });
 } catch (Exception e) {
 responseFuture.release();
 log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e);
 throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
 }
 } else {
 if (timeoutMillis <= 0) {
 throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast");
 } else {
 String info =
 String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
 timeoutMillis,
 this.semaphoreAsync.getQueueLength(),
 this.semaphoreAsync.availablePermits()
 );
 log.warn(info);
 throw new RemotingTimeoutException(info);
 }
 }
 }
 /**
 * 单向发送具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 public void invokeOnewayImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
 request.markOnewayRPC();
 boolean acquired = this.semaphoreOneway.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
 if (acquired) {
 final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreOneway);
 try {
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 once.release();
 if (!f.isSuccess()) {
 log.warn("send a request command to channel <" + channel.remoteAddress() + "> failed.");
 }
 }
 });
 } catch (Exception e) {
 once.release();
 log.warn("write send a request command to channel <" + channel.remoteAddress() + "> failed.");
 throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
 }
 } else {
 if (timeoutMillis <= 0) {
 throw new RemotingTooMuchRequestException("invokeOnewayImpl invoke too fast");
 } else {
 String info = String.format(
 "invokeOnewayImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
 timeoutMillis,
 this.semaphoreOneway.getQueueLength(),
 this.semaphoreOneway.availablePermits()
 );
 log.warn(info);
 throw new RemotingTimeoutException(info);
 }
 }
 }
 /**
 * 同步调用具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
 final long timeoutMillis)
 throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
 final int opaque = request.getOpaque();

 try {
 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
 this.responseTable.put(opaque, responseFuture);
 final SocketAddress addr = channel.remoteAddress();
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 } else {
 responseFuture.setSendRequestOK(false);
 }

 responseTable.remove(opaque);
 responseFuture.setCause(f.cause());
 responseFuture.putResponse(null);
 log.warn("send a request command to channel <" + addr + "> failed.");
 }
 });

 RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
 if (null == responseCommand) {
 if (responseFuture.isSendRequestOK()) {
 throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
 responseFuture.getCause());
 } else {
 throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
 }
 }

 return responseCommand;
 } finally {
 this.responseTable.remove(opaque);
 }
 }
 ...
}

Netty编码解码源码解析——`RemotingCommand`源码

netty消息数据包结构

RemotingCommand中编码、解码方法体现了其数据包格式和用法

package org.apache.rocketmq.remoting.protocol;

public class RemotingCommand {
 
 public ByteBuffer encode() {
 // 1> header length size
 int length = 4;

 // 2> header data length
 byte[] headerData = this.headerEncode();
 length += headerData.length;

 // 3> body data length
 if (this.body != null) {
 length += body.length;
 }

 ByteBuffer result = ByteBuffer.allocate(4 + length);

 // length
 result.putInt(length);

 // header length
 result.put(markProtocolType(headerData.length, serializeTypeCurrentRPC));

 // header data
 result.put(headerData);

 // body data;
 if (this.body != null) {
 result.put(this.body);
 }

 result.flip();

 return result;
 }
 public static RemotingCommand decode(final ByteBuffer byteBuffer) {
 int length = byteBuffer.limit();
 int oriHeaderLen = byteBuffer.getInt();
 int headerLength = getHeaderLength(oriHeaderLen);

 byte[] headerData = new byte[headerLength];
 // 此处使用的是java nio buffer
 byteBuffer.get(headerData);
 // header封装，长度+类型 对应数据包结构第二位
 RemotingCommand cmd = headerDecode(headerData, getProtocolType(oriHeaderLen));

 int bodyLength = length - 4 - headerLength;
 byte[] bodyData = null;
 if (bodyLength > 0) {
 bodyData = new byte[bodyLength];
 byteBuffer.get(bodyData);
 }
 cmd.body = bodyData;

 return cmd;
 }
 ...
}

其支持的序列化类型有两个

SerializeType序列化类型枚举

package org.apache.rocketmq.remoting.protocol;

public enum SerializeType {
 JSON((byte) 0),
 ROCKETMQ((byte) 1);
 ...
}

在NettyRemotingClient中的Netty编程模型也有编码、解码配置，其使用的是netty buffer

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingClient extends NettyRemotingAbstract implements RemotingClient {
 @Override
 public void start() {
 this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
 nettyClientConfig.getClientWorkerThreads(),
 new ThreadFactory() {

 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyClientWorkerThread_" + this.threadIndex.incrementAndGet());
 }
 });

 Bootstrap handler = this.bootstrap.group(this.eventLoopGroupWorker).channel(NioSocketChannel.class)
 .option(ChannelOption.TCP_NODELAY, true)
 .option(ChannelOption.SO_KEEPALIVE, false)
 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, nettyClientConfig.getConnectTimeoutMillis())
 .option(ChannelOption.SO_SNDBUF, nettyClientConfig.getClientSocketSndBufSize())
 .option(ChannelOption.SO_RCVBUF, nettyClientConfig.getClientSocketRcvBufSize())
 .handler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) throws Exception {
 ChannelPipeline pipeline = ch.pipeline();
 if (nettyClientConfig.isUseTLS()) {
 if (null != sslContext) {
 pipeline.addFirst(defaultEventExecutorGroup, "sslHandler", sslContext.newHandler(ch.alloc()));
 log.info("Prepend SSL handler");
 } else {
 log.warn("Connections are insecure as SSLContext is null!");
 }
 }
 pipeline.addLast(
 // 使用工作线程
 defaultEventExecutorGroup,
 // 编码
 new NettyEncoder(),
 // 解码
 new NettyDecoder(),
 new IdleStateHandler(0, 0, nettyClientConfig.getClientChannelMaxIdleTimeSeconds()),
 // 连接管理器
 new NettyConnectManageHandler(),
 new NettyClientHandler());
 }
 });

 this.timer.scheduleAtFixedRate(new TimerTask() {
 @Override
 public void run() {
 try {
 NettyRemotingClient.this.scanResponseTable();
 } catch (Throwable e) {
 log.error("scanResponseTable exception", e);
 }
 }
 }, 1000 * 3, 1000);

 if (this.channelEventListener != null) {
 this.nettyEventExecutor.start();
 }
 }
}

NettyEncoder源码

package org.apache.rocketmq.remoting.netty;

import io.netty.buffer.ByteBuf;
import java.nio.ByteBuffer;

public class NettyEncoder extends MessageToByteEncoder<RemotingCommand> {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);

 @Override
 public void encode(ChannelHandlerContext ctx, RemotingCommand remotingCommand, ByteBuf out)
 throws Exception {
 try {
 ByteBuffer header = remotingCommand.encodeHeader();
 out.writeBytes(header);
 byte[] body = remotingCommand.getBody();
 if (body != null) {
 out.writeBytes(body);
 }
 } catch (Exception e) {
 log.error("encode exception, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e);
 if (remotingCommand != null) {
 log.error(remotingCommand.toString());
 }
 RemotingUtil.closeChannel(ctx.channel());
 }
 }
}

NettyDecoder源码

其继承LengthFieldBasedFrameDecoder，支持灵活配置编解码参数

package org.apache.rocketmq.remoting.netty;

import io.netty.buffer.ByteBuf;
import io.netty.handler.codec.LengthFieldBasedFrameDecoder;
import java.nio.ByteBuffer;

public class NettyDecoder extends LengthFieldBasedFrameDecoder {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);

 private static final int FRAME_MAX_LENGTH =
 Integer.parseInt(System.getProperty("com.rocketmq.remoting.frameMaxLength", "16777216"));

 public NettyDecoder() {
 super(FRAME_MAX_LENGTH, 0, 4, 0, 4);
 }

 @Override
 public Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception {
 ByteBuf frame = null;
 try {
 frame = (ByteBuf) super.decode(ctx, in);
 if (null == frame) {
 return null;
 }

 ByteBuffer byteBuffer = frame.nioBuffer();

 return RemotingCommand.decode(byteBuffer);
 } catch (Exception e) {
 log.error("decode exception, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e);
 RemotingUtil.closeChannel(ctx.channel());
 } finally {
 if (null != frame) {
 frame.release();
 }
 }

 return null;
 }
}

消息返回状态详解

SEND_OK：消息发送成功。
FLUSH_DISK_TIMEOUT：消息发送成功，但消息刷盘失败，此时服务器宕机则消息丢失
FLUSH_SLAVE_TIMEOUT：主从同步消息时超时，从节点已经收到消息，若此时从节点宕机，则从节点消息丢失
SLAVE_NOT_AVAILABLE：从节点不可用，此时主节点宕机，则消息丢失

对于可靠消息投递，后三种情况均应该考虑补偿。

延迟消息投递

场景：消息发送到Broker后，要特定的时间才会被Consumer消费

特点：目前只支持固定精度的定时消息

固定精度消息原因：支持任意精度消息时，对消息持久化、排序产生较大性能开销

使用方法：对message设置setDelayTimeLevel级别即可

核心类：MessageStoreConfig和ScheduleMessageService

MessageStoreConfig源码

package org.apache.rocketmq.store.config;

public class MessageStoreConfig {
 // 延时投递级别，固定精度
 private String messageDelayLevel = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h";
 ...
}

自定义消息发送规则

场景：把消息发送到指定的队列（Message Queue）

处理类：MessageQueueSelector

用法：调用send(Message msg, MessageQueueSelector selector, Object arg)方法即可

实现代码片段，指定发送到id为1的队列中

SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
 // 外部方法的arg会传递到Object中
 @Override
 public MessageQueue select(List<MessageQueue> list, Message message, Object o) {
 Integer queueNumber = (Integer) o;
 return list.get(queueNumber);
 }
},1);

RocketMQ学习笔记2——入门

Wed, 23 Dec 2020 19:06:00 +0800

生产者模型使用

步骤

创建生产者对象DefaultMQProducer
设置NamesrvAddr
启动生产者服务
创建消息并发送

简单消息发送

创建测试主题

常量定义

public class Const {

	public static final String NAMESRV_ADDR_SINGLE = "10.139.12.52:9876";
}

public class Producer {

	public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
		
		DefaultMQProducer producer = new DefaultMQProducer("test_quick_producer_name");
		
		producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
		producer.start();
		
		for(int i = 0 ; i <5; i ++) {
			//	1.	创建消息
			Message message = new Message("test_quick_topic",	//	主题
					"TagA", //	标签
					"key" + i,	// 	用户自定义的key ,唯一的标识
					("Hello RocketMQ" + i).getBytes());	//	消息内容实体（byte[]）
			SendResult sr = producer.send(message, new MessageQueueSelector() {
				@Override
				public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
					Integer queueNumber = (Integer)arg;
					return mqs.get(queueNumber);
				}
			}, 2);
			System.err.println(sr);
			System.err.println("消息发出: " + sr);
		}
		producer.shutdown();
	}
}

运行结果

SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4]
10:45:15.129 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10911] result: true
10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10909] result: true
10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:9876] result: true

消费者模型使用

步骤

创建消费者对象DefaultMQPushConsumer
设置NamesrvAddr及消费位置ConsumeFromWhere
进行主题订阅Subscribe
注册监听并消费registerMessageListener

简单消息消费

public class Consumer {

	public static void main(String[] args) throws MQClientException {
		// 采用Push方式
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test_quick_consumer_name");
		
		consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
		// 匹配所有Tag
		consumer.subscribe("test_quick_topic", "*");
		
		consumer.registerMessageListener(new MessageListenerConcurrently() {
			
			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
				MessageExt me = msgs.get(0);
				try {
					String topic = me.getTopic();
					String tags = me.getTags();
					String keys = me.getKeys();
					
					String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET);
					System.err.println("topic: " + topic + ",tags: " + tags + ", keys: " + keys + ",body: " + msgBody);
				} catch (Exception e) {
					e.printStackTrace();
 // 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
					return ConsumeConcurrentlyStatus.RECONSUME_LATER;
				}
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		consumer.start();
		System.err.println("consumer start...");
	}
}

模拟异常重试

public class Consumer {

	public static void main(String[] args) throws MQClientException {
		// 采用Push方式
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test_quick_consumer_name");
		
		consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
		consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
		// 匹配所有Tag
		consumer.subscribe("test_quick_topic", "*");
		
		consumer.registerMessageListener(new MessageListenerConcurrently() {
			
			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
				MessageExt me = msgs.get(0);
				try {
					String topic = me.getTopic();
					String tags = me.getTags();
					String keys = me.getKeys();
					// 模拟异常
					if(keys.equals("key1")) {
						System.err.println("消息消费失败..");
						int a = 1/0;
					}
					
					String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET);
					System.err.println("topic: " + topic + ",tags: " + tags + ", keys: " + keys + ",body: " + msgBody);
				} catch (Exception e) {
					e.printStackTrace();
					// 重发次数
					int recousumeTimes = me.getReconsumeTimes();
					System.err.println("recousumeTimes: " + recousumeTimes);
					// 超过3次不再重发
					if(recousumeTimes == 3) {
						//		记录日志....
						// 	做补偿处理
						return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
					}
					// 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
					return ConsumeConcurrentlyStatus.RECONSUME_LATER;
				}
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		consumer.start();
		System.err.println("consumer start...");
	}
}

日志输出

consumer start...
topic: test_quick_topic,tags: TagA, keys: key0,body: Hello RocketMQ0
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 0
topic: test_quick_topic,tags: TagA, keys: key2,body: Hello RocketMQ2
topic: test_quick_topic,tags: TagA, keys: key3,body: Hello RocketMQ3
topic: test_quick_topic,tags: TagA, keys: key4,body: Hello RocketMQ4
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 1
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 2
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 3

四种集群环境

集群环境拓扑图

分类

单点模式（不推荐）
主从模式
双主模式（不推荐）
双主双从、多主多从模式

主从集群环境构建

优势：

主从模式可以保障消息的即时性与可靠性
当投递一条消息后关闭主节点，从节点可以继续提供消费者进行消费，但不能接受消息
主节点上线后会进行消费进度offset同步

基础环境

选择2台服务器进行部署验证

IP	IP
GY11-SyOCM33-101	10.139.12.52	主
GY11-SyOCM33-201	10.139.12.53	从

53节点按照单点部署时操作好对应目录和日志配置

修改主节点配置

52节点修改主节点配置文件broker-a.properties如下

修改了namesrvAddr

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876;10.139.12.53:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

修改从节点配置

修改从节点配置文件broker-a-s.properties如下

拷贝主节点配置后修改id和brokerRole

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876;10.139.12.53:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

主从配置同步到53节点即可

启动NameSrv

启动Namesrv（主从节点均执行）

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin
$ nohup sh mqnamesrv &

启动Broker

主节点启动

$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &

从节点启动

$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a-s.properties >/dev/null 2>&1 &

启动控制台

修改控制台配置地址参数为

rocketmq.config.namesrvAddr=10.139.12.52:9876;10.139.12.53:9876

数据高可用机制故障演练

投递一条消息

关闭主节点，剩余一个slave

启动消费者，消费正常

主节点上线，同步消息位点（offset）使消费端不会重复消费

RocketMQ学习笔记1——初探

Wed, 23 Dec 2020 19:04:00 +0800

RocketMQ整体认知

优势

支持集群模型、负载均衡、水平扩展能力
亿级别的消息堆积能力
采用零拷贝的原理、顺序写盘、随机读
丰富的API使用
代码优秀，底层通信框架采用Netty NIO框架
NameServer代替Zookeeper
强调集群无单点，可扩展，任意一点高可用，水平可扩展
消息失败重试机制，消息可查询
开源社区活跃，成熟度高（经过双十一考验）

核心概念模型

基本概念

消息生产者（Producer）：负责生产消息，一般由业务系统负责生产消息。
消息消费者（Consumer）：负责消费消息，一般是后台系统负责异步消费。
推动式消费（Push Consumer）： Consumer消费的一种类型，该模式下Broker收到数据后会主动推送给消费端。
拉取式消费（Pull Consumer）： Consumer消费的一种类型，应用通常主动调用Consumer的拉消息方法从Broker服务器拉消息。
生产者组（Producer Group）：同一类Producer的集合，这类Producer发送同一类消息且发送逻辑一致。
消费者组（Consumer Group）：同一类Consumer的集合，这类Consumer通常消费同一类消息且消费逻辑一致。
Broker：MQ消息服务（中转角色，用于消息存储于生产消息转发）

源码结构

源码分析以4.4.0为例

源码地址： https://github.com/apache/rocketmq

结构：

结构解析：

rocketmq-common：公用数据结构
rocketmq-distribution：编译模块，编译输出
rocketmq-filter：进行Broker过滤的不感兴趣的消息传输，减小带宽压力
rocketmq-logappender、rocketmq-logging：日志服务
rocketmq-namesrv：Namesrv服务，用于服务协调
rocketmq-openmessaging：对外提供服务
rocketmq-remoting：远程调用接口，封装Netty底层通信
rocketmq-srvutil：提供一些公用的工具方法，比如命令行解析参数
rocketmq-store：消息存储
rocketmq-test、rocketmq-example
rocketmq-tools：管理工具，比如有名的mqadmin工具

编译部署&环境搭建

选择1台服务器进行部署验证

IP	IP
GY11-SyOCM33-101	10.139.12.52

编译部署

在源码目录执行一下命令编译工程

$ mvn -Prelease-all -DskipTests clean package

编译成功后在rocketmq\distribution\target目录下生成apache-rocketmq.tar.gz包，使用该包解压与从官网下载一致，其他步骤参考二进制部署。

二进制包部署（单点）

此处从官网下载4.4.0版本并上传至服务器解压

下载地址

解压到~/rocketmq-test/rocketmq-all-4.4.0-bin-release/如下

$ ll
total 36
drwxr-xr-x 2 ocmuser gpocm 83 Jan 17 2019 benchmark
drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 bin
drwxr-xr-x 5 ocmuser gpocm 196 Jan 17 2019 conf
drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 lib
-rw-r--r-- 1 ocmuser gpocm 17336 Jan 17 2019 LICENSE
-rw-r--r-- 1 ocmuser gpocm 1337 Jan 17 2019 NOTICE
-rw-r--r-- 1 ocmuser gpocm 2481 Jan 17 2019 README.md

创建存储目录

$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index

RocketMQ 配置文件

进入提供好的样例配置文件目录，其中包含双主双从异步复制、双主双从同步复制和双主集群配置文件，此处选择双主双从异步复制作为配置模板

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf]$ll
total 36
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync
drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave
-rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf
-rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml
-rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml
-rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml
-rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml
-rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml

选择broker-a配置文件

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async]$ll
total 16
-rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-a.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties
-rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-b.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties

删除原有配置，加入如下配置（单点）

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

修改日志配置文件

$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs 
$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf && sed -i 's#${user.home}#/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release#g' *.xml

修改启动脚本参数

将默认8g的内存配置改成1g

$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runbroker.sh

runbroker.sh

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"
...

同理

$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runserver.sh

runserver.sh

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"
...

启动 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ nohup sh mqnamesrv &

检查进程

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin]$jps
65322 NamesrvStartup
65534 Jps

启动 BrokerServer A

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &

$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/namesrv.log

停止与数据清理

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ sh mqshutdown broker 
$ sh mqshutdown namesrv 
# --等待停止 
$ rm -rf /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index 
# --按照上面步骤重启 NameServer 与 BrokerServer

SpringBoot 控制台讲解

克隆工程：https://github.com/apache/rocketmq-externals.git

进入控制台配置文件rocketmq-externals\rocketmq-console\src\main\resources\application.properties修改配置

server.contextPath=
server.port=9090

## SSL setting
#server.ssl.key-store=classpath:rmqcngkeystore.jks
#server.ssl.key-store-password=rocketmq
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=rmqcngkey

#spring.application.index=true
spring.application.name=rocketmq-console
spring.http.encoding.charset=UTF-8
spring.http.encoding.enabled=true
spring.http.encoding.force=true
logging.config=classpath:logback.xml
#if this value is empty,use env value rocketmq.config.namesrvAddr NAMESRV_ADDR | now, you can set it in ops page.default localhost:9876
rocketmq.config.namesrvAddr=10.139.12.52:9876
#if you use rocketmq version < 3.5.8, rocketmq.config.isVIPChannel should be false.default true
rocketmq.config.isVIPChannel=
#rocketmq-console's data path:dashboard/monitor
rocketmq.config.dataPath=./data
#set it false if you don't want use dashboard.default true
rocketmq.config.enableDashBoardCollect=true
#set the message track trace topic if you don't want use the default one

SpringBoot源码分析2——走向自动装配

Thu, 10 Dec 2020 17:28:00 +0800

Spring Framework 手动装配

Spring模式注解装配

定义:一种用于声明在应用中扮演“组件”角色的注解
举例:@Component、@Service、@Configuration等
装配:< context: component-scan>或@Componentscan

官方仓库说明

Stereotype Annotations
A *stereotype annotation* is an annotation that is used to declare the role that a component plays within the application. For example, the @Repository annotation in the Spring Framework is a marker for any class that fulfills the role or stereotype of a repository (also known as Data Access Object or DAO).

@Component is a generic stereotype for any Spring-managed component. Any component annotated with @Component is a candidate for component scanning. Similarly, any component annotated with an annotation that is itself meta-annotated with @Component is also a candidate for component scanning. For example, @Service is meta-annotated with @Component.

Core Spring provides several stereotype annotations out of the box, including but not limited to: @Component, @Service, @Repository, @Controller, @RestController, and @Configuration. @Repository, @Service, etc. are specializations of @Component.

模式注解是一种用于声明在应用中扮演“组件”角色的注解。如 Spring Framework 中的 @Repository 标注在任何类上，用于扮演仓储角色的模式注解。

@Component 作为一种由 Spring 容器托管的通用模式组件，任何被 @Component 标准的组件均为组件扫描的候选对象。类似地，凡是被 @Component 元标注（meta-annotated）的注解，如 @Service ，当任何组件标注它时，也被视作组件扫描的候选对象（派生性）

模式注解举例

Spring Framework 注解	场景说明	起始版本
`@Repository`	数据仓储模式注解	2.0
`@Component`	通用组件模式注解	2.5
`@Service`	服务模式注解	2.5
`@Controller`	Web控制器模式注解	2.5
`@Configuration`	配置类模式注解	3.0

装配方式

<context:component-scan> 方式

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:context="http://www.springframework.org/schema/context"
 xsi:schemaLocation="http://www.springframework.org/schema/beans 
http://www.springframework.org/schema/beans/spring-beans.xsd
 http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-
context.xsd">
 
 <!-- 激活注解驱动特性 -->
 <context:annotation-config />
 
 <!-- 找寻被 @Component 或者其派生 Annotation 标记的类（Class），将它们注册为 Spring Bean -->
 <context:component-scan base-package="top.fjy8018.spring.boot" />
 
</beans>

@ComponentScan 方式

@ComponentScan(basePackages = "top.fjy8018.spring.boot")
public class SpringConfiguration {
 ...
}

Spring Framework手动装配

自定义模式注解

`@Component` “派生性”

package top.fjy8018.springboot.annotation;

import org.springframework.stereotype.Component;
import org.springframework.stereotype.Repository;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 10:36
 */
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Repository
public @interface FirstLevelRepository {

 /**
 * 保证签名一致性
 */
 String value() default "";
}

三个级别并无明显区别，体现注解派生性，即使用@FirstLevelRepository和使用@Component效果一致

派生关系

@Component
- @Repository
  - @FirstLevelRepository

package org.springframework.stereotype;

/**
 * @since 2.5
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Indexed
public @interface Component {

 /**
 * Component的签名
 */
 String value() default "";

}

Repository中元标注了@Component

package org.springframework.stereotype;

/**
 * @since 2.0
 * @see Component
 */
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Component
public @interface Repository {

 /**
 * 签名与Component一致
 */
 @AliasFor(annotation = Component.class)
 String value() default "";
}

`@Component` “层次性”

package top.fjy8018.springboot.annotation;

import org.springframework.stereotype.Component;
import org.springframework.stereotype.Repository;

import java.lang.annotation.*;

/**
 * Component层次性
 * 顶级 {@link Component}
 * 一级 {@link Repository @Repository}
 * 二级 {@link FirstLevelRepository}
 * 三级 {@link SecondLevelRepository}
 *
 * @author F嘉阳
 * @date 2018-09-09 10:54
 */
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@FirstLevelRepository
public @interface SecondLevelRepository {

 /**
 * 层次性、派生性都必须保证签名一致
 */
 String value() default "";
}

@Component
- @Repository
  - FirstLevelRepository
    - SecondLevelRepository

注解目标类，value 为bean的名称，此处不管用@FirstLevelRepository还是用SecondLevelRepository，效果都和使用@Component一致

package top.fjy8018.springboot.repository;

import top.fjy8018.springboot.annotation.FirstLevelRepository;

/**
 * @author F嘉阳
 * @date 2018-09-09 10:40
 */
//@Component("myRepository")
@FirstLevelRepository(value = "myRepository")
//@SecondLevelRepository(value = "myRepository")
public class RepositoryDemo {
}

仓储引导类，使用@ComponentScan指定扫描Bean路径

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.ComponentScan;
import top.fjy8018.springboot.repository.RepositoryDemo;

/**
 * @author F嘉阳
 * @date 2018-09-09 10:43
 */
@ComponentScan(basePackages = "top.fjy8018.springboot.repository")
public class RepositoryBootstrapDemo {

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(RepositoryBootstrapDemo.class)
 .web(WebApplicationType.NONE)
 .run(args);

 // 通过bean名称获取bean
 RepositoryDemo myRepository = applicationContext.getBean("myRepository", RepositoryDemo.class);

 // myRepository是否存在，若不存在则报错
 System.out.println("myRepository是否存在：" + myRepository);

 // 关闭上下文
 applicationContext.close();
 }
}

执行结果

SpringBoot源码中也有体现该层次关系

@SpringBootApplication
- @SpringBootConfiguration
  - @Configuration
    - @Component
- @EnableAutoConfiguration
  - @AutoConfigurationPackage

Spring @Enable 模块装配

Spring Framework 3.1 开始支持“@Enable 模块驱动”。所谓“模块”是指具备相同领域的功能组件集合，组合所形成一个独立的单元。比如 Web MVC 模块、AspectJ代理模块、Caching（缓存）模块、JMX（Java 管理扩展）模块、Async（异步处理）模块等。

举例:@EnableWebmvc、@EnableAutoConfiguration等
实现:注解方式、编程方式

@Enable 注解模块举例

框架实现	`@Enable` 注解模块	激活模块
Spring Framework	`@EnableWebMvc`	Web
	`@EnableTransactionManagement`	事务管理模块
	`@EnableCaching`	Caching 模块
	`@EnableMBeanExport`	JMX 模块
	`@EnableAsync`	异步处理模块
	`EnableWebFlux`	Web Flux 模块
	`@EnableAspectJAutoProxy`	AspectJ 代理模块

Spring Boot	`@EnableAutoConfiguration`	自动装配模块
	`@EnableManagementContext`	Actuator 管理模块
	`@EnableConfigurationProperties`	配置属性绑定模块
	`@EnableOAuth2Sso`	OAuth2 单点登录模块

Spring Cloud	`@EnableEurekaServer`	Eureka服务器模块
	`@EnableConfigServer`	配置服务器模块
	`@EnableFeignClients`	Feign客户端模块
	`@EnableZuulProxy`	服务网关
	`@EnableCircuitBreaker`	服务熔断模块

实现方式

注解驱动方式

Spring源码，使用@Import注入配置Bean

package org.springframework.web.servlet.config.annotation;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Import(DelegatingWebMvcConfiguration.class)
public @interface EnableWebMvc {}

接口编程方式

Spring源码——激活缓存注解

package org.springframework.cache.annotation;

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Import(CachingConfigurationSelector.class)
public @interface EnableCaching {...}

CachingConfigurationSelector类图

ImportSelector接口源码，该接口在3.1引入，目的在于增加灵活度

selectImports方法返回数组，即可以死是多个配置Bean

package org.springframework.context.annotation;

/**
 * @since 3.1
 */
public interface ImportSelector {
 
 String[] selectImports(AnnotationMetadata importingClassMetadata);
}

@Enable注解方式自定义实现

激活HelloWorld模块，使用@Import(HelloWorldConfiguration.class)直接注入HelloWorldConfiguration模块

package top.fjy8018.springboot.annotation;

import org.springframework.context.annotation.Import;
import top.fjy8018.springboot.configuration.HelloWorldConfiguration;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:35
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Import(HelloWorldConfiguration.class)
public @interface EnableHelloWorld {}

目标类，其注解@Configuration在Spring3.0时引入

package top.fjy8018.springboot.configuration;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:31
 */
@Configuration
public class HelloWorldConfiguration {

 /**
 * 方法名称即Bean名称
 * @return Hello World!
 */
 @Bean
 public String helloWorld() {
 return "Hello world!";
 }
}

package org.springframework.context.annotation;

/**
 * @since 3.0
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Component
public @interface Configuration {
 @AliasFor(annotation = Component.class)
 String value() default "";
}

@Enable接口编程方式自定义实现

参考Spring包名路径创建自定义Selector，通过HelloWorldImportSelector实现HelloWorldConfiguration的配置Bean导入

package top.fjy8018.springboot.annotation;

import org.springframework.context.annotation.ImportSelector;
import org.springframework.core.type.AnnotationMetadata;
import top.fjy8018.springboot.configuration.HelloWorldConfiguration;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:28
 */
public class HelloWorldImportSelector implements ImportSelector {

 /**
 * @param importingClassMetadata 元数据信息
 * @return 一个或多个配置Bean
 */
 @Override
 public String[] selectImports(AnnotationMetadata importingClassMetadata) {
 return new String[]{HelloWorldConfiguration.class.getName()};
 }
}

激活模块方式变化，使用@Import(HelloWorldImportSelector.class)注入HelloWorldImportSelector模块，在Selector里加载并返回 HelloWorldConfiguration 模块

package top.fjy8018.springboot.annotation;

import org.springframework.context.annotation.Import;
import top.fjy8018.springboot.configuration.HelloWorldConfiguration;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:35
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
//@Import(HelloWorldConfiguration.class)
@Import(HelloWorldImportSelector.class)
public @interface EnableHelloWorld {
}

实现逻辑

graph LR
A(EnableHelloWorld) --> B(HelloWorldImportSelector) 
 B-->C(HelloWorldConfiguration)
 C --> D(helloWorld)

此方法比直接注解驱动更有弹性，可增加switch条件返回不同的模块，源码样例CachingConfigurationSelector

package org.springframework.cache.annotation;
public class CachingConfigurationSelector extends AdviceModeImportSelector<EnableCaching> {
 
 @Override
 public String[] selectImports(AdviceMode adviceMode) {
 switch (adviceMode) {
 case PROXY:
 return getProxyImports();
 case ASPECTJ:
 return getAspectJImports();
 default:
 return null;
 }
 }
 ...
}

Spring条件装配

从 Spring Framework 3.1 开始，允许在 Bean 装配时增加前置条件判断

Spring 注解	场景说明	起始版本
`@Profile`	配置化条件装配	3.1
`@Conditional`	编程条件装配	4.0

条件注解实现方式

配置方式 - `@Profile`

在spring4后实现方式发生变化，依赖@Conditional和ProfileCondition实现

package org.springframework.context.annotation;

/**
* @since 3.1
*/
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(ProfileCondition.class)
public @interface Profile {...}

package org.springframework.context.annotation;

/**
 * @since 4.0
 */
class ProfileCondition implements Condition {...}

通过上下文对象和注解元信息判断

package org.springframework.context.annotation;

/**
 * @since 4.0
 */
@FunctionalInterface
public interface Condition {
 boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata);
}

编程方式 - `@Conditional`

ConditionalOnClass用于判断标识类在当前classpath下是否存在，该注解使用@Conditional搭配OnClassCondition实现

package org.springframework.boot.autoconfigure.condition;

/**
 * {@link Conditional} that only matches when the specified classes are on the classpath.
 */
@Target({ ElementType.TYPE, ElementType.METHOD })
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(OnClassCondition.class)
public @interface ConditionalOnClass {...}

package org.springframework.boot.autoconfigure.condition;

/**
 * {@link Condition} that checks if properties are defined in environment.
 * @since 1.1.0
 * @see ConditionalOnProperty
 */
@Order(Ordered.HIGHEST_PRECEDENCE + 40)
class OnPropertyCondition extends SpringBootCondition {
 @Override
 public ConditionOutcome getMatchOutcome(ConditionContext context,
 AnnotatedTypeMetadata metadata) {
 // 获取元信息
 List<AnnotationAttributes> allAnnotationAttributes = annotationAttributesFromMultiValueMap(
 metadata.getAllAnnotationAttributes(
 ConditionalOnProperty.class.getName()));
 List<ConditionMessage> noMatch = new ArrayList<>();
 List<ConditionMessage> match = new ArrayList<>();
 for (AnnotationAttributes annotationAttributes : allAnnotationAttributes) {
 ConditionOutcome outcome = determineOutcome(annotationAttributes,
 context.getEnvironment());
 (outcome.isMatch() ? match : noMatch).add(outcome.getConditionMessage());
 }
 if (!noMatch.isEmpty()) {
 return ConditionOutcome.noMatch(ConditionMessage.of(noMatch));
 }
 return ConditionOutcome.match(ConditionMessage.of(match));
 }
 ...
}

OnPropertyCondition为抽象类，实际仍然是Condition接口的实现

package org.springframework.boot.autoconfigure.condition;

public abstract class SpringBootCondition implements Condition {...}

ConditionalOnProperty用于判断配置在上下文中是否存在

package org.springframework.boot.autoconfigure.condition;

/**
 * @since 1.1.0
 */
@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.TYPE, ElementType.METHOD })
@Documented
@Conditional(OnPropertyCondition.class)
public @interface ConditionalOnProperty {...}

自定义条件装配

目标服务

package top.fjy8018.springboot.service;

/**
 * 计算模块
 * @author F嘉阳
 * @date 2018-09-09 11:56
 */
public interface CalculateService {

 /**
 * 多整数求和
 * @param values 多个整数
 * @return 累加值
 */
 Integer sum(Integer... values);
}

基于配置方式

服务实现类，分别使用 for循环实现和Lambda实现，均标注@Service作为Bean自动注入

指定@Profile("Java7")，当条件为Java7时才装配

package top.fjy8018.springboot.service.impl;

import org.springframework.context.annotation.Profile;
import org.springframework.stereotype.Service;
import top.fjy8018.springboot.service.CalculateService;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:57
 */
@Profile("Java7")
@Service
public class Java7CalculateServiceImpl implements CalculateService {

 /**
 * 多整数求和
 *
 * @param values 多个整数
 * @return 累加值
 */
 @Override
 public Integer sum(Integer... values) {
 System.out.println("计算模块java7 for循环实现");
 int sum = 0;
 for (int i = 0; i < values.length; i++) {
 sum += values[i];
 }
 return sum;
 }
}

指定@Profile("Java8")，当条件为Java7时才装配

package top.fjy8018.springboot.service.impl;

import org.springframework.context.annotation.Profile;
import org.springframework.stereotype.Service;
import top.fjy8018.springboot.service.CalculateService;

import java.util.stream.Stream;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:57
 */
@Profile("Java8")
@Service
public class Java8CalculateServiceImpl implements CalculateService {

 /**
 * 多整数求和
 * @param values 多个整数
 * @return 累加值
 */
 @Override
 public Integer sum(Integer... values) {
 System.out.println("计算模块java8 Lambda实现");
 return Stream.of(values).reduce(0, Integer::sum);
 }
}

启动类，通过scanBasePackages指定报名扫描，该方法与@ComponentScan(basePackages = "xxx")效果一致

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import top.fjy8018.springboot.service.CalculateService;

/**
 * @author F嘉阳
 * @date 2018-09-09 12:06
 */
@SpringBootApplication(scanBasePackages = "top.fjy8018.springboot.service")
public class CalculateServiceDemo {

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(CalculateServiceDemo.class)
 .profiles("Java8")
 .web(WebApplicationType.NONE)
 .run(args);

 // 直接用类获取bean
 CalculateService calculateService = applicationContext.getBean(CalculateService.class);

 // calculateService Bean 是否存在，若不添加Profile则必有报错
 Integer[] values = new Integer[10];
 for (int i = 1; i <= values.length; i++) {
 values[i - 1] = i;
 }
 System.out.println("calculateService Bean 求和: " + calculateService.sum(values));

 // 关闭上下文
 applicationContext.close();
 }
}

执行结果——java8

执行结果——java7

若不指定配置则报错

基于编程方式

系统属性条件判断实现类

package top.fjy8018.springboot.condition;

import org.springframework.context.annotation.Condition;
import org.springframework.context.annotation.ConditionContext;
import org.springframework.core.type.AnnotatedTypeMetadata;

import java.util.Map;

/**
 * @author F嘉阳
 * @date 2018-09-09 17:15
 */
public class OnSystemPropertyCondition implements Condition {

 /**
 * 条件匹配规则，此处可实现复杂逻辑判断
 * @param context 上下文
 * @param metadata 元信息
 * @return 是否匹配
 */
 @Override
 public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {

 // 通过类名称取元信息
 Map<String, Object> annotationAttributes = metadata.getAnnotationAttributes(ConditionalOnSystemProperty.class.getName());

 String propertyName = String.valueOf(annotationAttributes.get("name"));
 String propertyValue = String.valueOf(annotationAttributes.get("value"));

 // 取系统属性
 String systemPropertyValue = System.getProperty(propertyName);

 return propertyValue.equals(systemPropertyValue);
 }

Java 系统属性条件判断类装配，系统属性通过Java API获取java.lang.System#getProperty(java.lang.String)方法获取的属性值

package top.fjy8018.springboot.condition;

import org.springframework.context.annotation.Conditional;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 17:14
 */
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(OnSystemPropertyCondition.class)
public @interface ConditionalOnSystemProperty {

 /**
 * Java 系统属性名称
 * @return 系统属性名称
 */
 String name();

 /**
 * 系统属性值
 * @return 属性值
 */
 String value();
}

启动类，helloWorld仅在系统条件满足时自动装配，注解若作用于类上语义相同

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.Bean;
import top.fjy8018.springboot.condition.ConditionalOnSystemProperty;

/**
 * 系统属性条件引导类
 *
 * @author F嘉阳
 * @date 2018-09-09 17:26
 */
public class SystemPropertyConditionBootstrap {

 @Bean
 @ConditionalOnSystemProperty(name = "user.name", value = "11860")
 public String helloWorld() {
 return "Hello SpringBoot2.0";
 }

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(SystemPropertyConditionBootstrap.class)
 .web(WebApplicationType.NONE)
 .run(args);

 // 通过类型和方法名获取Bean
 String helloWorldBean = applicationContext.getBean("helloWorld", String.class);
 // 若系统参数条件不匹配则报找不到Bean错误，以此实现编程方式条件加载
 System.out.println("ConditionalOnSystemProperty 加载内容：" + helloWorldBean);

 applicationContext.close();
 }
}

执行结果

条件不满足则报错

Spring Boot 自动装配

在 Spring Boot 场景下，基于规约大于配置的原则，实现 Spring 组件自动装配的目的（0配置）。

装配: 模式注解、@Eηable模块、条件装配、工厂加载机制
实现: 激活自动装配、实现自动装配、配置自动装配实现

底层装配技术

Spring 模式注解装配
Spring @Enable 模块装配
Spring 条件装配装配
Spring 工厂加载机制
- 实现类： SpringFactoriesLoader
- 配置资源： META-INF/spring.factories

package org.springframework.core.io.support;

/**
 * @since 3.2
 */
public abstract class SpringFactoriesLoader {

 public static final String FACTORIES_RESOURCE_LOCATION = "META-INF/spring.factories";
 
 public static <T> List<T> loadFactories(Class<T> factoryClass, @Nullable ClassLoader classLoader) {
 Assert.notNull(factoryClass, "'factoryClass' must not be null");
 // 获取类加载器
 ClassLoader classLoaderToUse = classLoader;
 if (classLoaderToUse == null) {
 classLoaderToUse = SpringFactoriesLoader.class.getClassLoader();
 }
 List<String> factoryNames = loadFactoryNames(factoryClass, classLoaderToUse);
 if (logger.isTraceEnabled()) {
 logger.trace("Loaded [" + factoryClass.getName() + "] names: " + factoryNames);
 }
 List<T> result = new ArrayList<>(factoryNames.size());
 // 迭代装配
 for (String factoryName : factoryNames) {
 result.add(instantiateFactory(factoryName, factoryClass, classLoaderToUse));
 }
 AnnotationAwareOrderComparator.sort(result);
 return result;
 }
 ...
}

自动装配举例

参考 META-INF/spring.factories

实现方法

激活自动装配 - @EnableAutoConfiguration
实现自动装配 - XXXAutoConfiguration
配置自动装配实现 - META-INF/spring.factories

目的在于使用META-INF/spring.factories文件配置替代@ComponentScan手动装配注解，从而实现自动装配

而Spring大多数自动装配都由starter模块负责

自定义自动装配

设计逻辑

HelloWorldAutoConfiguration

条件判断： user.name == "Mercy"
模式注解： @Configuration
@Enable 模块：流程图

引导类

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;

/**
 * @author F嘉阳
 * @date 2018-09-10 17:13
 */
@EnableAutoConfiguration
public class HelloWorldAutoConfigurationBootstrap {

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(HelloWorldAutoConfigurationBootstrap.class)
 .web(WebApplicationType.NONE)
 .run(args);

 // 通过bean名称获取helloWorld Bean
 String helloWorldBean = applicationContext.getBean("helloWorld", String.class);

 // helloWorld Bean 是否存在
 System.out.println("helloWorld Bean 内容: " + helloWorldBean);

 // 关闭上下文
 applicationContext.close();
 }
}

配置Bean，使用模式注解标注 @Configuration，该类实际上实现了条件装配、模式注解、自动装配

package top.fjy8018.springboot.configuration;

import org.springframework.context.annotation.Configuration;
import top.fjy8018.springboot.annotation.EnableHelloWorld;
import top.fjy8018.springboot.condition.ConditionalOnSystemProperty;

/**
 * @author F嘉阳
 * @date 2018-09-10 17:23
 */
@Configuration
@EnableHelloWorld
@ConditionalOnSystemProperty(name = "user.name", value = "11860")
public class HelloWorldAutoConfiguration {}

创建spring.factories，每行全类名顶格开始，防止在某些版本下产生不稳定的BUG

# 自动装配
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
top.fjy8018.springboot.configuration.HelloWorldAutoConfiguration

执行结果与之前一致

SpringBoot源码分析1——总览

Thu, 10 Dec 2020 17:11:00 +0800

SpringBoot 2.0 特点

组件自动装配

规约大于配置,专注核心业务

模式注解、@ Enable模块、条件装配、加载机制

外部化配置

一次构建、按需调配,到处运行

Environment抽象、生命周期、破坏性变更

嵌入式容器

内置容器、无需部署、独立运行

Servlet Web容器、 Reactive Web容器

Spring Boot Starter

简化依赖、按需装配、自我包含

依赖管理、装配条件、装配顺序

Production-Ready

一站式运维、生态无缝整合

健康检査、数据指标、@ Endpoint管控

SpringBoot 与Java EE规范

Web: Servlet( JSR-315, JSR-340）
SQL: JDBC(SR-221)
数据校验: Bean validation(JSR303、JSR-349)
缓存: Java Caching API(JSR-107)
Web Sockets: Java api for Web Socket(JSR-356
Web services: JAX-WS( JSR-224）
Java管理:JMX(JSR3)
消息:JMS(JSR-914)

核心特性介绍

三大特性

组件自动装配: Web mvc、 Web flux、JDBC等
嵌入式Web容器: Tomcat、 Jetty以及 Undertow
生产准备特性:指标、健康检査、外部化配置等

组件自动装配

激活： @EnableAutoConfiguration 配置： /META-INF/spring.factories 实质为key-value形式实现： XXXAutoConfiguration

官方文档样例

import org.springframework.boot.*;
import org.springframework.boot.autoconfigure.*;
import org.springframework.web.bind.annotation.*;

@RestController
@EnableAutoConfiguration
public class Example {

 @RequestMapping("/")
 String home() {
 return "Hello World!";
 }

 public static void main(String[] args) throws Exception {
 SpringApplication.run(Example.class, args);
 }

}

实际上@SpringBootApplication已经包含该注解

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@SpringBootConfiguration
@EnableAutoConfiguration
@ComponentScan(excludeFilters = {
 @Filter(type = FilterType.CUSTOM, classes = TypeExcludeFilter.class),
 @Filter(type = FilterType.CUSTOM, classes = AutoConfigurationExcludeFilter.class) })
public @interface SpringBootApplication {
}

在IDEA中使用ctrl+shift+n可查看Spring源码中包含大量spring.factories文件

样例文件：

org/springframework/boot/spring-boot-autoconfigure/2.0.4.RELEASE/spring-boot-autoconfigure-2.0.4.RELEASE.jar!/META-INF/spring.factories

# Initializers
org.springframework.context.ApplicationContextInitializer=\
org.springframework.boot.autoconfigure.SharedMetadataReaderFactoryContextInitializer,\
org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener

# Application Listeners
org.springframework.context.ApplicationListener=\
org.springframework.boot.autoconfigure.BackgroundPreinitializer

# Auto Configuration Import Listeners
org.springframework.boot.autoconfigure.AutoConfigurationImportListener=\
org.springframework.boot.autoconfigure.condition.ConditionEvaluationReportAutoConfigurationImportListener

# Auto Configuration Import Filters
org.springframework.boot.autoconfigure.AutoConfigurationImportFilter=\
org.springframework.boot.autoconfigure.condition.OnClassCondition

# 自动装配配置，自动装载Bean列表
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
org.springframework.boot.autoconfigure.admin.SpringApplicationAdminJmxAutoConfiguration,\
org.springframework.boot.autoconfigure.web.servlet.WebMvcAutoConfiguration,\
...

WebMvcAutoConfiguration源码，其包含大量条件装配，若未检测到该类的依赖，则不会被装配

@Configuration
@ConditionalOnWebApplication(type = Type.SERVLET)
@ConditionalOnClass({ Servlet.class, DispatcherServlet.class, WebMvcConfigurer.class })
@ConditionalOnMissingBean(WebMvcConfigurationSupport.class)
@AutoConfigureOrder(Ordered.HIGHEST_PRECEDENCE + 10)
@AutoConfigureAfter({ DispatcherServletAutoConfiguration.class,
 ValidationAutoConfiguration.class })
public class WebMvcAutoConfiguration {
...

嵌入式Web容器

Web servlet: Tomcat、 Jetty和 Undertow
Web reactive Netty Web Server

生产准备特性

指标:/ actuator/ metrics
健康检查:/ actuator/ health
外部化配置:/ actuator/ configprops

Web应用介绍

传统 Servlet应用概览

Servlet组件: Servlet、 Filter、 Listener
Servlet注册: Servlet注解、 Spring Bean、 RegistrationBean
异步非阻塞:异步 Servlet、非阻塞 Servlet

传统 Servelt 应用

加入springboot应用通用依赖前缀多为spring-boot-starter-xxx

加入Web依赖

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
</dependency>

Maven依赖变化

由此可看出默认的Web容器为Tomcat（内嵌Servlet容器）

Servlet 组件

Servlet
- 实现
  - @WebServlet
  - HttpServlet
  - 注册
- URL 映射
  - @WebServlet(urlPatterns = "/my/servlet")
- 注册 @ServletComponentScan(basePackages = "top.fjy8018.springboot.web.servlet")
Filter
Listener

Servlet 注册

Servlet 注解

@ServletComponentScan +
- @WebServlet
- @WebFilter
- @WebListener

Spring Bean

@Bean +
- Servlet
- Filter
- Listener
RegistrationBean
- ServletRegistrationBean
- FilterRegistrationBean
- ServletListenerRegistrationBean

Servlet 实现

参考源码包位置，由于出现了Reactive Web容器，SpringBoot 1.x的包路径与2.x有所不同，2.x包路径变更为org.springframework.boot.autoconfigure.web.servlet

HttpServlet

package top.fjy8018.springboot.web.servlet;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

/**
 * 结合servlet规范注册
 *
 * @author F嘉阳
 * @date 2018-09-08 19:17
 */
@WebServlet(urlPatterns = "/myservlet")
public class MyServlet extends HttpServlet {

 @Override
 protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
 resp.getWriter().print("hello world");
 }
}

Servlet 3.0规范要求必须实现HttpServlet
其内部提供多种方法实现（ctrl+F12）
urlPatterns = "/myservlet"指明浏览器映射

注册

在主类中注册（包扫描）

package top.fjy8018.springboot;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.servlet.ServletComponentScan;

@SpringBootApplication
@ServletComponentScan(basePackages = "top.fjy8018.springboot.web.servlet")
public class OverviewApplication {

 public static void main(String[] args) {
 SpringApplication.run(OverviewApplication.class, args);
 }
}

启动后从控制台可看出映射已经成功

异步非阻塞 Servlet

异步 Servlet

javax.servlet.ServletRequest#startAsync()
javax.servlet.AsyncContext

非阻塞 Servlet

javax.servlet.ServletInputStream#setReadListener
- javax.servlet.ReadListener
javax.servlet.ServletOutputStream#setWriteListener
- javax.servlet.WriteListener

异步非阻塞Servlet实现

package top.fjy8018.springboot.web.servlet;

import javax.servlet.AsyncContext;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

/**
 * 结合servlet规范注册
 * asyncSupported指定异步servlet，默认false
 *
 * @author F嘉阳
 * @date 2018-09-08 19:17
 */
@WebServlet(urlPatterns = "/myservlet", asyncSupported = true)
public class MyServlet extends HttpServlet {

 @Override
 protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
 AsyncContext asyncContext = req.startAsync();

 // 执行任务
 asyncContext.start(() -> {
 try {
 resp.getWriter().print("hello world");

 // 触发完成，必须步骤，否则客户端发生超时
 asyncContext.complete();
 } catch (IOException e) {
 e.printStackTrace();
 }
 });
 }
}

asyncSupported = true 启用异步支持，否则报500错误

Spring Web MVC 应用介绍

Web mvo视图:模板引擎、内容协商、异常处理等
Web mVc rest:资源服务、资源跨域、服务发现等

Web MVC 视图

ViewResolver
View

源码

package org.springframework.web.servlet;

import java.util.Locale;
import org.springframework.lang.Nullable;

public interface ViewResolver {
 @Nullable
 View resolveViewName(String viewName, Locale locale) throws Exception;

}

通过指定名称和路径加载视图

视图渲染View

package org.springframework.web.servlet;

public interface View {
 ...

 void render(@Nullable Map<String, ?> model, HttpServletRequest request, HttpServletResponse response)
 throws Exception;

}

提供接口供不同的模板引擎实现ViewResolver

模板引擎

Thymeleaf
Freemarker
JSP

若同时有多种模板引擎，则需要内容协商

内容协商

ContentNegotiationConfigurer
ContentNegotiationStrategy
ContentNegotiatingViewResolver

异常处理

@ExceptionHandler
HandlerExceptionResolver
- ExceptionHandlerExceptionResolver
BasicErrorController (Spring Boot默认白页404)

源码

@Controller
@RequestMapping("${server.error.path:${error.path:/error}}")
public class BasicErrorController extends AbstractErrorController {
 
 @RequestMapping(produces = "text/html")
 public ModelAndView errorHtml(HttpServletRequest request,
 HttpServletResponse response) {
 HttpStatus status = getStatus(request);
 Map<String, Object> model = Collections.unmodifiableMap(getErrorAttributes(
 request, isIncludeStackTrace(request, MediaType.TEXT_HTML)));
 response.setStatus(status.value());
 ModelAndView modelAndView = resolveErrorView(request, response, status, model);
 return (modelAndView != null) ? modelAndView : new ModelAndView("error", model);
 }
 
 @RequestMapping
 @ResponseBody
 public ResponseEntity<Map<String, Object>> error(HttpServletRequest request) {
 Map<String, Object> body = getErrorAttributes(request,
 isIncludeStackTrace(request, MediaType.ALL));
 HttpStatus status = getStatus(request);
 return new ResponseEntity<>(body, status);
 }
}

其根据浏览器类型提供两种方式渲染，分别为html和json

{
 "timestamp": "2019-03-13T04:47:18.445+0000",
 "status": 404,
 "error": "Not Found",
 "message": "Not Found",
 "path": "/hello"
}

Web MVC REST

资源服务

@RequestMapping
@GetMapping
@ResponseBody
@RequestBody

从源码分析可知@GetMapping是从@RequestMapping变化而来，通过委派（别名）方式实现，并且在SpringMVC 4.3后引入

package org.springframework.web.bind.annotation;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import org.springframework.core.annotation.AliasFor;

/**
 * Annotation for mapping HTTP {@code GET} requests onto specific handler
 * methods.
 *
 * <p>Specifically, {@code @GetMapping} is a <em>composed annotation</em> that
 * acts as a shortcut for {@code @RequestMapping(method = RequestMethod.GET)}.
 *
 *
 * @author Sam Brannen
 * @since 4.3
 * @see PostMapping
 * @see PutMapping
 * @see DeleteMapping
 * @see PatchMapping
 * @see RequestMapping
 */
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@RequestMapping(method = RequestMethod.GET)
public @interface GetMapping {
 /**
 * 委派目标 {@link RequestMapping#name}.
 */
 @AliasFor(annotation = RequestMapping.class)
 String name() default "";
 ...
}

资源跨域

CrossOrigin
WebMvcConfigurer#addCorsMappings
传统解决方案
- IFrame
- JSONP

源码

package org.springframework.web.servlet.config.annotation;

/**
 * @since 3.1
 */
public interface WebMvcConfigurer {
 /**
 * 跨域配置
 * @since 4.2
 */
 default void addCorsMappings(CorsRegistry registry) {
 }
}

注解实现方式

package org.springframework.web.bind.annotation;

/**
 * @since 4.2
 */
@Target({ ElementType.METHOD, ElementType.TYPE })
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface CrossOrigin {...}

两者均来自springframework，属于不同的实现维度（接口编程 / 注解驱动）

服务发现

HATEOS

核心组件

DispatcherServlet——Servlet实现请求转发到映射URL
HandlerMapping——处理映射URL方法
HandlerAdapter——方法处理
ViewResolver——交由视图处理
…

Spring WebFlux （Spring 5+）

Reactor 基础

Java Lambda —— Reactive Stream实现（JDK 9 flow api、RxJava）
Mono
Flux

Web Flux 核心

Web MVC 注解兼容

@Controller
@RequestMapping
@ResponseBody
@RequestBody
…

函数式声明

RouterFunction

异步非阻塞

Servlet 3.1 +
Netty Reactor

使用场景

页面渲染
REST 应用

性能测试

国外博客

结论：Webflux 性能不会显著提升，但Webflux提升吞吐量

Web Server 应用

切换 Web server
自定义 Servlet Web server
自定义 Reactive Web server

切换 Web Server

切换其他 Servlet 容器

Tomcat -> Jetty

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 <exclusions>
 <!-- tomcat加载优先级高于jetty，若不加排除，则仍然加载tomcat -->
 <exclusion>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-tomcat</artifactId>
 </exclusion>
 </exclusions>
</dependency>
<!-- 使用 Jetty 容器 -->
<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-jetty</artifactId>
</dependency>

替换 Servlet 容器

WebFlux

<!--使用webflux必须注释web依赖，当web和flux共存时，web优先级更高-->
<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

注释后相关类和的注解也已经失效，需要删除

@SpringBootApplication
//@ServletComponentScan(basePackages = "top.fjy8018.springboot.web.servlet")
public class SpringbootApplication {

 public static void main(String[] args) {
 SpringApplication.run(SpringbootApplication.class, args);
 }
}

依赖变化

启动后变为Netty容器

自定义 Servlet Web Server

WebServerFactoryCustomizer

SpringBoot 2.0 新增支持接口

package org.springframework.boot.web.server;

/**
 * @since 2.0.0
 * @see WebServerFactoryCustomizerBeanPostProcessor
 */
@FunctionalInterface
public interface WebServerFactoryCustomizer<T extends WebServerFactory> {
 
 void customize(T factory);
}

该接口为主接口，其下有大量实现，IDEA通过ctrl+alt+B或者ctrl+alt+左键点击查看

其中TomcatServletWebServerFactoryCustomizer就是实现类之一，但在启用Webflux后该类无效

package org.springframework.boot.autoconfigure.web.servlet;

/**
 * {@link WebServerFactoryCustomizer} to apply {@link ServerProperties} to Tomcat web
 * servers.
 *
 * @author Brian Clozel
 * @author Phillip Webb
 * @since 2.0.0
 */
public class TomcatServletWebServerFactoryCustomizer
 implements WebServerFactoryCustomizer<TomcatServletWebServerFactory>, Ordered {
 @Override
 public void customize(TomcatServletWebServerFactory factory) {
 ServerProperties.Tomcat tomcatProperties = this.serverProperties.getTomcat();
 if (!ObjectUtils.isEmpty(tomcatProperties.getAdditionalTldSkipPatterns())) {
 factory.getTldSkipPatterns()
 .addAll(tomcatProperties.getAdditionalTldSkipPatterns());
 }
 if (tomcatProperties.getRedirectContextRoot() != null) {
 customizeRedirectContextRoot(factory,
 tomcatProperties.getRedirectContextRoot());
 }
 if (tomcatProperties.getUseRelativeRedirects() != null) {
 customizeUseRelativeRedirects(factory,
 tomcatProperties.getUseRelativeRedirects());
 }
 }
 ...
}

自定义 Reactive Web Server

ReactiveWebServerFactoryCustomizer

其源码显示提供了最基础的实现，包括端口、SSL等，允许自行扩展实现

package org.springframework.boot.autoconfigure.web.reactive;

/**
 * {@link WebServerFactoryCustomizer} to apply {@link ServerProperties} to reactive
 * servers.
 * @since 2.0.0
 */
public class ReactiveWebServerFactoryCustomizer implements
 WebServerFactoryCustomizer<ConfigurableReactiveWebServerFactory>, Ordered {

 private final ServerProperties serverProperties;

 public ReactiveWebServerFactoryCustomizer(ServerProperties serverProperties) {
 this.serverProperties = serverProperties;
 }

 @Override
 public int getOrder() {
 return 0;
 }

 @Override
 public void customize(ConfigurableReactiveWebServerFactory factory) {
 PropertyMapper map = PropertyMapper.get().alwaysApplyingWhenNonNull();
 map.from(this.serverProperties::getPort).to(factory::setPort);
 map.from(this.serverProperties::getAddress).to(factory::setAddress);
 map.from(this.serverProperties::getSsl).to(factory::setSsl);
 map.from(this.serverProperties::getCompression).to(factory::setCompression);
 map.from(this.serverProperties::getHttp2).to(factory::setHttp2);
 }
}

数据相关介绍

关系型数据

JDBC:数据源、 JdbcTemplate、自动装配
JPA:实体映射关系、实体操作、自动装配
事务∶ Spring事务抽象、JDBC事务处理、自动装配

JDBC

依赖

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>

依赖变化，在SpringBoot2.0 后依赖新的连接池 HikariCP

数据源

javax.sql.DataSource
JdbcTemplate
自动装配
DataSourceAutoConfiguration

DataSource

连接池多数实现是对DataSource的二次封装，提高执行效率

package javax.sql;

/**
 * @since 1.4
 */

public interface DataSource extends CommonDataSource, Wrapper {
 Connection getConnection() throws SQLException;
 
 Connection getConnection(String username, String password)
 throws SQLException;
}

自动装配

JdbcTemplate自动装配，其依赖数据源配置项，故有@AutoConfigureAfter注解标识，该类执行于DataSourceAutoConfiguration之后，额外还有多个条件装配的标识

package org.springframework.boot.autoconfigure.jdbc;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for {@link JdbcTemplate} and
 * {@link NamedParameterJdbcTemplate}.
 * @since 1.4.0
 */
@Configuration
@ConditionalOnClass({ DataSource.class, JdbcTemplate.class })
@ConditionalOnSingleCandidate(DataSource.class)
@AutoConfigureAfter(DataSourceAutoConfiguration.class)
@EnableConfigurationProperties(JdbcProperties.class)
public class JdbcTemplateAutoConfiguration {...}

DataSourceAutoConfiguration

package org.springframework.boot.autoconfigure.jdbc;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for {@link DataSource}.
 */
@Configuration
@ConditionalOnClass({ DataSource.class, EmbeddedDatabaseType.class })
@EnableConfigurationProperties(DataSourceProperties.class)
@Import({ DataSourcePoolMetadataProvidersConfiguration.class,
 DataSourceInitializationConfiguration.class })
public class DataSourceAutoConfiguration {...}

该类属于自动装配，故必然在spring.factories中，而数据源下即JdbcTemplateAutoConfiguration

此时直接启动会得到缺少数据源的错误报告

***************************
APPLICATION FAILED TO START
***************************

Description:

Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.

Reason: Failed to determine a suitable driver class


Action:

Consider the following:
 If you want an embedded database (H2, HSQL or Derby), please put it on the classpath.
 If you have database settings to be loaded from a particular profile you may need to activate it (no profiles are currently active).

JPA

依赖

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>

其属于SpringData项目，基于hibernate实现，同时也自动依赖JDBC

实体映射关系

@javax.persistence.OneToOne
@javax.persistence.OneToMany
@javax.persistence.ManyToOne
@javax.persistence.ManyToMany
…

实体操作

javax.persistence.EntityManager

属于Java类库的API

package javax.persistence;

public interface EntityManager {

 /**
 * 存
 */
 public void persist(Object entity);

 /**
 * 更新
 */
 public <T> T merge(T entity);

 /**
 * 删除
 */
 public void remove(Object entity);

 /**
 * 查询以及多个重载方法
 */
 public <T> T find(Class<T> entityClass, Object primaryKey);

 public <T> T find(Class<T> entityClass, Object primaryKey,
 Map<String, Object> properties);
 ...
}

自动装配

HibernateJpaAutoConfiguration

该类也同属于自动装配，且需要JDBC提供数据源，故其加载顺序标识为DataSourceAutoConfiguration之后

package org.springframework.boot.autoconfigure.orm.jpa;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for Hibernate JPA.
 */
@Configuration
@ConditionalOnClass({ LocalContainerEntityManagerFactoryBean.class, EntityManager.class })
@Conditional(HibernateEntityManagerCondition.class)
@EnableConfigurationProperties(JpaProperties.class)
@AutoConfigureAfter({ DataSourceAutoConfiguration.class })
@Import(HibernateJpaConfiguration.class)
public class HibernateJpaAutoConfiguration {...}

事务（Transaction）

依赖

<dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-tx</artifactId>
</dependency>

Spring 事务抽象

PlatformTransactionManager

该接口属于事务根接口，不仅限于数据库事务，默认实现包含分布式事务JtaTransactionManager和数据库事务DataSourceTransactionManager

package org.springframework.transaction;

import org.springframework.lang.Nullable;

/**
 * <p>The default implementations of this strategy interface are
 * {@link org.springframework.transaction.jta.JtaTransactionManager} and
 * {@link org.springframework.jdbc.datasource.DataSourceTransactionManager}

 * @see org.springframework.transaction.support.TransactionTemplate
 * @see org.springframework.transaction.interceptor.TransactionInterceptor
 * @see org.springframework.transaction.interceptor.TransactionProxyFactoryBean
 */
public interface PlatformTransactionManager {...}

JDBC 事务处理

DataSourceTransactionManager

内部通过IOC自动注入数据源接口

package org.springframework.jdbc.datasource;

@SuppressWarnings("serial")
public class DataSourceTransactionManager extends AbstractPlatformTransactionManager
 implements ResourceTransactionManager, InitializingBean {

 @Nullable
 private DataSource dataSource;
 ...
}

自动装配

TransactionAutoConfiguration

自动装配的条件是Spring事务依赖存在PlatformTransactionManager

package org.springframework.boot.autoconfigure.transaction;

/**
 * {@link org.springframework.boot.autoconfigure.EnableAutoConfiguration
 * Auto-configuration} for Spring transaction.
 * @since 1.3.0
 */
@Configuration
@ConditionalOnClass(PlatformTransactionManager.class)
@AutoConfigureAfter({ JtaAutoConfiguration.class, HibernateJpaAutoConfiguration.class,
 DataSourceTransactionManagerAutoConfiguration.class,
 Neo4jDataAutoConfiguration.class })
@EnableConfigurationProperties(TransactionProperties.class)
public class TransactionAutoConfiguration {...}

功能扩展介绍

Spring Boot 应用

Spring Application:失败分析、应用特性、事件监听等
Spring Boot配置:外部化配置、 Profile、配置属性
Spring Boot starter: Starter开发、最佳实践

SpringApplication

失败分析

FailureAnalysisReporter

该类在Spring Boot 1.4后开始支持

package org.springframework.boot.diagnostics;

/**
 * Reports a {@code FailureAnalysis} to the user.
 * @since 1.4.0
 */
@FunctionalInterface
public interface FailureAnalysisReporter {...}

应用特性

SpringApplication Fluent API

原始默认启动方法为SpringApplication.run(SpringbootApplication.class, args)其实际返回可配置上下文对象ConfigurableApplicationContext，可通过Builder模式重写

package top.fjy8018.springboot;

@SpringBootApplication
public class SpringbootApplication {

 public static void main(String[] args) {
 // Fluent API
 // 等价建造者设计模式调用方法
 new SpringApplicationBuilder(SpringbootApplication.class)
 // 不使用web模式
 .web(WebApplicationType.NONE)
 // 自定义参数
 .properties("name=fjy")
 .run(args);
 }
}

2.0 版本下提供3种WEB容器类型

package org.springframework.boot;

/**
 * @since 2.0.0
 */
public enum WebApplicationType {
 NONE,
 SERVLET,
 REACTIVE
}

Spring Boot 配置

外部化配置

ConfigurationProperty
@Profile

从SpringBoot 2.0 后引入，Origin类型支持其他配资源引入，非传统仅支持键值对，由此支持多数据源

package org.springframework.boot.context.properties.source;

/**
 * @since 2.0.0
 */
public final class ConfigurationProperty
 implements OriginProvider, Comparable<ConfigurationProperty> {
 
 private final ConfigurationPropertyName name;

 private final Object value;

 private final Origin origin;
 ...
}

官方文档

此处官方文档存在一定疏漏，部分场景配置无法实现

Spring Boot uses a very particular PropertySource order that is designed to allow sensible overriding of values. Properties are considered in the following order:

Devtools global settings properties on your home directory (~/.spring-boot-devtools.properties when devtools is active).

@TestPropertySource annotations on your tests.

@SpringBootTest#properties annotation attribute on your tests.

Command line arguments.

Properties from SPRING_APPLICATION_JSON (inline JSON embedded in an environment variable or system property).

ServletConfig init parameters.

ServletContext init parameters.

JNDI attributes from java:comp/env.

Java System properties (System.getProperties()).

OS environment variables.

A RandomValuePropertySource that has properties only in random.*.

Profile-specific application properties outside of your packaged jar (application-{profile}.properties and YAML variants).

Profile-specific application properties packaged inside your jar (application-{profile}.properties and YAML variants).

Application properties outside of your packaged jar (application.properties and YAML variants).

Application properties packaged inside your jar (application.properties and YAML variants).

@PropertySource annotations on your @Configuration classes.

Default properties (specified by setting SpringApplication.setDefaultProperties).

Profile支持以注解方式条件开启配置，用于区分开发环境和生产环境等

package org.springframework.context.annotation;

/**
 * @since 3.1
 */
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(ProfileCondition.class)
public @interface Profile {
 
 String[] value();
}

配置属性

PropertySources

提供两种编程方式：接口、注解，支持保存多个配置，按照17种顺序排列，该接口在SpringBoot中重新设计变为ConfigurationProperty

package org.springframework.core.env;

/**
 * @since 3.1
 * @see PropertySource
 */
public interface PropertySources extends Iterable<PropertySource<?>> {...}

运维管理介绍

官方文档

Spring Boot Actuator

端点（Endpoints）

Web Endpoints
JMX Endpoints

依赖

<!--运维监控-->
<dependency>
 <groupId>org.springframework.boot</groupId> 
 <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

启动效果

访问健康检查接口/actuator/health返回应用状态

{
 "status": "UP"
}

访问根路径

{
 "_links": {
 "self": {
 "href": "http://localhost:8080/actuator",
 "templated": false
 },
 "health": {
 "href": "http://localhost:8080/actuator/health",
 "templated": false
 },
 "info": {
 "href": "http://localhost:8080/actuator/info",
 "templated": false
 }
 }
}

依据SpringBoot打印的进程号或者依据应用名称使用Jconsole监控，该操作使用的为JMX端点

健康检查（Health Checks）

Health
HealthIndicator

使用JMX在默认情况下能获取比Web端点更多的信息，原因是JMX支持使用安全连接，具有更好的安全性

官方文档指出默认状态只暴露health和info，可通过配置文件调整

28.4 Actuator Security
For security purposes, all actuators other than /health and /info are disabled by default. The management.endpoints.web.exposure.include property can be used to enable the actuators.

在application.properties文件中加入配置重启应用

# 暴露所有健康监控点
management.endpoints.web.exposure.include=*

从日志输出可知增加了更多的URL映射

但尽管配置后，JMX显示的信息仍然比Web端更丰富，例如在JMX中不仅看到状态，还能看到磁盘、内存等信息，而Web端仅看到状态

指标（Metrics）

内建 Metrics
Web Endpoint : /actuator/metrics
自定义 Metrics

Web端点列举支持指标

{
 "names": [
 "jvm.memory.max",
 "jvm.gc.memory.promoted",
 "jvm.memory.used",
 "jvm.gc.max.data.size",
 "jvm.gc.pause",
 "jvm.memory.committed",
 "system.cpu.count",
 "logback.events",
 "jvm.buffer.memory.used",
 "jvm.threads.daemon",
 "system.cpu.usage",
 "jvm.gc.memory.allocated",
 "jvm.threads.live",
 "jvm.threads.peak",
 "process.uptime",
 "http.server.requests",
 "process.cpu.usage",
 "jvm.classes.loaded",
 "jvm.classes.unloaded",
 "jvm.gc.live.data.size",
 "jvm.buffer.count",
 "jvm.buffer.total.capacity",
 "process.start.time"
 ]
}

访问/actuator/metrics/jvm.memory.max可查看相关指标，转换后可知最大内存为5.2G

{
 "name": "jvm.memory.max",
 "description": "The maximum amount of memory in bytes that can be used for memory management",
 "baseUnit": "bytes",
 "measurements": [
 {
 "statistic": "VALUE",
 "value": 5595725823
 }
 ],
 "availableTags": [
 {
 "tag": "area",
 "values": [
 "heap",
 "nonheap"
 ]
 },
 {
 "tag": "id",
 "values": [
 "Compressed Class Space",
 "PS Survivor Space",
 "PS Old Gen",
 "Metaspace",
 "PS Eden Space",
 "Code Cache"
 ]
 }
 ]
}

设计模式学习——行为型

Wed, 22 Jul 2020 16:50:17 +0800

模板方法模式

介绍

定义:定义了一个算法的骨架,并允许子类为一个或多个步骤提供实现

模板方法使得子类可以在不改变算法结构的情况下,重新定义算法的某些步骤

类型：行为型

适用场景

一次性实现一个算法的不变的部分,并将可变的行为留给子类来实现
各子类中公共的行为被提取出来并集中到一个公共父类中, 从而避免代码重复

优点

提高复用性
提高扩展性
符合开闭原则

缺点

类数目增加
增加了系统实现的复杂度
继承关系自身缺点,如果父类添加新的抽象方法,所有子类都要改一遍

分类和扩展

钩子方法
- 提供了默认的行为，子类可以在必要时进行扩展

相关设计模式

模板方法模式和工厂方法模式
- 工厂方法是模板方法的一种特殊实现
模板方法模式和策略模式
- 策略模式在于实现算法可以互相替换而不影响应用层，不改变算法流程
- 模板方法定义了一个算法的骨架，不明确的交给子类实现，可以改变算法流程

代码样例

类图

抽象课程

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;
import org.apache.ibatis.mapping.MappedStatement;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.Collection;

/**
 * 抽象课程
 * <p>
 * {@link java.util.AbstractList} 提供抽象List实现
 * 例如 {@link java.util.AbstractList#addAll(int, Collection)} 定义通过模板
 * {@link java.util.AbstractMap} 同理
 * <p>
 * {@link javax.servlet.http.HttpServlet#doGet(HttpServletRequest, HttpServletResponse)} 也是模板定义
 * {@link org.apache.ibatis.executor.BaseExecutor#doUpdate(MappedStatement, Object)} 等方法交由子类实现
 * 例如 {@link org.apache.ibatis.executor.SimpleExecutor#doUpdate(MappedStatement, Object)}
 *
 * @author F嘉阳
 * @date 2020/3/4 10:59
 */
@Slf4j
public abstract class ACourse {

 /**
 * 固定不变的行为
 * 定义模板，不允许子类覆盖，防止影响流程执行顺序
 */
 protected final void makeCourse() {
 this.makePPT();
 this.makeVideo();
 // 调用可选方法
 if (needWriteArticle()) {
 this.writeArticle();
 }
 this.packageCourse();
 }

 /**
 * 固定不变的行为
 * 满足所有子类需求
 */
 final void makePPT() {
 log.info("制作PPT");
 }

 final void makeVideo() {
 log.info("制作视频");
 }

 /**
 * 可选行为
 * <p>
 * 手记可写可不写，但写的手记的流程还是一致的，故也声明为final
 * 通过定义钩子方法，允许子类覆盖
 */
 final void writeArticle() {
 log.info("编写手记");
 }

 /**
 * 钩子方法，提供默认实现
 *
 * @return
 */
 protected boolean needWriteArticle() {
 return false;
 }

 /**
 * 必须实现的行为
 * 课程内容打包
 */
 abstract void packageCourse();
}

具体课程实现

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;

/**
 * 具体课程实现
 *
 * @author F嘉阳
 * @date 2020/3/4 11:05
 */
@Slf4j
public class JavaCourse extends ACourse {
 /**
 * 必须实现的行为
 * 课程内容打包
 */
 @Override
 void packageCourse() {
 log.info("提供Java学习课程源码");
 }
}

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;

/**
 * 具体课程实现
 * @author F嘉阳
 * @date 2020/3/4 11:06
 */
@Slf4j
public class VueCourse extends ACourse {

 private boolean needWriteArticleFlag = false;

 /**
 * 通过构造器开放给实例决定
 * @param needWriteArticleFlag
 */
 public VueCourse(boolean needWriteArticleFlag) {
 this.needWriteArticleFlag = needWriteArticleFlag;
 }

 /**
 * 必须实现的行为
 * 课程内容打包
 */
 @Override
 void packageCourse() {
 log.info("提供Vue学习课程源码");
 log.info("提供Vue学习课程多媒体素材");
 }

 /**
 * 钩子方法，提供默认实现
 *
 * @return
 */
 @Override
 protected boolean needWriteArticle() {
 return this.needWriteArticleFlag;
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/4 11:08
 */
@Slf4j
class ACourseTest {

 @Test
 void makeJavaCourse() {
 ACourse javaCourse = new JavaCourse();
 javaCourse.makeCourse();
 }

 @Test
 void makeVueCourse() {
 ACourse vueCourse = new VueCourse(true);
 vueCourse.makeCourse();
 }
}

输出

11:27:56.942 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作PPT
11:27:56.957 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作视频
11:27:56.957 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.JavaCourse - 提供Java学习课程源码


11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作PPT
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作视频
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 编写手记
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.VueCourse - 提供Vue学习课程源码
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.VueCourse - 提供Vue学习课程多媒体素材

迭代器模式

介绍

定义提供一种方法,顺序访问一个集合对象中的各个元素,而又不暴露该对象的内部表示

类型：行为型

适用场景

访问一个集合对象的内容而无需暴露它的内部表示
为遍历不同的集合结构提供一个统一的接口

优点

分离了集合对象的遍历行为

缺点

类的个数成对增加

相关设计模式

迭代器模式和访问者模式
- 访问者模式扩展开放的部分，再作用于对象的操作上
- 迭代器模式扩展开放的部分，再作用于对象的种类上

代码样例

迭代器一般使用现成的，而不会自己实现

类图

课程接口

package top.fjy8018.designpattern.pattern.behavior.iterator;

import lombok.AllArgsConstructor;
import lombok.Getter;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:49
 */
@AllArgsConstructor
@Getter
public class Course {

 private String name;

}

迭代器

package top.fjy8018.designpattern.pattern.behavior.iterator;

import java.util.ArrayList;

/**
 * JDK迭代器
 * {@link java.util.Iterator}
 * 集合内部类实现 {@link ArrayList.Itr#hasNext()}
 * 同时做了进一步扩展 {@link ArrayList.ListItr}
 *
 * mybatis中 {@link org.apache.ibatis.cursor.defaults.DefaultCursor}
 * 其持有游标迭代器对象 {@link org.apache.ibatis.cursor.defaults.DefaultCursor.CursorIterator}
 *
 * @author F嘉阳
 * @date 2020/3/4 14:49
 */
public interface CourseIterator {

 /**
 * 获取下一个课程
 *
 * @return
 */
 Course nextCourse();

 /**
 * 是否是最后一个课程
 * @return
 */
 boolean isLastCourse();
}

迭代器实现

package top.fjy8018.designpattern.pattern.behavior.iterator;

import lombok.extern.slf4j.Slf4j;

import java.util.List;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:50
 */
@Slf4j
public class CourseIteratorImpl implements CourseIterator {

 private List courseList;

 private int position;

 private Course course;

 public CourseIteratorImpl(List courseList){
 this.courseList=courseList;
 }

 @Override
 public Course nextCourse() {
 log.info("返回课程,位置是: {}",position);
 course=(Course)courseList.get(position);
 position++;
 return course;
 }

 @Override
 public boolean isLastCourse(){
 if(position< courseList.size()){
 return false;
 }
 return true;
 }
}

对课程进行处理

package top.fjy8018.designpattern.pattern.behavior.iterator;

/**
 * 对课程进行处理
 *
 * @author F嘉阳
 * @date 2020/3/4 14:48
 */
public interface CourseAggregate {

 void addCourse(Course course);

 void removeCourse(Course course);

 /**
 * 获取课程迭代器
 * @return
 */
 CourseIterator getCourseIterator();
}

实现

package top.fjy8018.designpattern.pattern.behavior.iterator;

import java.util.ArrayList;
import java.util.List;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:50
 */
public class CourseAggregateImpl implements CourseAggregate {

 private List courseList;

 public CourseAggregateImpl() {
 this.courseList = new ArrayList();
 }

 @Override
 public void addCourse(Course course) {
 courseList.add(course);
 }

 @Override
 public void removeCourse(Course course) {
 courseList.remove(course);
 }

 @Override
 public CourseIterator getCourseIterator() {
 return new CourseIteratorImpl(courseList);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.iterator;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:52
 */
@Slf4j
class CourseIteratorTest {

 @Test
 void nextCourse() {
 Course course1 = new Course("Java电商一期");
 Course course2 = new Course("Java电商二期");
 Course course3 = new Course("Java设计模式精讲");
 Course course4 = new Course("Python课程");
 Course course5 = new Course("算法课程");
 Course course6 = new Course("前端课程");


 CourseAggregate courseAggregate = new CourseAggregateImpl();

 courseAggregate.addCourse(course1);
 courseAggregate.addCourse(course2);
 courseAggregate.addCourse(course3);
 courseAggregate.addCourse(course4);
 courseAggregate.addCourse(course5);
 courseAggregate.addCourse(course6);

 log.info("-----课程列表-----");
 printCourses(courseAggregate);

 courseAggregate.removeCourse(course4);
 courseAggregate.removeCourse(course5);

 log.info("-----删除操作之后的课程列表-----");
 printCourses(courseAggregate);
 }

 public static void printCourses(CourseAggregate courseAggregate){
 CourseIterator courseIterator= courseAggregate.getCourseIterator();
 while(!courseIterator.isLastCourse()){
 Course course=courseIterator.nextCourse();
 log.info(course.getName());
 }
 }
}

输出

15:18:23.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - -----课程列表-----
15:18:23.124 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 0
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java分布式
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 1
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java系统设计
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 2
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java设计模式精讲
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 3
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Python课程
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 4
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 算法课程
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 5
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 前端课程
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - -----删除操作之后的课程列表-----
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 0
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java分布式
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 1
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java系统设计
15:18:23.127 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 2
15:18:23.127 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java设计模式精讲
15:18:23.132 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 3
15:18:23.132 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 前端课程

策略模式

介绍

定义：定义了算法家族,分别封装起来,让它们之间可以互相替换, 此模式让算法的变化不会影响到使用算法的用户。

例如有大量if…else的情况

类型：行为型

适用场景

系统有很多类，而他们的区别仅仅在于他们的行为不同
一个系统需要动态地在几种算法中选择一种

优点

开闭原则
避免使用多重条件转移语句
提高算法的保密性和安全性

缺点

客户端必须知道所有的策略类,并自行决定使用哪一个策略类。
产生很多策略类

相关设计模式

策略模式和工厂模式
策略模式和状态模式

代码样例

改造前——初级策略模式

无法避免if…else

类图

促销接口

package top.fjy8018.designpattern.pattern.behavior.strategy;

/**
 * 促销接口
 *
 * @author F嘉阳
 * @date 2020/3/4 15:37
 */
public interface PromotionStrategy {

 void doPromotion();
}

满减促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 满减促销策略
 *
 * @author F嘉阳
 * @date 2020/3/4 15:38
 */
@Slf4j
public class ManJianPromotionStrategy implements PromotionStrategy {
 @Override
 public void doPromotion() {
 log.info("满减促销，满200-20");
 }
}

立减促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 立减促销策略
 *
 * @author F嘉阳
 * @date 2020/3/4 15:39
 */
@Slf4j
public class LiJianPromotionStrategy implements PromotionStrategy{
 @Override
 public void doPromotion() {
 log.info("立减促销，课程价格直接减去配置的价格");
 }
}

返现促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 返现促销策略
 *
 * @author F嘉阳
 * @date 2020/3/4 15:38
 */
@Slf4j
public class FanXianPromotionStrategy implements PromotionStrategy{
 @Override
 public void doPromotion() {
 log.info("返现促销，返回金额存放到用户余额中");
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/4 15:46
 */
@Slf4j
class PromotionStrategyTest {

 @Test
 void existIfElse() {
 String promotionKey = "LIJIAN";
 PromotionActivity activity = null;
 if (StringUtils.equals(promotionKey,"LIJIAN")){
 // 方法和类本身线程安全
 activity= new PromotionActivity(new LiJianPromotionStrategy());
 }else if(StringUtils.equals(promotionKey,"MANJIAN")){
 activity = new PromotionActivity(new FanXianPromotionStrategy());
 }

 activity.executePromotionStrategy();
 }
}

输出

16:08:28.582 [main] INFO top.fjy8018.designpattern.pattern.behavior.strategy.LiJianPromotionStrategy - 立减促销，课程价格直接减去配置的价格

改造后——结合工厂和策略模式

类图

空促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 空策略
 *
 * @author F嘉阳
 * @date 2020/3/4 16:16
 */
@Slf4j
public class EmptyPromotionStrategy implements PromotionStrategy {
 @Override
 public void doPromotion() {
 log.info("空促销策略");
 }
}

促销策略工厂

package top.fjy8018.designpattern.pattern.behavior.strategy;

import org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory;
import org.springframework.beans.factory.support.CglibSubclassingInstantiationStrategy;
import org.springframework.beans.factory.support.SimpleInstantiationStrategy;
import org.springframework.core.io.FileSystemResource;
import org.springframework.core.io.PathResource;

import java.util.Arrays;
import java.util.Comparator;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

/**
 * 促销策略工厂
 * <p>
 * {@link java.util.Comparator} 比较器，类似于抽象策略，有大量实现类，所有具体策略都要实现定义好的接口方法
 * 例如 {@link com.google.common.collect.ExplicitOrdering}
 * 应用层可以逐级传递比较器 {@link Arrays#sort(Object[], Comparator)}
 * {@link java.util.TreeMap#compare(Object, Object)} 利用内部的比较器处理
 * <p>
 * Spring资源策略抽象 {@link org.springframework.core.io.Resource}
 * 提供大量策略实现 {@link PathResource} {@link FileSystemResource}
 * <p>
 * {@link org.springframework.beans.factory.support.InstantiationStrategy} 初始化策略接口
 * 初始化具体策略实现 {@link SimpleInstantiationStrategy} {@link CglibSubclassingInstantiationStrategy}
 * 初始化策略在 {@link AbstractAutowireCapableBeanFactory} 中被使用
 *
 * @author F嘉阳
 * @date 2020/3/4 16:09
 */
public class PromotionStrategyFactory {

 private static Map<String, PromotionStrategy> STRATEGY_MAP = new ConcurrentHashMap<>();

 /**
 * 无促销
 */
 private static final PromotionStrategy NON_PROMOTION = new EmptyPromotionStrategy();

 static {
 STRATEGY_MAP.put(PromotionKey.FANXIAN, new FanXianPromotionStrategy());
 STRATEGY_MAP.put(PromotionKey.MANJIAN, new ManJianPromotionStrategy());
 STRATEGY_MAP.put(PromotionKey.LIJIAN, new LiJianPromotionStrategy());
 }

 /**
 * 禁止实例化
 */
 private PromotionStrategyFactory() {
 }

 /**
 * 开放获取策略的方法
 *
 * @param promotionKey
 * @return
 */
 public static PromotionStrategy getPromotionStrategy(String promotionKey) {
 PromotionStrategy strategy = STRATEGY_MAP.get(promotionKey);
 return strategy == null ? NON_PROMOTION : strategy;
 }

 /**
 * 常量定义
 */
 private interface PromotionKey {
 String LIJIAN = "LIJIAN";
 String FANXIAN = "FANXIAN";
 String MANJIAN = "MANJIAN";
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/3/4 15:46
 */
@Slf4j
class PromotionStrategyTest {

 @Test
 void nonIfElse() {
 String promotionKey = "LIJIAN";
 PromotionActivity activity = new PromotionActivity(PromotionStrategyFactory.getPromotionStrategy(promotionKey));
 activity.executePromotionStrategy();
 }
}

输出

16:08:28.582 [main] INFO top.fjy8018.designpattern.pattern.behavior.strategy.LiJianPromotionStrategy - 立减促销，课程价格直接减去配置的价格

解释器模式

介绍

定义：给定一个语言，定义它的文法的一种表示，并定义一个解释器，这个解释器使用该表示来解释语言中的句子。
为了解释一种语言，而为语言创建的解释器
类型：行为型

适用场景（较少）

某个特定类型问题发生频率足够高

优点

语法由很多类表示,容易改变及扩展此语言

缺点

当语法规则数目太多时,增加了系统复杂度

相关设计模式

解释器模式和适配器模式

代码样例

类图

解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * 解释器
 * <p>
 * JDK 源码
 * {@link java.util.regex.Pattern} 正则解释器
 * {@link org.springframework.expression.spel.standard.SpelExpressionParser} Spring解释器
 *
 * @author F嘉阳
 * @date 2020/3/5 14:45
 */
public interface Interpreter {
 int interpret();
}

加法解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * 加法解释器
 *
 * @author F嘉阳
 * @date 2020/3/5 14:46
 */
public class AddInterpreter implements Interpreter {

 /**
 * 定义两个表达式
 */
 private Interpreter firstExpression, secondExpression;

 public AddInterpreter(Interpreter firstExpression, Interpreter secondExpression) {
 this.firstExpression = firstExpression;
 this.secondExpression = secondExpression;
 }

 @Override
 public int interpret() {
 // 此时传入为NumberInterpreter类型，调interpret方法转int类型
 return this.firstExpression.interpret() + this.secondExpression.interpret();
 }

 @Override
 public String toString() {
 return "+";
 }
}

乘法解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * 乘法解释器
 *
 * @author F嘉阳
 * @date 2020/3/5 14:47
 */
public class MultiInterpreter implements Interpreter {

 private Interpreter firstExpression, secondExpression;

 public MultiInterpreter(Interpreter firstExpression, Interpreter secondExpression) {
 this.firstExpression = firstExpression;
 this.secondExpression = secondExpression;
 }

 @Override
 public int interpret() {
 return this.firstExpression.interpret() * this.secondExpression.interpret();
 }

 @Override
 public String toString() {
 return "*";
 }
}

类型解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:47
 */
public class NumberInterpreter implements Interpreter {
 private int number;

 public NumberInterpreter(int number) {
 this.number = number;
 }

 public NumberInterpreter(String number) {
 this.number = Integer.parseInt(number);
 }

 @Override
 public int interpret() {
 return this.number;
 }
}

操作工具类

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:47
 */
public class OperatorUtil {
 public static boolean isOperator(String symbol) {
 return (symbol.equals("+") || symbol.equals("*"));

 }

 public static Interpreter getExpressionObject(Interpreter firstExpression, Interpreter secondExpression, String symbol) {
 if (symbol.equals("+")) {
 return new AddInterpreter(firstExpression, secondExpression);
 } else if (symbol.equals("*")) {
 return new MultiInterpreter(firstExpression, secondExpression);
 }
 return null;
 }
}

自定义解释器实现

package top.fjy8018.designpattern.pattern.behavior.interpreter;

import lombok.extern.slf4j.Slf4j;

import java.util.Stack;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:48
 */
@Slf4j
public class CustomExpressionParser {
 private Stack<Interpreter> stack = new Stack<Interpreter>();

 public int parse(String str) {
 String[] strItemArray = str.split(" ");
 for (String symbol : strItemArray) {
 if (!OperatorUtil.isOperator(symbol)) {
 Interpreter numberExpression = new NumberInterpreter(symbol);
 stack.push(numberExpression);
 log.info(String.format("入栈: %d", numberExpression.interpret()));
 } else {
 //是运算符号，可以计算
 Interpreter firstExpression = stack.pop();
 Interpreter secondExpression = stack.pop();
 log.info(String.format("出栈: %d 和 %d",
 firstExpression.interpret(), secondExpression.interpret()));
 Interpreter operator = OperatorUtil.getExpressionObject(firstExpression, secondExpression, symbol);
 log.info(String.format("应用运算符: %s", operator));
 int result = operator.interpret();
 NumberInterpreter resultExpression = new NumberInterpreter(result);
 stack.push(resultExpression);
 log.info(String.format("阶段结果入栈: %d", resultExpression.interpret()));
 }
 }
 int result = stack.pop().interpret();
 return result;

 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.interpreter;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import org.springframework.expression.Expression;
import org.springframework.expression.spel.standard.SpelExpressionParser;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:48
 */
@Slf4j
class InterpreterTest {

 @Test
 void interpret() {
 String myInterpreter = "6 100 11 + *";
 CustomExpressionParser expressionParser = new CustomExpressionParser();
 int result = expressionParser.parse(myInterpreter);
 log.info("解释器计算结果: " + result);
 }

 @Test
 void springTest() {
 org.springframework.expression.ExpressionParser parser = new SpelExpressionParser();
 Expression expression = parser.parseExpression("100 * 2 + 400 * 1 + 66");
 int result = (Integer) expression.getValue();
 log.info(String.valueOf(result));
 }
}

输出

14:52:28.086 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 6
14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 100
14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 11
14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 出栈: 11 和 100
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 应用运算符: +
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 阶段结果入栈: 111
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 出栈: 111 和 6
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 应用运算符: *
14:52:28.091 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 阶段结果入栈: 666
14:52:28.091 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.InterpreterTest - 解释器计算结果: 666


14:52:28.550 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.InterpreterTest - 666

观察者模式

介绍

定义：定义了对象之间的一对多依赖，让多个观察者对象同时监听某一个主题对象，当主题对象发生变化时，它的所有依赖者（观察者）都会收到通知并更新

类型：行为型

适用场景

关联行为场景，建立一套触发机制

优点

观察者模式支持广播通信
观察者和被观察者之间建立一个抽象的耦合

缺点

使用要得当,要避免循环调用
观察者之间有过多的细节依赖、提高时间消耗及程序复杂度

代码样例——JDK观察者

类图

课程问题

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.AllArgsConstructor;
import lombok.Getter;

/**
 * 课程问题
 *
 * @author F嘉阳
 * @date 2020/3/5 15:23
 */
@Getter
@AllArgsConstructor
public class Question {

 private String username;

 private String content;

}

课程（被观察者）

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;

import java.util.EventListener;
import java.util.Observable;
import java.util.Vector;

/**
 * 课程
 * 被观察者类
 *
 * {@link Observable} 使用 {@link Vector} 保证线程安全
 *
 * JDK源码
 * {@link java.awt.Event} 事件监听器
 * {@link javax.servlet.ServletRequestListener} HTTP请求事件监听器
 * 继承 {@link EventListener}，有大量实现，例如{@link org.springframework.beans.factory.parsing.ReaderEventListener}
 *
 * @author F嘉阳
 * @date 2020/3/5 15:23
 */
@Slf4j
@Getter
@AllArgsConstructor
public class Course extends Observable {

 private String courseName;

 /**
 * 产生问题
 * @param course
 * @param question
 */
 public void produceQuestion(Course course,Question question){
 log.info("{}在{}提交了一个问题",question.getUsername(),course.getCourseName());
 // 状态变化
 setChanged();
 // 通知观察者
 notifyObservers(question);
 }
}

老师实体（观察者）

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;

import java.util.Observable;
import java.util.Observer;

/**
 * 老师实体
 *
 * 老师观察课程是否产生了新的问题
 * @author F嘉阳
 * @date 2020/3/5 15:24
 */
@Slf4j
@Getter
@AllArgsConstructor
public class Teacher implements Observer {

 private String teacherName;

 /**
 * This method is called whenever the observed object is changed. An
 * application calls an <tt>Observable</tt> object's
 * <code>notifyObservers</code> method to have all the object's
 * observers notified of the change.
 *
 * @param o the observable object.
 * @param arg an argument passed to the <code>notifyObservers</code>
 */
 @Override
 public void update(Observable o, Object arg) {
 Course course = (Course) o;
 Question question = (Question) arg;
 log.info("{}老师的{}课程接收到一个{}提交的问题，内容为：{}",
 teacherName,course.getCourseName(),question.getUsername(),question.getContent());
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/5 15:36
 */
@Slf4j
class TeacherTest {

 @Test
 void update() {
 Course course = new Course("Java设计模式");
 Teacher teacher = new Teacher("fjy8018");

 course.addObserver(teacher);

 // 业务逻辑
 Question question = new Question("fjy","问题内容XXX");

 course.produceQuestion(course,question);
 }
}

输出

15:38:34.437 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.Course - fjy在Java设计模式提交了一个问题
15:38:34.444 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.Teacher - fjy8018老师的Java设计模式课程接收到一个fjy提交的问题，内容为：问题内容XXX

代码样例——Guava监听器

package top.fjy8018.designpattern.pattern.behavior.observer.guava;

import com.google.common.eventbus.Subscribe;
import lombok.extern.slf4j.Slf4j;

import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CopyOnWriteArraySet;

/**
 * Guava事件监听器
 * {@link EventBus} 使用内部对象 {@link SubscriberRegistry} 内部使用 {@link ConcurrentMap} 存储订阅者
 * 同时使用 {@link CopyOnWriteArraySet} 保证线程安全和订阅者唯一
 * 故{@link com.google.common.eventbus.Subscriber}
 * 重写了 {@link com.google.common.eventbus.Subscriber#equals(Object)}
 * 和 {@link com.google.common.eventbus.Subscriber#hashCode()} 方法
 *
 * @author F嘉阳
 * @date 2020/3/5 15:56
 */
@Slf4j
public class GuavaEvent {

 @Subscribe
 public void subscribe(String str) {
 // 业务逻辑
 log.info("执行订阅方法，传入参数是：{}", str);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.observer;

import com.google.common.eventbus.EventBus;
import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import top.fjy8018.designpattern.pattern.behavior.observer.guava.Course;
import top.fjy8018.designpattern.pattern.behavior.observer.guava.GuavaEvent;

/**
 * @author F嘉阳
 * @date 2020/3/5 15:58
 */
@Slf4j
class GuavaEventTest {

 @Test
 void subscribe() {
 EventBus eventBus = new EventBus();
 GuavaEvent event = new GuavaEvent();
 // 注册
 eventBus.register(event);
 eventBus.post("提交事件的内容");
 }
}

输出

16:24:27.716 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.GuavaEvent - 执行订阅方法，传入参数是：提交事件的内容

备忘录模式

介绍

定义：保存一个对象的某个状态，以便在适当的时候恢复对象。（类似历史记录存储）

“后悔药”

类型：行为型

适用场景

保存及恢复数据相关业务场景
后悔的时候，即想恢复到之前的状态

优点

为用户提供一种可恢复机制
存档信息的封装

缺点

资源占用

相关设计模式

备忘录模式和状态模式

代码样例

类图

文章

package top.fjy8018.designpattern.pattern.behavior.memento;

import lombok.AllArgsConstructor;
import lombok.Data;

/**
 * 文章，字段不宜过多，否则导致过大内存占用
 *
 * @author F嘉阳
 * @date 2020/3/6 17:00
 */
@Data
@AllArgsConstructor
public class Article {

 private String title;

 private String content;

 private String imgs;

 /**
 * 保存
 * @return
 */
 public ArticleMemento saveToMemento() {
 ArticleMemento articleMemento = new ArticleMemento(this.title,this.content,this.imgs);
 return articleMemento;
 }

 /**
 * 撤销
 * @param articleMemento
 */
 public void undoFromMemento(ArticleMemento articleMemento) {
 this.title = articleMemento.getTitle();
 this.content = articleMemento.getContent();
 this.imgs = articleMemento.getImgs();
 }
}

文章备忘录（快照）

package top.fjy8018.designpattern.pattern.behavior.memento;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.ToString;
import org.springframework.binding.message.StateManageableMessageContext;

/**
 * 文章备忘录（快照）
 * 快照不需要set方法，只能通过构造器注入
 *
 * Spring工作流中源码 {@link StateManageableMessageContext#createMessagesMemento()}
 *
 * @author F嘉阳
 * @date 2020/3/6 17:03
 */
@Getter
@ToString
@AllArgsConstructor
public class ArticleMemento {

 private String title;

 private String content;

 private String imgs;
}

备忘录核心，栈实现管理者

package top.fjy8018.designpattern.pattern.behavior.memento;

import java.util.Stack;

/**
 * 保存一个对象的某个状态，以便在适当的时候恢复对象
 *
 * @author F嘉阳
 * @date 2020/3/6 17:04
 */
public class ArticleMementoManager {

 /**
 * 使用栈特性实现动作撤销
 */
 private final Stack<ArticleMemento> ARTICLE_MEMENTO_STACK = new Stack<ArticleMemento>();

 public ArticleMemento getMemento() {
 ArticleMemento articleMemento= ARTICLE_MEMENTO_STACK.pop();
 return articleMemento;
 }

 public void addMemento(ArticleMemento articleMemento)
 {
 ARTICLE_MEMENTO_STACK.push(articleMemento);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.memento;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/3/6 17:07
 */
@Slf4j
class ArticleMementoTest {

 @Test
 void getTitle() {
 ArticleMementoManager articleMementoManager = new ArticleMementoManager();

 Article article= new Article("如影随行的设计模式A","手记内容A","手记图片A");
 // 存档
 ArticleMemento articleMemento = article.saveToMemento();

 articleMementoManager.addMemento(articleMemento);
 log.info("标题:"+article.getTitle()+" 内容:"+article.getContent()+" 图片:"+article.getImgs()+" 暂存成功");

 log.info("手记完整信息:"+article);
 
 log.info("修改手记start");

 article.setTitle("如影随行的设计模式B");
 article.setContent("手记内容B");
 article.setImgs("手记图片B");

 log.info("修改手记end");

 log.info("手记完整信息:"+article);

 articleMemento = article.saveToMemento();
 articleMementoManager.addMemento(articleMemento);
 
 article.setTitle("如影随行的设计模式C");
 article.setContent("手记内容C");
 article.setImgs("手记图片C");

 log.info("暂存回退start");
 // 取档
 log.info("回退出栈1次");
 articleMemento = articleMementoManager.getMemento();
 article.undoFromMemento(articleMemento);

 log.info("回退出栈2次");
 articleMemento = articleMementoManager.getMemento();
 article.undoFromMemento(articleMemento);

 log.info("暂存回退end");
 log.info("手记完整信息:"+article);
 }
}

输出

17:33:51.154 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 标题:如影随行的设计模式A 内容:手记内容A 图片:手记图片A 暂存成功
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式A, content=手记内容A, imgs=手记图片A)
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 修改手记start
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 修改手记end
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式B, content=手记内容B, imgs=手记图片B)
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 暂存回退start
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 回退出栈1次
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 回退出栈2次
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 暂存回退end
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式A, content=手记内容A, imgs=手记图片A)

命令模式

介绍

定义：将请求封装成对象,以便使用不同的请求

特点：命令模式解决了应用程序中对象的职责以及它们之间的通信方式

类型：行为型

适用场景

请求调用者和请求接收者需要解耦,使得调用者和接收者不直接交互
需要抽象出等待执行的行为

优点

降低耦合
容易扩展新命令或者一组命令

缺点

命令的无限扩展会增加类的数量，提高系统实现复杂度

相关设计模式

命令模式和备忘录模式

代码样例

类图

命令抽象

package top.fjy8018.designpattern.pattern.behavior.command;

/**
 * 命令抽象
 * <p>
 * Java相关源码 {@link Runnable} 可理解为命令抽象
 * 其实现类为具体的命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
public interface Command {
 void execute();
}

操作视频

package top.fjy8018.designpattern.pattern.behavior.command;

import lombok.AllArgsConstructor;

/**
 * 关闭视频命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
@AllArgsConstructor
public class CloseCourseVideoCommand implements Command {
 private final CourseVideo courseVideo;

 @Override
 public void execute() {
 courseVideo.close();
 }
}

打开视频命令

package top.fjy8018.designpattern.pattern.behavior.command;

import lombok.AllArgsConstructor;

/**
 * 打开视频命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
@AllArgsConstructor
public class OpenCourseVideoCommand implements Command {
 private final CourseVideo courseVideo;

 @Override
 public void execute() {
 courseVideo.open();
 }
}

关闭视频命令

package top.fjy8018.designpattern.pattern.behavior.command;

import lombok.AllArgsConstructor;

/**
 * 关闭视频命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
@AllArgsConstructor
public class CloseCourseVideoCommand implements Command {
 private final CourseVideo courseVideo;

 @Override
 public void execute() {
 courseVideo.close();
 }
}

命令执行人

package top.fjy8018.designpattern.pattern.behavior.command;

import java.util.ArrayList;
import java.util.List;

/**
 * 命令执行人
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
public class Staff {
 /**
 * 保证执行顺序
 */
 private final List<Command> commandList = new ArrayList<Command>();

 public void addCommand(Command command) {
 commandList.add(command);
 }

 public void executeCommands() {
 for (Command command : commandList) {
 command.execute();
 }
 commandList.clear();
 }

}

测试

package top.fjy8018.designpattern.pattern.behavior.command;

import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/7/21 16:02
 */
class CommandTest {

 @Test
 void doTest() {
 CourseVideo courseVideo = new CourseVideo("Java设计模式精讲");
 OpenCourseVideoCommand openCourseVideoCommand = new OpenCourseVideoCommand(courseVideo);
 CloseCourseVideoCommand closeCourseVideoCommand = new CloseCourseVideoCommand(courseVideo);

 Staff staff = new Staff();

 staff.addCommand(openCourseVideoCommand);
 staff.addCommand(closeCourseVideoCommand);

 staff.executeCommands();
 }
}

输出

Java设计模式精讲课程视频开放
Java设计模式精讲课程视频关闭

中介者模式

介绍

定义：定义一个封装一组对象如何交互的对象

特点：通过使对象明确地相互引用来促进松散耦合,并允许独立地改变它们的交互

适用场景

系统中对象之间存在复杂的引用关系，产生的相互依赖关系结构混乱且难以理解
交互的公共行为，如果需要改变行为则可以增加新的中介者类

优点

将一对多转化成了一对一、降低程序复杂度
类之间解耦

缺点

中介者过多,导致系统复杂

相关设计模式

中介者模式和观察者模式

代码样例

类图

学习群组（中介者）

package top.fjy8018.designpattern.pattern.behavior.mediator;

import java.util.Date;
import java.util.Timer;

/**
 * 学习群组（中介者）
 * <p>
 * jdk源码 {@link Timer#schedule(java.util.TimerTask, Date)} 多个方法重载
 * 都是调用{@link Timer#sched(java.util.TimerTask, long, long)} 实现的，由底层方法进行整体协调
 *
 * @author F嘉阳
 * @date 2020/7/21 16:20
 */
public class StudyGroup {

 public static void showMessage(User user, String message) {
 System.out.println(new Date().toString() + " [" + user.getName() + "] : " + message);
 }
}

用户

package top.fjy8018.designpattern.pattern.behavior.mediator;

import lombok.AllArgsConstructor;
import lombok.Data;

/**
 * @author F嘉阳
 * @date 2020/7/21 16:20
 */
@Data
@AllArgsConstructor
public class User {
 private String name;

 public void sendMessage(String message) {
 // 只和中介者交互
 StudyGroup.showMessage(this, message);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.mediator;

import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/7/21 16:25
 */
class MediatorTest {

 @Test
 void doTest() {
 User fjy = new User("F嘉阳");
 User tom = new User("Tom");

 fjy.sendMessage(" Hey! Tom! Let's learn Design Pattern");
 tom.sendMessage("OK! F嘉阳");
 }
}

输出

Tue Jul 21 16:47:03 CST 2020 [F嘉阳] : Hey! Tom! Let's learn Design Pattern
Tue Jul 21 16:47:03 CST 2020 [Tom] : OK! F嘉阳

责任链模式

介绍

定义：为请求创建一个接收此次请求对象的链

类型：行为型

适用场景

一个请求的处理需要多个对象当中的一个或几个协作处理

优点

请求的发送者和接收者(请求的处理)解耦
责任链可以动态组合

缺点

责任链太长或者处理时间过长,影响性能
责任链有可能过多

相关设计模式

责任链模式和状态模式

代码样例

类图

模拟二级审批场景

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import javax.servlet.FilterChain;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;

/**
 * 模拟二级审批场景
 * <p>
 * 审批抽象
 * <p>
 * java 过滤器实现 {@link javax.servlet.Filter#doFilter(ServletRequest, ServletResponse, FilterChain)}
 * 实现 {@link FilterChain#doFilter(ServletRequest, ServletResponse)} 接口的组成一条过滤器链
 * 所有子类通过调用 chain.doFilter(request, response) 实现链式传递
 * <p>
 * SpringSecurity过滤器链实现也类似
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
public abstract class Approver {
 /**
 * 责任链核心：包含与自身相同的类型对象
 */
 protected Approver approver;

 /**
 * 设置下一个节点，组成链条
 *
 * @param approver
 */
 public void setNextApprover(Approver approver) {
 this.approver = approver;
 }

 /**
 * 交由子类实现
 *
 * @param course
 */
 public abstract void deploy(Course course);
}

手记审批

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import org.apache.commons.lang3.StringUtils;

/**
 * 手记审批
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
public class ArticleApprover extends Approver {
 @Override
 public void deploy(Course course) {
 if (StringUtils.isNotEmpty(course.getArticle())) {
 System.out.println(course.getName() + "含有手记,批准");
 // 是否为末尾
 if (approver != null) {
 // 执行下一个处理链
 approver.deploy(course);
 }
 } else {
 System.out.println(course.getName() + "不含有手记,不批准,流程结束");
 return;
 }
 }
}

视频审批

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import org.apache.commons.lang3.StringUtils;

/**
 * 视频审批
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
public class VideoApprover extends Approver {
 @Override
 public void deploy(Course course) {
 if (StringUtils.isNotEmpty(course.getVideo())) {
 System.out.println(course.getName() + "含有视频,批准");
 if (approver != null) {
 approver.deploy(course);
 }
 } else {
 System.out.println(course.getName() + "不含有视频,不批准,流程结束");
 return;
 }
 }
}

课程

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import lombok.Data;

/**
 * 课程
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
@Data
public class Course {
 private String name;

 private String article;

 private String video;
}

测试

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/7/21 15:06
 */
class ChainofResponsibilityTest {

 @Test
 void doTest() {
 Approver articleApprover = new ArticleApprover();
 Approver videoApprover = new VideoApprover();

 Course course = new Course();
 course.setName("Java设计模式");
 course.setArticle("Java设计模式的手记");
 course.setVideo("Java设计模式的视频");

 articleApprover.setNextApprover(videoApprover);

 articleApprover.deploy(course);
 }
}

输出

Java设计模式含有手记,批准
Java设计模式含有视频,批准

访问者模式

介绍

定义：封装作用于某数据结构(如List/set/Map等)中的各元素的操作

特性：可以在不改变各元素的类的前提下定义作用于这些元素的操作

类型：行为型

适用场景

一个数据结构如(List/set/Map等)包含很多类型对象
数据结构与数据操作分离

优点

增加新的操作很容易即增加一个新的访问者

缺点

增加新的数据结构困难
具体元素变更比较麻烦

相关设计模式

访问者模式和迭代器模式

代码样例

类图

课程抽象

package top.fjy8018.designpattern.pattern.behavior.visitor;

import lombok.Data;

/**
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
@Data
public abstract class Course {
 private String name;

 public abstract void accept(IVisitor visitor);

}

实战课

package top.fjy8018.designpattern.pattern.behavior.visitor;

import lombok.Data;
import lombok.EqualsAndHashCode;

/**
 * 实战课，包含价格
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
@Data
@EqualsAndHashCode(callSuper = true)
public class CodingCourse extends Course {
 private int price;

 @Override
 public void accept(IVisitor visitor) {
 visitor.visit(this);
 }
}

免费课

package top.fjy8018.designpattern.pattern.behavior.visitor;

/**
 * 免费课
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
public class FreeCourse extends Course {

 /**
 * 接受策略
 *
 * @param visitor
 */
 @Override
 public void accept(IVisitor visitor) {
 visitor.visit(this);
 }
}

访问者

package top.fjy8018.designpattern.pattern.behavior.visitor;

/**
 * 封装操作数据结构的方法
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
public interface IVisitor {

 void visit(FreeCourse freeCourse);

 void visit(CodingCourse codingCourse);
}

访问者核心实现

package top.fjy8018.designpattern.pattern.behavior.visitor;

/**
 * 访问者
 * 核心：对不同类型产生不同的操作行为
 * <p>
 * jdk源码 {@link java.nio.file.FileVisitor} 实现对文件的访问
 * spring源码 {@link org.springframework.beans.factory.config.BeanDefinitionVisitor} 遍历Bean具体属性并填充
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
public class Visitor implements IVisitor {

 /**
 * 访问免费课程，打印所有免费课程名称
 *
 * @param freeCourse
 */
 @Override
 public void visit(FreeCourse freeCourse) {
 System.out.println("免费课程:" + freeCourse.getName());
 }

 /**
 * 访问实战课程，打印所有实战课程名称及价格
 *
 * @param codingCourse
 */
 @Override
 public void visit(CodingCourse codingCourse) {
 System.out.println("实战课程:" + codingCourse.getName() + " 价格:" + codingCourse.getPrice() + "元");
 }

}

测试

package top.fjy8018.designpattern.pattern.behavior.visitor;

import org.junit.jupiter.api.Test;

import java.util.ArrayList;
import java.util.List;

/**
 * @author F嘉阳
 * @date 2020/7/22 16:19
 */
class VisitorTest {

 @Test
 void visit() {
 List<Course> courseList = new ArrayList<>();

 FreeCourse freeCourse = new FreeCourse();
 freeCourse.setName("SpringMVC数据绑定");

 CodingCourse codingCourse = new CodingCourse();
 codingCourse.setName("Java设计模式精讲");
 codingCourse.setPrice(299);
 // 数组组装
 courseList.add(freeCourse);
 courseList.add(codingCourse);

 for (Course course : courseList) {
 course.accept(new Visitor());
 }
 }
}

输出

免费课程:SpringMVC数据绑定
实战课程:Java设计模式精讲 价格:299元

状态模式

介绍

定义：允许一个对象在其内部状态改变时改变它的行为

类型：行为型

适用场景

一个对象存在多个状态(不同状态下行为不同),且状态可相互转换

优点

将不同的状态隔离
把各种状态的转换逻辑,分布到 State的子类中,减少相互间依赖
增加新的状态非常简单

缺点

状态多的业务场景导致类数目增加,系统变复杂

相关设计模式

状态模式和享元模式

代码样例

类图

视频播放状态转换场景

package top.fjy8018.designpattern.pattern.behavior.state;

import javax.faces.lifecycle.Lifecycle;

/**
 * 视频播放状态转换场景
 * <p>
 * {@link Lifecycle#execute(javax.faces.context.FacesContext)} 该方法通过外部状态控制改变行为
 * {@link javax.faces.webapp.FacesServlet} 状态控制
 *
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public abstract class CourseVideoState {
 /**
 * 视频上下文
 */
 protected CourseVideoContext courseVideoContext;

 public void setCourseVideoContext(CourseVideoContext courseVideoContext) {
 this.courseVideoContext = courseVideoContext;
 }

 public abstract void play();

 public abstract void speed();

 public abstract void pause();

 public abstract void stop();

}

视频上下文

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * 视频上下文
 *
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class CourseVideoContext {

 /**
 * 存储当前状态
 */
 private CourseVideoState courseVideoState;

 /**
 * 结合享元模式设置为内存共享
 */
 public final static PlayState PLAY_STATE = new PlayState();
 public final static StopState STOP_STATE = new StopState();
 public final static PauseState PAUSE_STATE = new PauseState();
 public final static SpeedState SPEED_STATE = new SpeedState();

 public CourseVideoState getCourseVideoState() {
 return courseVideoState;
 }

 public void setCourseVideoState(CourseVideoState courseVideoState) {
 this.courseVideoState = courseVideoState;
 // 把自己本身设置到实例上下文中
 this.courseVideoState.setCourseVideoContext(this);
 }
 public void play(){
 this.courseVideoState.play();
 }

 public void speed(){
 this.courseVideoState.speed();
 }

 public void stop(){
 this.courseVideoState.stop();
 }

 public void pause(){
 this.courseVideoState.pause();
 }
}

各状态实现

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class PlayState extends CourseVideoState {

 @Override
 public void play() {
 System.out.println("正常播放课程视频状态");
 }

 @Override
 public void speed() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.SPEED_STATE);
 }

 @Override
 public void pause() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PAUSE_STATE);
 }

 @Override
 public void stop() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE);
 }
}

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class PauseState extends CourseVideoState {

 @Override
 public void play() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE);
 }

 @Override
 public void speed() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.SPEED_STATE);
 }

 @Override
 public void pause() {
 System.out.println("暂停播放课程视频状态");
 }

 @Override
 public void stop() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE);
 }
}

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class SpeedState extends CourseVideoState {
 @Override
 public void play() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE);
 }

 @Override
 public void speed() {
 System.out.println("快进播放课程视频状态");
 }

 @Override
 public void pause() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PAUSE_STATE);
 }

 @Override
 public void stop() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE);
 }
}

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class StopState extends CourseVideoState {
 @Override
 public void play() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE);
 }

 @Override
 public void speed() {
 System.out.println("ERROR 停止状态不能快进!!");
 }

 @Override
 public void pause() {
 System.out.println("ERROR 停止状态不能暂停!!");
 }

 @Override
 public void stop() {
 System.out.println("停止播放课程视频状态");
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.state;

import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/7/22 16:19
 */
class PlayStateTest {

 @Test
 void play() {
 CourseVideoContext courseVideoContext = new CourseVideoContext();
 courseVideoContext.setCourseVideoState(new PlayState());

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());
 courseVideoContext.pause();

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());

 courseVideoContext.speed();

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());

 courseVideoContext.stop();

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());

 courseVideoContext.speed();
 }
}

输出

当前状态:PlayState
当前状态:PauseState
当前状态:SpeedState
当前状态:StopState
ERROR 停止状态不能快进!!

SpringBoot2.x Data JPA 多数据源爬坑

Sat, 30 Mar 2019 10:44:00 +0800

JPA 官方文档

官方文档

5.1.2. Annotation-based Configuration

The Spring Data JPA repositories support can be activated not only through an XML namespace but also by using an annotation through JavaConfig, as shown in the following example:

Example 51. Spring Data JPA repositories using JavaConfig

@Configuration
@EnableJpaRepositories
@EnableTransactionManagement
class ApplicationConfig {

 @Bean
 public DataSource dataSource() {

 EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder();
 return builder.setType(EmbeddedDatabaseType.HSQL).build();
 }

 @Bean
 public LocalContainerEntityManagerFactoryBean entityManagerFactory() {

 HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
 vendorAdapter.setGenerateDdl(true);

 LocalContainerEntityManagerFactoryBean factory = new LocalContainerEntityManagerFactoryBean();
 factory.setJpaVendorAdapter(vendorAdapter);
 factory.setPackagesToScan("com.acme.domain");
 factory.setDataSource(dataSource());
 return factory;
 }

 @Bean
 public PlatformTransactionManager transactionManager(EntityManagerFactory entityManagerFactory) {

 JpaTransactionManager txManager = new JpaTransactionManager();
 txManager.setEntityManagerFactory(entityManagerFactory);
 return txManager;
 }
}

文档中说明，基于注解的配置包含三个关键Bean：

PlatformTransactionManager——事务管理
LocalContainerEntityManagerFactoryBean——实体类管理
DataSource——数据源

同时还有一个关键注解@EnableJpaRepositories

SpringBoot官方文档

官方文档

多数据源使用@Primary指明优先级

You can even go further by leveraging what DataSourceProperties does for you — that is, by providing a default embedded database with a sensible username and password if no URL is provided. You can easily initialize a DataSourceBuilder from the state of any DataSourceProperties object, so you could also inject the DataSource that Spring Boot creates automatically. However, that would split your configuration into two namespaces: url, username, password, type, and driver on spring.datasource and the rest on your custom namespace (app.datasource). To avoid that, you can redefine a custom DataSourceProperties on your custom namespace, as shown in the following example:
@Bean
@Primary
@ConfigurationProperties("app.datasource")
public DataSourceProperties dataSourceProperties() {
	return new DataSourceProperties();
}

@Bean
@ConfigurationProperties("app.datasource.configuration")
public HikariDataSource dataSource(DataSourceProperties properties) {
	return properties.initializeDataSourceBuilder().type(HikariDataSource.class)
			.build();
}
84.2 Configure Two DataSources
If you need to configure multiple data sources, you can apply the same tricks that are described in the previous section. You must, however, mark one of the DataSource instances as @Primary, because various auto-configurations down the road expect to be able to get one by type.

If you create your own DataSource, the auto-configuration backs off. In the following example, we provide the exact same feature set as the auto-configuration provides on the primary data source:
@Bean
@Primary
@ConfigurationProperties("app.datasource.first")
public DataSourceProperties firstDataSourceProperties() {
	return new DataSourceProperties();
}

@Bean
@Primary
@ConfigurationProperties("app.datasource.first.configuration")
public HikariDataSource firstDataSource() {
	return firstDataSourceProperties().initializeDataSourceBuilder()
			.type(HikariDataSource.class).build();
}

@Bean
@ConfigurationProperties("app.datasource.second")
public BasicDataSource secondDataSource() {
	return DataSourceBuilder.create().type(BasicDataSource.class).build();
}

或者使用两个EntityManagers，并结合@EnableJpaRepositories进行配置

84.10 Use Two EntityManagers
Even if the default EntityManagerFactory works fine, you need to define a new one. Otherwise, the presence of the second bean of that type switches off the default. To make it easy to do, you can use the convenient EntityManagerBuilder provided by Spring Boot. Alternatively, you can just theLocalContainerEntityManagerFactoryBean directly from Spring ORM, as shown in the following example:
// add two data sources configured as above

@Bean
public LocalContainerEntityManagerFactoryBean customerEntityManagerFactory(
		EntityManagerFactoryBuilder builder) {
	return builder
			.dataSource(customerDataSource())
			.packages(Customer.class)
			.persistenceUnit("customers")
			.build();
}

@Bean
public LocalContainerEntityManagerFactoryBean orderEntityManagerFactory(
		EntityManagerFactoryBuilder builder) {
	return builder
			.dataSource(orderDataSource())
			.packages(Order.class)
			.persistenceUnit("orders")
			.build();
}
The configuration above almost works on its own. To complete the picture, you need to configure TransactionManagers for the two EntityManagers as well. If you mark one of them as @Primary, it could be picked up by the default JpaTransactionManager in Spring Boot. The other would have to be explicitly injected into a new instance. Alternatively, you might be able to use a JTA transaction manager that spans both.

If you use Spring Data, you need to configure @EnableJpaRepositories accordingly, as shown in the following example:
@Configuration
@EnableJpaRepositories(basePackageClasses = Customer.class,
		entityManagerFactoryRef = "customerEntityManagerFactory")
public class CustomerConfiguration {
	...
}

@Configuration
@EnableJpaRepositories(basePackageClasses = Order.class,
		entityManagerFactoryRef = "orderEntityManagerFactory")
public class OrderConfiguration {
	...
}

源码分析

自动装配类中声明了配置Bean和条件装配顺序

package org.springframework.boot.autoconfigure.data.jpa;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for Spring Data's JPA Repositories.
 * <p>
 * Activates when there is a bean of type {@link javax.sql.DataSource} configured in the
 * context, the Spring Data JPA
 * {@link org.springframework.data.jpa.repository.JpaRepository} type is on the classpath,
 * and there is no other, existing
 * {@link org.springframework.data.jpa.repository.JpaRepository} configured.
 * <p>
 * Once in effect, the auto-configuration is the equivalent of enabling JPA repositories
 * using the {@link org.springframework.data.jpa.repository.config.EnableJpaRepositories}
 * annotation.
 * <p>
 * This configuration class will activate <em>after</em> the Hibernate auto-configuration.
 *
 * @author Phillip Webb
 * @author Josh Long
 * @see EnableJpaRepositories
 */
@Configuration
@ConditionalOnBean(DataSource.class)
@ConditionalOnClass(JpaRepository.class)
@ConditionalOnMissingBean({ JpaRepositoryFactoryBean.class,
		JpaRepositoryConfigExtension.class })
@ConditionalOnProperty(prefix = "spring.data.jpa.repositories", name = "enabled", havingValue = "true", matchIfMissing = true)
@Import(JpaRepositoriesAutoConfigureRegistrar.class)
@AutoConfigureAfter({ HibernateJpaAutoConfiguration.class,
		TaskExecutionAutoConfiguration.class })
public class JpaRepositoriesAutoConfiguration {

	@Bean
	@Conditional(BootstrapExecutorCondition.class)
	public EntityManagerFactoryBuilderCustomizer entityManagerFactoryBootstrapExecutorCustomizer(
			Map<String, AsyncTaskExecutor> taskExecutors) {
		return (builder) -> {
			AsyncTaskExecutor bootstrapExecutor = determineBootstrapExecutor(
					taskExecutors);
			if (bootstrapExecutor != null) {
				builder.setBootstrapExecutor(bootstrapExecutor);
			}
		};
	}

	private AsyncTaskExecutor determineBootstrapExecutor(
			Map<String, AsyncTaskExecutor> taskExecutors) {
		if (taskExecutors.size() == 1) {
			return taskExecutors.values().iterator().next();
		}
		return taskExecutors
				.get(TaskExecutionAutoConfiguration.APPLICATION_TASK_EXECUTOR_BEAN_NAME);
	}

	private static final class BootstrapExecutorCondition extends AnyNestedCondition {

		BootstrapExecutorCondition() {
			super(ConfigurationPhase.REGISTER_BEAN);
		}

		@ConditionalOnProperty(prefix = "spring.data.jpa.repositories", name = "bootstrap-mode", havingValue = "deferred", matchIfMissing = false)
		static class DeferredBootstrapMode {

		}

		@ConditionalOnProperty(prefix = "spring.data.jpa.repositories", name = "bootstrap-mode", havingValue = "lazy", matchIfMissing = false)
		static class LazyBootstrapMode {

		}

	}
}

引入的配置类中声明了EnableJpaRepositories注解

package org.springframework.boot.autoconfigure.data.jpa;

class JpaRepositoriesAutoConfigureRegistrar
		extends AbstractRepositoryConfigurationSourceSupport {

	private BootstrapMode bootstrapMode = null;

	@Override
	protected Class<? extends Annotation> getAnnotation() {
		return EnableJpaRepositories.class;
	}

	@Override
	protected Class<?> getConfiguration() {
		return EnableJpaRepositoriesConfiguration.class;
	}

	@Override
	protected RepositoryConfigurationExtension getRepositoryConfigurationExtension() {
		return new JpaRepositoryConfigExtension();
	}

	@Override
	protected BootstrapMode getBootstrapMode() {
		return (this.bootstrapMode == null) ? super.getBootstrapMode()
				: this.bootstrapMode;
	}

	@Override
	public void setEnvironment(Environment environment) {
		super.setEnvironment(environment);
		configureBootstrapMode(environment);
	}

	private void configureBootstrapMode(Environment environment) {
		String property = environment
				.getProperty("spring.data.jpa.repositories.bootstrap-mode");
		if (StringUtils.hasText(property)) {
			this.bootstrapMode = BootstrapMode
					.valueOf(property.toUpperCase(Locale.ENGLISH));
		}
	}
 
	@EnableJpaRepositories
	private static class EnableJpaRepositoriesConfiguration {}
}

EnableJpaRepositories注解定义

package org.springframework.data.jpa.repository.config;

/**
 * Annotation to enable JPA repositories. Will scan the package of the annotated configuration class for Spring Data
 * repositories by default.
 *
 * @author Oliver Gierke
 * @author Thomas Darimont
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@Import(JpaRepositoriesRegistrar.class)
public @interface EnableJpaRepositories {

	String[] value() default {};

	String[] basePackages() default {};

	Class<?>[] basePackageClasses() default {};

	Filter[] includeFilters() default {};

	Filter[] excludeFilters() default {};

	String repositoryImplementationPostfix() default "Impl";

	String namedQueriesLocation() default "";

	Key queryLookupStrategy() default Key.CREATE_IF_NOT_FOUND;

	Class<?> repositoryFactoryBeanClass() default JpaRepositoryFactoryBean.class;

	Class<?> repositoryBaseClass() default DefaultRepositoryBaseClass.class;

	/**
	 * 实体管理工厂
	 * @return
	 */
	String entityManagerFactoryRef() default "entityManagerFactory";

	/**
	 * 事务管理
	 * @return
	 */
	String transactionManagerRef() default "transactionManager";

	boolean considerNestedRepositories() default false;

	boolean enableDefaultTransactions() default true;

	/**
	 * Configures when the repositories are initialized in the bootstrap lifecycle. {@link BootstrapMode#DEFAULT}
	 * (default) means eager initialization except all repository interfaces annotated with {@link Lazy},
	 * {@link BootstrapMode#LAZY} means lazy by default including injection of lazy-initialization proxies into client
	 * beans so that those can be instantiated but will only trigger the initialization upon first repository usage (i.e a
	 * method invocation on it). This means repositories can still be uninitialized when the application context has
	 * completed its bootstrap. {@link BootstrapMode#DEFERRED} is fundamentally the same as {@link BootstrapMode#LAZY},
	 * but triggers repository initialization when the application context finishes its bootstrap.
	 * 
	 * @return
	 * @since 2.1
	 */
	BootstrapMode bootstrapMode() default BootstrapMode.DEFAULT;
}

查找关键Bean的注册时机

package org.springframework.boot.autoconfigure.orm.jpa;

@Configuration
@ConditionalOnClass({ LocalContainerEntityManagerFactoryBean.class, EntityManager.class })
@Conditional(HibernateEntityManagerCondition.class)
@EnableConfigurationProperties(JpaProperties.class)
@AutoConfigureAfter({ DataSourceAutoConfiguration.class })
@Import(HibernateJpaConfiguration.class)
public class HibernateJpaAutoConfiguration {}

进入配置Bean

package org.springframework.boot.autoconfigure.orm.jpa;

@Configuration
@EnableConfigurationProperties(HibernateProperties.class)
@ConditionalOnSingleCandidate(DataSource.class)
class HibernateJpaConfiguration extends JpaBaseConfiguration {}

在其父类JpaBaseConfiguration中声明了装配时机和条件

package org.springframework.boot.autoconfigure.orm.jpa;

/**
 * 该类在2.0版本进行了重构
 * @since 2.0.0
 */
@Configuration
@EnableConfigurationProperties(JpaProperties.class)
@Import(DataSourceInitializedPublisher.Registrar.class)
public abstract class JpaBaseConfiguration implements BeanFactoryAware {
 
 @Bean
	@ConditionalOnMissingBean
	public PlatformTransactionManager transactionManager() {
		JpaTransactionManager transactionManager = new JpaTransactionManager();
		if (this.transactionManagerCustomizers != null) {
 // 默认使用方法名称
			this.transactionManagerCustomizers.customize(transactionManager);
		}
		return transactionManager;
	}

	@Bean
	@ConditionalOnMissingBean
	public JpaVendorAdapter jpaVendorAdapter() {
		AbstractJpaVendorAdapter adapter = createJpaVendorAdapter();
		adapter.setShowSql(this.properties.isShowSql());
		adapter.setDatabase(this.properties.determineDatabase(this.dataSource));
		adapter.setDatabasePlatform(this.properties.getDatabasePlatform());
		adapter.setGenerateDdl(this.properties.isGenerateDdl());
		return adapter;
	}

	@Bean
	@ConditionalOnMissingBean
	public EntityManagerFactoryBuilder entityManagerFactoryBuilder(
			JpaVendorAdapter jpaVendorAdapter,
			ObjectProvider<PersistenceUnitManager> persistenceUnitManager,
			ObjectProvider<EntityManagerFactoryBuilderCustomizer> customizers) {
		EntityManagerFactoryBuilder builder = new EntityManagerFactoryBuilder(
				jpaVendorAdapter, this.properties.getProperties(),
				persistenceUnitManager.getIfAvailable());
		customizers.orderedStream()
				.forEach((customizer) -> customizer.customize(builder));
		return builder;
	}

	@Bean
	@Primary
	@ConditionalOnMissingBean({ LocalContainerEntityManagerFactoryBean.class,
			EntityManagerFactory.class })
	public LocalContainerEntityManagerFactoryBean entityManagerFactory(
			EntityManagerFactoryBuilder factoryBuilder) {
		Map<String, Object> vendorProperties = getVendorProperties();
		customizeVendorProperties(vendorProperties);
		return factoryBuilder.dataSource(this.dataSource).packages(getPackagesToScan())
				.properties(vendorProperties).mappingResources(getMappingResources())
				.jta(isJta()).build();
	}
}

测试

爬完文档和源码，对需要配置Bean有了基本了解，新建工程进行多数据源测试

以当前最新版本SpringBoot进行测试

maven

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <parent>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-parent</artifactId>
 <version>2.1.3.RELEASE</version>
 <relativePath/> <!-- lookup parent from repository -->
 </parent>
 <groupId>top.fjy8018</groupId>
 <artifactId>jpadatasource</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <name>jpadatasource</name>
 <description>Demo project for Spring Boot</description>

 <properties>
 <java.version>1.8</java.version>
 </properties>

 <dependencies>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-data-jpa</artifactId>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 </dependency>

 <dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <scope>runtime</scope>
 </dependency>

 <dependency>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 </dependency>

 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-test</artifactId>
 <scope>test</scope>
 </dependency>
 </dependencies>

 <build>
 <plugins>
 <plugin>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-maven-plugin</artifactId>
 </plugin>
 </plugins>
 </build>

</project>

实体类

通过包路径区分主库和备份库

主库实体类

package top.fjy8018.jpadatasource.entity.primary;

import lombok.Data;

import javax.persistence.Entity;
import javax.persistence.Id;

/**
 * @author F嘉阳
 * @date 2019-03-30 9:11
 */
@Data
@Entity
public class Product {

 @Id
 private Integer id;

 private String name;

 private Integer price;
}

备份库实体类

package top.fjy8018.jpadatasource.entity.backup;

import lombok.Data;

import javax.persistence.Entity;
import javax.persistence.Id;

/**
 * @author F嘉阳
 * @date 2019-03-30 9:36
 */
@Data
@Entity(name = "tb_order")
public class Order {

 @Id
 private Integer id;

 private String orderName;

 private Integer price;
}

坑一：表名关键字冲突

若此处不指定表名为tb_order则会与MySQL关键字冲突导致建表失败

报错：

org.hibernate.tool.schema.spi.CommandAcceptanceException: Error executing DDL "
 create table Order (
 id integer not null,
 orderName varchar(255),
 price integer,
 primary key (id)
 ) engine=InnoDB" via JDBC Statement
 ...

DAO

package top.fjy8018.jpadatasource.repository.backup;

import org.springframework.data.jpa.repository.JpaRepository;
import top.fjy8018.jpadatasource.entity.backup.Order;

/**
 * @author F嘉阳
 * @date 2019-03-30 9:12
 */
public interface OrderRepository extends JpaRepository<Order, Integer> {
}

package top.fjy8018.jpadatasource.repository.primary;

import org.springframework.data.jpa.repository.JpaRepository;
import top.fjy8018.jpadatasource.entity.primary.Product;

/**
 * @author F嘉阳
 * @date 2019-03-30 9:12
 */
public interface ProductRepository extends JpaRepository<Product, Integer> {
}

配置Bean

官方配置样例：

Both data sources are also bound for advanced customizations. For instance, you could configure them as follows:

app.datasource.first.url=jdbc:mysql://localhost/first
app.datasource.first.username=dbuser
app.datasource.first.password=dbpass
app.datasource.first.configuration.maximum-pool-size=30

app.datasource.second.url=jdbc:mysql://localhost/second
app.datasource.second.username=dbuser
app.datasource.second.password=dbpass
app.datasource.second.max-total=30

You can apply the same concept to the secondary DataSource as well, as shown in the following example:

@Bean
@Primary
@ConfigurationProperties("app.datasource.first")
public DataSourceProperties firstDataSourceProperties() {
	return new DataSourceProperties();
}

@Bean
@Primary
@ConfigurationProperties("app.datasource.first.configuration")
public HikariDataSource firstDataSource() {
	return firstDataSourceProperties().initializeDataSourceBuilder()
			.type(HikariDataSource.class).build();
}

@Bean
@ConfigurationProperties("app.datasource.second")
public DataSourceProperties secondDataSourceProperties() {
	return new DataSourceProperties();
}

@Bean
@ConfigurationProperties("app.datasource.second.configuration")
public BasicDataSource secondDataSource() {
	return secondDataSourceProperties().initializeDataSourceBuilder()
			.type(BasicDataSource.class).build();
}

The preceding example configures two data sources on custom namespaces with the same logic as Spring Boot would use in auto-configuration. Note that each configuration sub namespace provides advanced settings based on the chosen implementation.

参照编写配置文件，此处注意SpringBoot 2.x的驱动变化

坑二：mysql驱动类名变化

spring:
 datasource:
 first:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: financial_adminer
 password: financial_adminer_pass
 url: jdbc:mysql://localhost/jpa_test?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC
 configuration:
 maximum-pool-size: 30
 second:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: financial_adminer
 password: financial_adminer_pass
 url: jdbc:mysql://localhost/jpa_test_bak?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC
 configuration:
 maximum-pool-size: 20

 jpa:
 hibernate:
 ddl-auto: update
 show-sql: true
 database-platform: org.hibernate.dialect.MySQL55Dialect
 properties:
 hibernate:
 format_sql: true
 use_sql_comments: true

配置类

package top.fjy8018.jpadatasource.config;

import com.zaxxer.hikari.HikariDataSource;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;

/**
 * @author F嘉阳
 * @date 2019-03-30 9:24
 */
@Configuration
public class DataAccessConfig {
 @Bean
 @Primary
 @ConfigurationProperties("spring.datasource.first")
 public DataSourceProperties firstDataSourceProperties() {
 return new DataSourceProperties();
 }

 @Bean
 @Primary
 @ConfigurationProperties("spring.datasource.first.configuration")
 public HikariDataSource firstDataSource() {
 return firstDataSourceProperties().initializeDataSourceBuilder()
 .type(HikariDataSource.class).build();
 }

 @Bean
 @ConfigurationProperties("spring.datasource.second")
 public DataSourceProperties secondDataSourceProperties() {
 return new DataSourceProperties();
 }

 @Bean
 @ConfigurationProperties("spring.datasource.second.configuration")
 public HikariDataSource secondDataSource() {
 return secondDataSourceProperties().initializeDataSourceBuilder()
 .type(HikariDataSource.class).build();
 }
}

配置完成后发现只在主库自动建表，备份库无变化，属于预期情况，因为目前仅配置了三个必须Bean的其中一个

继续配置实体管理和事务管理Bean

官方文档样例

84.10 Use Two EntityManagers
Even if the default EntityManagerFactory works fine, you need to define a new one. Otherwise, the presence of the second bean of that type switches off the default. To make it easy to do, you can use the convenient EntityManagerBuilder provided by Spring Boot. Alternatively, you can just theLocalContainerEntityManagerFactoryBean directly from Spring ORM, as shown in the following example:
// add two data sources configured as above

@Bean
public LocalContainerEntityManagerFactoryBean customerEntityManagerFactory(
		EntityManagerFactoryBuilder builder) {
	return builder
			.dataSource(customerDataSource())
			.packages(Customer.class)
			.persistenceUnit("customers")
			.build();
}

@Bean
public LocalContainerEntityManagerFactoryBean orderEntityManagerFactory(
		EntityManagerFactoryBuilder builder) {
	return builder
			.dataSource(orderDataSource())
			.packages(Order.class)
			.persistenceUnit("orders")
			.build();
}
The configuration above almost works on its own. To complete the picture, you need to configure TransactionManagers for the two EntityManagers as well. If you mark one of them as @Primary, it could be picked up by the default JpaTransactionManager in Spring Boot. The other would have to be explicitly injected into a new instance. Alternatively, you might be able to use a JTA transaction manager that spans both.

If you use Spring Data, you need to configure @EnableJpaRepositories accordingly, as shown in the following example:
@Configuration
@EnableJpaRepositories(basePackageClasses = Customer.class,
		entityManagerFactoryRef = "customerEntityManagerFactory")
public class CustomerConfiguration {
	...
}

@Configuration
@EnableJpaRepositories(basePackageClasses = Order.class,
		entityManagerFactoryRef = "orderEntityManagerFactory")
public class OrderConfiguration {
	...
}

工程配置

package top.fjy8018.jpadatasource.config;

import com.zaxxer.hikari.HikariDataSource;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.orm.jpa.JpaTransactionManager;
import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean;
import org.springframework.transaction.PlatformTransactionManager;
import top.fjy8018.jpadatasource.entity.backup.Order;
import top.fjy8018.jpadatasource.entity.primary.Product;
import top.fjy8018.jpadatasource.repository.backup.OrderRepository;
import top.fjy8018.jpadatasource.repository.primary.ProductRepository;

import javax.sql.DataSource;

/**
 * @author F嘉阳
 * @date 2019-03-30 9:24
 */
@Configuration
public class DataAccessConfig {
 @Bean
 @Primary
 @ConfigurationProperties("spring.datasource.first")
 public DataSourceProperties firstDataSourceProperties() {
 return new DataSourceProperties();
 }

 @Bean
 @Primary
 @ConfigurationProperties("spring.datasource.first.configuration")
 public HikariDataSource firstDataSource() {
 return firstDataSourceProperties().initializeDataSourceBuilder()
 .type(HikariDataSource.class).build();
 }

 @Bean
 @ConfigurationProperties("spring.datasource.second")
 public DataSourceProperties secondDataSourceProperties() {
 return new DataSourceProperties();
 }

 @Bean
 @ConfigurationProperties("spring.datasource.second.configuration")
 public HikariDataSource secondDataSource() {
 return secondDataSourceProperties().initializeDataSourceBuilder()
 .type(HikariDataSource.class).build();
 }

 @Bean
 public LocalContainerEntityManagerFactoryBean firstEntityManagerFactory(
 EntityManagerFactoryBuilder builder, @Qualifier("firstDataSource") DataSource dataSource) {
 return builder
 .dataSource(dataSource)
 .packages(Product.class)
 .persistenceUnit("first")
 .build();
 }

 @Bean
 public LocalContainerEntityManagerFactoryBean secondEntityManagerFactory(
 EntityManagerFactoryBuilder builder, @Qualifier("secondDataSource") DataSource dataSource) {
 return builder
 .dataSource(dataSource)
 .packages(Order.class)
 .persistenceUnit("second")
 .build();
 }

 @Bean
 public PlatformTransactionManager primaryTransactionManager(
 @Qualifier("firstEntityManagerFactory") LocalContainerEntityManagerFactoryBean entityManagerFactory) {
 return new JpaTransactionManager(entityManagerFactory.getObject());
 }

 @Bean
 public PlatformTransactionManager backupTransactionManager(
 @Qualifier("secondEntityManagerFactory") LocalContainerEntityManagerFactoryBean entityManagerFactory) {
 return new JpaTransactionManager(entityManagerFactory.getObject());
 }

 @EnableJpaRepositories(basePackageClasses = ProductRepository.class,
 entityManagerFactoryRef = "firstEntityManagerFactory")
 @Primary
 public class PrimaryConfiguration {
 }

 @EnableJpaRepositories(basePackageClasses = OrderRepository.class,
 entityManagerFactoryRef = "secondEntityManagerFactory")
 public class secondConfiguration {
 }
}

坑三：官方文档样例有误

此时配置直接启动会报错

Method requestMappingHandlerMapping in org.springframework.boot.autoconfigure.web.servlet.WebMvcAutoConfiguration$EnableWebMvcConfiguration required a single bean, but 2 were found:
	- firstEntityManagerFactory: defined by method 'firstEntityManagerFactory' in class path resource [top/fjy8018/jpadatasource/config/DataAccessConfig.class]
	- secondEntityManagerFactory: defined by method 'secondEntityManagerFactory' in class path resource [top/fjy8018/jpadatasource/config/DataAccessConfig.class]


Action:

Consider marking one of the beans as @Primary, updating the consumer to accept multiple beans, or using @Qualifier to identify the bean that should be consumed

意思是自动装配时发现两个类型一致的Bean，主要指定优先级，而官方文档对此处可能出现的问题并未提示

指定firstEntityManagerFactory为优先Bean，同时文档样例创建了事务管理器但未配置到@EnableJpaRepositories注解中，此处也一并配置

最终配置类如下

package top.fjy8018.jpadatasource.config;

import com.zaxxer.hikari.HikariDataSource;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.orm.jpa.JpaTransactionManager;
import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean;
import org.springframework.transaction.PlatformTransactionManager;
import top.fjy8018.jpadatasource.entity.backup.Order;
import top.fjy8018.jpadatasource.entity.primary.Product;
import top.fjy8018.jpadatasource.repository.backup.OrderRepository;
import top.fjy8018.jpadatasource.repository.primary.ProductRepository;

import javax.sql.DataSource;

/**
 * @author F嘉阳
 * @date 2019-03-30 9:24
 */
@Configuration
public class DataAccessConfig {
 @Bean
 @Primary
 @ConfigurationProperties("spring.datasource.first")
 public DataSourceProperties firstDataSourceProperties() {
 return new DataSourceProperties();
 }

 @Bean
 @Primary
 @ConfigurationProperties("spring.datasource.first.configuration")
 public HikariDataSource firstDataSource() {
 return firstDataSourceProperties().initializeDataSourceBuilder()
 .type(HikariDataSource.class).build();
 }

 @Bean
 @ConfigurationProperties("spring.datasource.second")
 public DataSourceProperties secondDataSourceProperties() {
 return new DataSourceProperties();
 }

 @Bean
 @ConfigurationProperties("spring.datasource.second.configuration")
 public HikariDataSource secondDataSource() {
 return secondDataSourceProperties().initializeDataSourceBuilder()
 .type(HikariDataSource.class).build();
 }

 @Bean
 public LocalContainerEntityManagerFactoryBean firstEntityManagerFactory(
 EntityManagerFactoryBuilder builder, @Qualifier("firstDataSource") DataSource dataSource) {
 return builder
 .dataSource(dataSource)
 .packages(Product.class)
 .persistenceUnit("first")
 .build();
 }

 @Bean
 public LocalContainerEntityManagerFactoryBean secondEntityManagerFactory(
 EntityManagerFactoryBuilder builder, @Qualifier("secondDataSource") DataSource dataSource) {
 return builder
 .dataSource(dataSource)
 .packages(Order.class)
 .persistenceUnit("second")
 .build();
 }

 @Bean
 public PlatformTransactionManager primaryTransactionManager(
 @Qualifier("firstEntityManagerFactory") LocalContainerEntityManagerFactoryBean entityManagerFactory) {
 return new JpaTransactionManager(entityManagerFactory.getObject());
 }

 @Bean
 public PlatformTransactionManager backupTransactionManager(
 @Qualifier("secondEntityManagerFactory") LocalContainerEntityManagerFactoryBean entityManagerFactory) {
 return new JpaTransactionManager(entityManagerFactory.getObject());
 }

 @EnableJpaRepositories(basePackageClasses = ProductRepository.class,
 entityManagerFactoryRef = "firstEntityManagerFactory", transactionManagerRef = "primaryTransactionManager")
 @Primary
 public class PrimaryConfiguration {
 }

 @EnableJpaRepositories(basePackageClasses = OrderRepository.class,
 entityManagerFactoryRef = "secondEntityManagerFactory", transactionManagerRef = "backupTransactionManager")
 public class secondConfiguration {
 }
}

坑四：JPA不会自动建表

配置完成后启动

发现日志中建表SQL没有输出，同时查看数据库，确认表未创建

但官方文档对多数据源的配置说明已经结束，令人疑惑

自动建表语句没生成猜想可能由于与ddl相关的属性未配置导致

进入配置文件查看ddl配置，发现默认为false

将其配置为true后总配置文件如下

spring:
 datasource:
 first:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: financial_adminer
 password: financial_adminer_pass
 url: jdbc:mysql://120.79.226.26/jpa_test?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC
 configuration:
 maximum-pool-size: 30
 second:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: financial_adminer
 password: financial_adminer_pass
 url: jdbc:mysql://120.79.226.26/jpa_test_bak?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC
 configuration:
 maximum-pool-size: 20

 jpa:
 hibernate:
 ddl-auto: update
 show-sql: true
 database-platform: org.hibernate.dialect.MySQL55Dialect
 properties:
 hibernate:
 format_sql: true
 use_sql_comments: true
 generate-ddl: true

启动工程发现自动建表生效

查看数据库情况，与预期一致，主库只包含主库的表，备份库只包含备份库的表，多数据配置完成

坑五：hibernate默认配置被覆盖

若在单数据源配置下应当无需配置该项，同时发现本应该默认将驼峰属性名映射为_拼接名称策略也失效，说明自定义配置源时覆盖了默认配置或者导致默认配置失效

查询源码分析原因，发现hibernate相关配置均在HibernateProperties配置类中

package org.springframework.boot.autoconfigure.orm.jpa;

/**
 * Configuration properties for Hibernate.
 *
 * @author Stephane Nicoll
 * @since 2.1.0
 * @see JpaProperties
 */
@ConfigurationProperties("spring.jpa.hibernate")
public class HibernateProperties {}

而该类在HibernateJpaConfiguration中进行注入，但配置类中进行了条件装配注解@ConditionalOnSingleCandidate标注，导致在多数据源情况下不满足该配置，类没有被装配到Spring Bean中

package org.springframework.boot.autoconfigure.orm.jpa;

/**
 * {@link JpaBaseConfiguration} implementation for Hibernate.
 * @since 2.0.0
 */
@Configuration
@EnableConfigurationProperties(HibernateProperties.class)
@ConditionalOnSingleCandidate(DataSource.class)
class HibernateJpaConfiguration extends JpaBaseConfiguration {

	private final HibernateProperties hibernateProperties;

	HibernateJpaConfiguration(DataSource dataSource, JpaProperties jpaProperties,
			ConfigurableListableBeanFactory beanFactory,
			ObjectProvider<JtaTransactionManager> jtaTransactionManager,
			ObjectProvider<TransactionManagerCustomizers> transactionManagerCustomizers,
			HibernateProperties hibernateProperties,
			ObjectProvider<Collection<DataSourcePoolMetadataProvider>> metadataProviders,
			ObjectProvider<SchemaManagementProvider> providers,
			ObjectProvider<PhysicalNamingStrategy> physicalNamingStrategy,
			ObjectProvider<ImplicitNamingStrategy> implicitNamingStrategy,
			ObjectProvider<HibernatePropertiesCustomizer> hibernatePropertiesCustomizers) {
		super(dataSource, jpaProperties, jtaTransactionManager,
				transactionManagerCustomizers);
		this.hibernateProperties = hibernateProperties;
		this.defaultDdlAutoProvider = new HibernateDefaultDdlAutoProvider(providers);
		this.poolMetadataProvider = new CompositeDataSourcePoolMetadataProvider(
				metadataProviders.getIfAvailable());
		this.hibernatePropertiesCustomizers = determineHibernatePropertiesCustomizers(
				physicalNamingStrategy.getIfAvailable(),
				implicitNamingStrategy.getIfAvailable(), beanFactory,
				hibernatePropertiesCustomizers.orderedStream()
						.collect(Collectors.toList()));
	}
 ...
}

解决方法就只能自己工程内实现该类的配置，注意，Spring 2.x版本自动装配变化较大，当前配置基于最新版2.1.3有效，并不代表对后续版本一定兼容

工程配置

package top.fjy8018.jpadatasource.config;

import com.zaxxer.hikari.HikariDataSource;
import org.hibernate.boot.model.naming.ImplicitNamingStrategy;
import org.hibernate.boot.model.naming.PhysicalNamingStrategy;
import org.hibernate.cfg.AvailableSettings;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.config.ConfigurableListableBeanFactory;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties;
import org.springframework.boot.autoconfigure.orm.jpa.HibernatePropertiesCustomizer;
import org.springframework.boot.autoconfigure.orm.jpa.HibernateSettings;
import org.springframework.boot.autoconfigure.orm.jpa.JpaProperties;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.jdbc.EmbeddedDatabaseConnection;
import org.springframework.boot.jdbc.SchemaManagement;
import org.springframework.boot.jdbc.SchemaManagementProvider;
import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.orm.hibernate5.SpringBeanContainer;
import org.springframework.orm.jpa.JpaTransactionManager;
import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.util.ClassUtils;
import top.fjy8018.jpadatasource.entity.backup.Order;
import top.fjy8018.jpadatasource.entity.primary.Product;
import top.fjy8018.jpadatasource.repository.backup.OrderRepository;
import top.fjy8018.jpadatasource.repository.primary.ProductRepository;

import javax.sql.DataSource;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.function.Supplier;
import java.util.stream.Collectors;
import java.util.stream.StreamSupport;

/**
 * @author F嘉阳
 * @date 2019-03-30 9:24
 */
@Configuration
public class DataAccessConfig {

 private final ObjectProvider<SchemaManagementProvider> providers;

 private final HibernateProperties hibernateProperties;

 private final JpaProperties properties;

 private final ObjectProvider<PhysicalNamingStrategy> physicalNamingStrategy;

 private final ObjectProvider<ImplicitNamingStrategy> implicitNamingStrategy;

 private final ConfigurableListableBeanFactory beanFactory;

 private final ObjectProvider<HibernatePropertiesCustomizer> hibernatePropertiesCustomizers;

 @Autowired
 public DataAccessConfig(ObjectProvider<SchemaManagementProvider> providers, HibernateProperties hibernateProperties, JpaProperties properties, ObjectProvider<PhysicalNamingStrategy> physicalNamingStrategy, ObjectProvider<ImplicitNamingStrategy> implicitNamingStrategy, ConfigurableListableBeanFactory beanFactory, ObjectProvider<HibernatePropertiesCustomizer> hibernatePropertiesCustomizers) {
 this.providers = providers;
 this.hibernateProperties = hibernateProperties;
 this.properties = properties;
 this.physicalNamingStrategy = physicalNamingStrategy;
 this.implicitNamingStrategy = implicitNamingStrategy;
 this.beanFactory = beanFactory;
 this.hibernatePropertiesCustomizers = hibernatePropertiesCustomizers;
 }

 @Bean
 @Primary
 @ConfigurationProperties("spring.datasource.first")
 public DataSourceProperties firstDataSourceProperties() {
 return new DataSourceProperties();
 }

 @Bean
 @Primary
 @ConfigurationProperties("spring.datasource.first.configuration")
 public HikariDataSource firstDataSource() {
 return firstDataSourceProperties().initializeDataSourceBuilder()
 .type(HikariDataSource.class).build();
 }

 @Bean
 @ConfigurationProperties("spring.datasource.second")
 public DataSourceProperties secondDataSourceProperties() {
 return new DataSourceProperties();
 }

 @Bean
 @ConfigurationProperties("spring.datasource.second.configuration")
 public HikariDataSource secondDataSource() {
 return secondDataSourceProperties().initializeDataSourceBuilder()
 .type(HikariDataSource.class).build();
 }

 @Bean
 @Primary
 public LocalContainerEntityManagerFactoryBean firstEntityManagerFactory(
 EntityManagerFactoryBuilder builder, @Qualifier("firstDataSource") DataSource dataSource) {
 return builder
 .dataSource(dataSource)
 // 加入规约配置
 .properties(getVendorProperties(dataSource))
 .packages(Product.class)
 .persistenceUnit("first")
 .build();
 }

 @Bean
 public LocalContainerEntityManagerFactoryBean secondEntityManagerFactory(
 EntityManagerFactoryBuilder builder, @Qualifier("secondDataSource") DataSource dataSource) {
 return builder
 .dataSource(dataSource)
 .properties(getVendorProperties(dataSource))
 .packages(Order.class)
 .persistenceUnit("second")
 .build();
 }

 @Bean
 public PlatformTransactionManager primaryTransactionManager(
 @Qualifier("firstEntityManagerFactory") LocalContainerEntityManagerFactoryBean entityManagerFactory) {
 return new JpaTransactionManager(entityManagerFactory.getObject());
 }

 @Bean
 public PlatformTransactionManager backupTransactionManager(
 @Qualifier("secondEntityManagerFactory") LocalContainerEntityManagerFactoryBean entityManagerFactory) {
 return new JpaTransactionManager(entityManagerFactory.getObject());
 }

 @EnableJpaRepositories(basePackageClasses = ProductRepository.class,
 entityManagerFactoryRef = "firstEntityManagerFactory", transactionManagerRef = "primaryTransactionManager")
 @Primary
 public class PrimaryConfiguration {
 }

 @EnableJpaRepositories(basePackageClasses = OrderRepository.class,
 entityManagerFactoryRef = "secondEntityManagerFactory", transactionManagerRef = "backupTransactionManager")
 public class secondConfiguration {
 }

 /**
 * 获取配置文件信息
 *
 * @param dataSource
 * @return
 */
 private Map<String, Object> getVendorProperties(DataSource dataSource) {
 List<HibernatePropertiesCustomizer> hibernatePropertiesCustomizers = determineHibernatePropertiesCustomizers(
 physicalNamingStrategy.getIfAvailable(),
 implicitNamingStrategy.getIfAvailable(), beanFactory,
 this.hibernatePropertiesCustomizers.orderedStream()
 .collect(Collectors.toList()));
 Supplier<String> defaultDdlMode = () -> new HibernateDefaultDdlAutoProvider(providers)
 .getDefaultDdlAuto(dataSource);
 return new LinkedHashMap<>(this.hibernateProperties.determineHibernateProperties(
 properties.getProperties(),
 new HibernateSettings().ddlAuto(defaultDdlMode)
 .hibernatePropertiesCustomizers(
 hibernatePropertiesCustomizers)));
 }

 /**
 * 命名策略自动判断
 *
 * @param physicalNamingStrategy
 * @param implicitNamingStrategy
 * @param beanFactory
 * @param hibernatePropertiesCustomizers
 * @return
 */
 private List<HibernatePropertiesCustomizer> determineHibernatePropertiesCustomizers(
 PhysicalNamingStrategy physicalNamingStrategy,
 ImplicitNamingStrategy implicitNamingStrategy,
 ConfigurableListableBeanFactory beanFactory,
 List<HibernatePropertiesCustomizer> hibernatePropertiesCustomizers) {
 List<HibernatePropertiesCustomizer> customizers = new ArrayList<>();
 if (ClassUtils.isPresent(
 "org.hibernate.resource.beans.container.spi.BeanContainer",
 getClass().getClassLoader())) {
 customizers
 .add((properties) -> properties.put(AvailableSettings.BEAN_CONTAINER,
 new SpringBeanContainer(beanFactory)));
 }
 if (physicalNamingStrategy != null || implicitNamingStrategy != null) {
 customizers.add(new NamingStrategiesHibernatePropertiesCustomizer(
 physicalNamingStrategy, implicitNamingStrategy));
 }
 customizers.addAll(hibernatePropertiesCustomizers);
 return customizers;
 }

 /**
 * 自动进行建表操作
 */
 class HibernateDefaultDdlAutoProvider implements SchemaManagementProvider {

 private final Iterable<SchemaManagementProvider> providers;

 HibernateDefaultDdlAutoProvider(Iterable<SchemaManagementProvider> providers) {
 this.providers = providers;
 }

 public String getDefaultDdlAuto(DataSource dataSource) {
 if (!EmbeddedDatabaseConnection.isEmbedded(dataSource)) {
 return "none";
 }
 SchemaManagement schemaManagement = getSchemaManagement(dataSource);
 if (SchemaManagement.MANAGED.equals(schemaManagement)) {
 return "none";
 }
 return "create-drop";

 }

 @Override
 public SchemaManagement getSchemaManagement(DataSource dataSource) {
 return StreamSupport.stream(this.providers.spliterator(), false)
 .map((provider) -> provider.getSchemaManagement(dataSource))
 .filter(SchemaManagement.MANAGED::equals).findFirst()
 .orElse(SchemaManagement.UNMANAGED);
 }

 }

 private static class NamingStrategiesHibernatePropertiesCustomizer
 implements HibernatePropertiesCustomizer {

 private final PhysicalNamingStrategy physicalNamingStrategy;

 private final ImplicitNamingStrategy implicitNamingStrategy;

 NamingStrategiesHibernatePropertiesCustomizer(
 PhysicalNamingStrategy physicalNamingStrategy,
 ImplicitNamingStrategy implicitNamingStrategy) {
 this.physicalNamingStrategy = physicalNamingStrategy;
 this.implicitNamingStrategy = implicitNamingStrategy;
 }

 /**
 * 数据库命名映射策略
 *
 * @param hibernateProperties the JPA vendor properties to customize
 */
 @Override
 public void customize(Map<String, Object> hibernateProperties) {
 if (this.physicalNamingStrategy != null) {
 hibernateProperties.put("hibernate.physical_naming_strategy",
 this.physicalNamingStrategy);
 }
 if (this.implicitNamingStrategy != null) {
 hibernateProperties.put("hibernate.implicit_naming_strategy",
 this.implicitNamingStrategy);
 }
 }
 }
}

配置Bean完成后则规约配置会生效，便无需在配置文件显示配置DDL语句执行开关

启动日志也可看出结果符合预期

总结

由此发现，在SpringBoot版本迁移过程中不一定互相兼容，同时官方文档尽管全面，但随着版本变化文档也有不准确情况。而如果完全依靠爬源码复制源码配置也是无法实现最简单化的配置，需要结合文档、源码、日志逐一分析

工程源码已开放，注意配置自己的数据库连接地址，工程内的连接地址已经失效

GitHub源码

源码学习 on F嘉阳

RocketMQ学习笔记5——原理解析

消息存储结构

同步\异步刷盘

异步刷盘

同步刷盘

同步\异步复制

同步复制

异步复制

同步双写

高可用机制

最佳实践

NameServer协调者

为什么不用zookeeper

RocketMQ学习笔记4——消费者核心

配置参数详解

PushConsumer核心参数

集群与广播模式

集群模式

样例

广播模式

消费存储核心-OffSet

集群模式-RemoteBrokerOffsetStore

广播模式-LocalFileOffsetStore

PushConsumer长轮询模式分析

长轮询过程

源码分析

DefaultMQPullConsumer消息拉取模式

生产者样例

使用普通Pull轮询模式

Pull定时任务实现样例

实现原理

RocketMQ学习笔记3——生产者核心

配置参数

主从同步机制

元数据同步源码分析

同步Topic配置信息源码

同步消费者偏移量源码

消息同步源码

逻辑

连接Master源码

同步\异步消息发送

实现分类

实例

同步消息发送源码分析

底层Netty通信

RemotingClient源码

RemotingServer源码

NettyRemotingServer源码

NettyRemotingAbstract源码

Netty编码解码源码解析——RemotingCommand源码

消息返回状态详解

延迟消息投递

自定义消息发送规则

RocketMQ学习笔记2——入门

生产者模型使用

消费者模型使用

四种集群环境

主从集群环境构建

基础环境

修改主节点配置

修改从节点配置

启动NameSrv

启动Broker

启动控制台

数据高可用机制故障演练

RocketMQ学习笔记1——初探

RocketMQ整体认知

核心概念模型

基本概念

源码结构

编译部署&环境搭建

编译部署

二进制包部署（单点）

创建存储目录

RocketMQ 配置文件

修改日志配置文件

修改启动脚本参数

启动 NameServer

启动 BrokerServer A

集群模式-`RemoteBrokerOffsetStore`

广播模式-`LocalFileOffsetStore`

`DefaultMQPullConsumer`消息拉取模式

`RemotingClient`源码

`RemotingServer`源码

`NettyRemotingServer`源码

`NettyRemotingAbstract`源码

Netty编码解码源码解析——`RemotingCommand`源码

`@Component` “派生性”

`@Component` “层次性”

配置方式 - `@Profile`

编程方式 - `@Conditional`