学习 on F嘉阳

本地LLM训练

Sun, 30 Mar 2025 11:44:00 +0800

参考资料

jingyaogong/minimind

环境准备

本地环境配置

CPU 12700
内存 32G DDR4 3200
GPU RTX 2060Super
系统 Win11
CUDA 12.6
Python 3.12

由于需要本地编译，windows环境需要安装Microsoft Visual C++ 编译器，tiktoken编译需要rust环境，安装地址：

克隆并安装环境

git clone https://github.com/jingyaogong/minimind.git
cd minimind
pip install -r requirements.txt

注意：此处不建议使用清华源、也不建议使用conda虚拟环境，会导致pytorch包版本匹配失败

依赖安装检查，执行以下代码

import torch 
print(torch.__version__) 
print(torch.version.cuda) 
print(torch.cuda.is_available())

输入如下即成功

(base) PS F:\java_workspace\minimind> python .\test_env.py
2.6.0+cu126
12.6
True

按照文档指引下载默认的预训练集和监督训练集

minimind_dataset · 数据集

训练过程

预训练

执行预训练

python train_pretrain.py

在我的环境下需要3小时才能完成预训练

执行预训练，得到pretrain_*.pth作为预训练的输出权重（其中为模型的dimension，默认为512）

监督微调

python train_full_sft.py

执行监督全参数微调，在我的环境下需要2.5小时

效果测试

测试全参微调

python eval_model.py --model_mode 1

从结果来看，能进行对话，但受限于参数量，很多知识不完整，对话回答质量比较一般。

模型优化

受本地环境限制，只能完成监督微调步骤，后续步骤执行会爆显存，耗时显著增加。后续学习改为使用阿里云PAI-DSW环境

基于 ChatGLM2-6B 搭建个人知识库

Wed, 06 Sep 2023 22:43:00 +0800

背景

ChatGLM2-6B 是开源中英双语对话模型 ChatGLM-6B的第二代版本，提升了性能和推理能力，可结合Langchain-Chatchat和Embedding 可实现基于本地知识库的推理能力

准备环境

由于需要构建强大的知识库而非通过prompt组装问答方式，故需要更大的显存和内存，此处直接使用阿里云提供的A10显卡，可免费试用3个月

申请机器

创建工作空间

创建实例

启动机器后进入工作台

部署ChatGLM2-6B

下载仓库

参考官方仓库的部署指南

$ git clone https://github.com/THUDM/ChatGLM2-6B
$ cd ChatGLM2-6B

然后使用 pip 安装依赖：

$ pip install -r requirements.txt

安装git lfs

阿里云的git源直接安装lfs会存在问题，需要安装前置依赖

$ apt -y install build-essential nghttp2 libnghttp2-dev libssl-dev
$ git lfs install --skip-smudge

下载模型

$ git clone https://huggingface.co/THUDM/chatglm2-6b /mnt/workspace/model/chatglm2-6b
$ git lfs fetch
$ git lfs checkout

修改启动配置

此处只使用webUI模式验证，修改 web_demo.py

tokenizer= AutoTokenizer.from_pretrained("/mnt/workspace/model/chatglm2-6b", trust_remote_code=True)
model= AutoModel.from_pretrained("/mnt/workspace/model/chatglm2-6b", trust_remote_code=True).half().cuda()

# 要暴露在公网上
demo.queue().launch(share=True, inbrowser=True, server_name='0.0.0.0', server_port=7860)

安装启动依赖

$ pip install gradio

启动WebUI

$ python web_demo.py

启动成功

使用非量化模型，显存占用12G

问答验证

基于A10显卡推理，运算速度非常快，对问题的理解能力也比之前体验的chatglm-6b更加准确，但依然有不少文字错误

部署Langchain知识库应用

介绍

Langchain-Chatchat是一种利用 langchain 思想实现的基于本地知识库的问答应用，目标期望建立一套对中文场景与开源模型支持友好、可离线运行的知识库问答解决方案。

原理示意图

更多介绍参考官方仓库

下载仓库

Langchain依赖的python包有些许差别，需要先升级python依赖版本

$ pip3 install --upgrade pip
$ pip uninstall accelerate
$ pip install accelerate>=0.20.3

# 拉取仓库
$ git clone https://github.com/chatchat-space/Langchain-Chatchat.git

# 进入目录
$ cd Langchain-Chatchat

# 安装全部依赖
$ pip install -r requirements.txt

下载Embedding模型

此处使用官方推荐的m3e-base模型为例

$ git clone https://huggingface.co/moka-ai/m3e-base
$ git lfs fetch
$ git lfs checkout

修改配置

修改Langchain模型引用配置

$ cd Langchain-Chatchat

复制样例文件，创建新配置configs/model_config.py 和 configs/server_config.py

本地的 LLM 模型本地存储路径写在 llm_model_dict 对应模型的 local_model_path 属性中

llm_model_dict={
 "chatglm2-6b": {
 "local_model_path": "/mnt/workspace/model/chatglm2-6b",
 "api_base_url": "http://127.0.0.1:9090/v1", # "name"修改为 FastChat 服务中的"api_base_url"
 "api_key": "EMPTY"
 },
 }

本地的 Embedding 模型本地存储路径写在 embedding_model_dict 对应模型位置

embedding_model_dict = {
 "m3e-base": "/mnt/workspace/model/m3e-base",
 }

重建知识库

首次使用模型需要重建知识库

$ python init_database.py --recreate-vs

启动知识库

一键启动所有 Fastchat 服务、API 服务、WebUI 服务

$ python startup.py -a

问答验证

LLM模型验证，连接本地ChatGLM2-6B成功

工程自带样例知识库，可以使用内置知识库验证，此处跳过，直接使用个人知识库验证

构建知识库

基金研报知识库

从天天基金拉取研报：中信证券：把握宏观政策接连发力下电商板块估值修复机会_天天基金网 (eastmoney.com)

将内容存储为txt上传到知识库

基于基金研报知识库问答，总结准确，速度非常快

推理过程日志，显存占用14G

个人面试知识库

将Java并发知识点转为txt类型知识库上传并创建新的知识库空间

不使用知识库情况下回答

使用知识库情况下回答

显然使用知识库回答更加准确

ChatGLM-6B 本地化部署体验

Sun, 28 May 2023 19:50:00 +0800

背景

在ChatGPT引爆大语言模型热潮下，国内清华大学开源了包含62亿参数的GLM-6B，由于使用的参数较少，因此能支持在消费级显卡上私有化部署体验。

硬件

CPU：12700

内存：32G

GPU：RTX 2060 SUPER 8G

实践-ChatGLM-6B官方版本

介绍

GitHub - THUDM/ChatGLM-6B: ChatGLM-6B: An Open Bilingual Dialogue Language Model | 开源双语对话语言模型

ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于 General Language Model (GLM) 架构，具有 62 亿参数。结合模型量化技术，用户可以在消费级的显卡上进行本地部署（INT4 量化级别下最低只需 6GB 显存）。 ChatGLM-6B 使用了和 ChatGPT 相似的技术，针对中文问答和对话进行了优化。经过约 1T 标识符的中英双语训练，辅以监督微调、反馈自助、人类反馈强化学习等技术的加持，62 亿参数的 ChatGLM-6B 已经能生成相当符合人类偏好的回答，更多信息请参考我们的博客。

部署

采用秋叶一键部署包，使用8G显存版本

启动成功，访问web-ui

对话

对话流畅度与chatgpt类似，精准度不如gpt4

由于显存限制，咨询过大的问题会直接导致爆显存无法回答

实践-闻达版本

介绍

GitHub - wenda-LLM/wenda: 闻达：一个LLM调用平台。为小模型外挂知识库查找和设计自动执行动作，实现不亚于于大模型的生成能力

闻达基于GLM-6B扩展，支持了本地知识库和联网知识库，提供了近似new bing的体验

部署

使用官方一键包，默认使用6g显存模型

启动成功

对话-离线

只使用已有模型数据，询问

回答内容文不对题

主动提供正确数据内容后再询问，此时回答正确

对话-使用在线知识库

打开联网知识库，清理历史后重新询问，不需要提供本地知识也可以自行通过网络检索后回答正确

系统设计问题

文档解析

直接读取文档

直接使用读取文档并对话的方式很容易脱离文档内容，导致答非所问

尽管使用的是6g显存的版本，读取较大的文档时也会爆显存

手动模拟文档读取

GLM文档分析实际上是将文档识别后转换为特定prompt进行问答对话

参考内嵌prompt

手动复制文档内容进行模拟对话，内容回答正确

类AutoGPT功能

使用wenda提供的“根据标题写论文功能”可自动触发AutoGPT模式

其会自己列提纲，并逐个扩展，无需手工操作

总结

在消费级甜品卡上已经能实现接近chatgpt3的能力，ChatGLM-130B更是拥有1300亿参数的中英双语稠密模型，在能力上更进一步。

未来GLM企业级私有化部署可能会被包装为简单易用的产品向政府、企业推广，满足数据保密要求。

Java 17 生产实践

Fri, 06 May 2022 11:04:00 +0800

背景

版本发布

2021年3月16日 Java 17 作为继Java 11的下一个LTS版本正式发布，JDK 17 作为下一代 LTS 将提供至少到 2026 年的支持。

同业动态

2022年ThoughtWorks 26期技术雷达也提到，建议各个组织将Java 17纳入评估。

评估 82. Java 17

我们通常不会专门介绍某一语言的新版本，但我们还是想关注一下 Java 新的长期支持（LTS）版本——Java 17。尽管出现了前景不错的新特性，比如模式匹配等预览特性，但其实向新 LTS 版本升级的方案更应该吸引各个组织的兴趣。我们建议各个组织在 Java 有新的发布时对其进行评估，确保能够恰当地适配这些新特性和版本。尽管定期更新有利于简化开发并且方便管理，但许多组织却意外地并不经常更新语言的版本。希望通过 LTS 版本的升级以及开发团队对语言的定期更新，能够使生产软件免于因为“更新成本太高”而一直困在 Java 的过时版本上。

框架发展情况

Spring Framework 6和SpringBoot 3.0 将在2022年底正式发布，将基于Java 17构建。

Spring Boot 3.0.0 M1 Release Notes · spring-projects/spring-boot Wiki (github.com)

Minimum Requirements Changes
Spring Boot 3.0 makes the following changes to its minimum supported versions:

Gradle 7.3

Jakarta EE 9

Java 17

Kotlin 1.6

Spring Framework 6

其他团队落地情况

在此之前，我所在的团队使用的Java版本一直停留在Java8，今年经过其他开发团队的尝试，我们团队在4月份也正式决定在生产环境使用Java 17。

实践情况

开发组件版本

本次计划在内部运营管理系统重构中使用Java 17作为基础版本，其他核心组件版本如下

SpringBoot 2.6.6
mapstruct 1.4.2.Final
mybatis-plus 3.5.1
jdk 17.0.1

开发时SpringBoot 3.0还处于M2里程碑阶段，暂时不适合带入生产环境，等未来3.0 GA后再考虑升级

基础镜像

原本使用的基于jdk8 的Docker image也需要替换成基于jdk 17的image

带来的问题：

原本基于jdk8构建并注入到镜像中的opentelemetry-javaagent暂时不可用，等待后续基于jdk17 构建后才能重新启用

JVM参数

由于从Java11开始G1替代CMS作为默认的垃圾收集器，Java14开始移除了CMS GC。

故团队内定义的系统框架配置的GC调优参数基本废弃，并且可以使用Java 14新引入的JVM参数，例如-XX:+ShowCodeDetailsInExceptionMessages

开启前

Exception in thread "main" java.lang.NullPointerException
 at Book.main(Book.java:5)

开启后

Exception in thread "main" java.lang.NullPointerException: 
 Cannot assign field "book" because "shoopingcart.buy" is null
 at Book.main(Book.java:5)

语法特性

从Java8到Java17期间新增的语法特性很多，此处提及几个比较常用的

快速创建不可变集合

不可变对象具有许多优点，包括：

可安全使用不受信任的库。
线程安全。
不需要支持变化判断，所有不可变的集合实现都比其可变的同级实现更节省内存。

Java9开始新增List、Map、Set等of方法快速创建不可变集合

List.of(prodStrategy, instStrategy)

实现源码

 static <E> List<E> of(E e1, E e2) {
 return new ImmutableCollections.List12<>(e1, e2);
 }

Java16开始，Stream类中也新增toList()方法，快速将流转成不可变集合

List<Long> ids = idList.stream().map(ContactId::id).toList();

实现源码

 default List<T> toList() {
 return (List<T>) Collections.unmodifiableList(new ArrayList<>(Arrays.asList(this.toArray())));
 }

注意：如果需要将流转成可变集合，还是需要使用collect(Collectors.toList())方法

record类型

Java14开始新增record类型，用于表示不可变class，实践中可以方便的表示值对象，无需使用@Value注解

public record ContactId(@NotNull(message = "ID不能为空") Long id) {
}

instanceof模式匹配

Java14开始支持模式匹配，无需做强制类型转换

	private void configMessageConverters(List<HttpMessageConverter<?>> converters) {
		converters.removeIf(x -> x instanceof StringHttpMessageConverter || x instanceof MappingJackson2HttpMessageConverter);
		converters.add(new StringHttpMessageConverter(Charsets.UTF_8));
		converters.add(new MappingJackson2HttpMessageConverter(objectMapper));
	}

创建类型安全的流

Java9 开始，新增ofNullable方法创建类型安全的流对象，避免在调用stream方法时报NPE

// query.getDescColumns()可能为null
List<OrderItem> descItems = Stream.ofNullable(query.getDescColumns())
				.flatMap(Collection::stream)
				.map(OrderItem::desc)
				.toList();

性能和安全提升

从同一段功能逻辑中测试发现，Java8和Java17在并发场景下性能差距达到2倍以上

并且同样的逻辑（使用了并发安全的工具）在并发量达到50000的时候在Java8中依然有线程安全问题，而Java17中不会有线程安全问题

外部测评结果

从规划调度引擎 OptaPlanner 项目（原文 (opens new window)）对 JDK 17和 JDK 11 的性能基准测试进行了对比来看：

对于 G1GC（默认），Java 17 比 Java 11 快 8.66%；

对于 ParallelGC，Java 17 比 Java 11 快 6.54%；

Parallel GC 整体比 G1 GC 快 16.39%

简而言之，JDK17 更快，高吞吐量垃圾回收器比低延迟垃圾回收器更快。

GraphQL实践2——Spring-GraphQL集成JPA与MySQL

Mon, 15 Nov 2021 15:38:00 +0800

介绍

上一篇介绍到用第三方库集成GraphQL，目前spring-graphql项目已经出到1.0.0-M3版本，属于内部预览版，此处尝鲜验证

引入依赖

由于SpringBoot 2.6.0还未发布，因此需要引入较多依赖

<dependencies>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 </dependency>
 <dependency>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 <optional>true</optional>
 </dependency>
 <dependency>
 <groupId>org.springframework.experimental</groupId>
 <artifactId>graphql-spring-boot-starter</artifactId>
 <version>1.0.0-M3</version>
 </dependency>
 <!--升级到Spring Boot 2.6后可以移除-->
 <dependency>
 <groupId>org.springframework.data</groupId>
 <artifactId>spring-data-commons</artifactId>
 <version>2.6.0-RC1</version>
 </dependency>
 <dependency>
 <groupId>org.springframework.data</groupId>
 <artifactId>spring-data-jpa</artifactId>
 <version>2.6.0-RC1</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/org.hibernate.javax.persistence/hibernate-jpa-2.1-api -->
 <dependency>
 <groupId>org.hibernate.javax.persistence</groupId>
 <artifactId>hibernate-jpa-2.1-api</artifactId>
 <version>1.0.2.Final</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/com.querydsl/querydsl-jpa -->
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-jpa</artifactId>
 <version>4.4.0</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/com.querydsl/querydsl-apt -->
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-apt</artifactId>
 <version>4.4.0</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/javax.annotation/javax.annotation-api -->
 <dependency>
 <groupId>javax.annotation</groupId>
 <artifactId>javax.annotation-api</artifactId>
 </dependency>
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-core</artifactId>
 </dependency>
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-jpa</artifactId>
 </dependency>
 <dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <scope>runtime</scope>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-data-jpa</artifactId>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-test</artifactId>
 <scope>test</scope>
 </dependency>
</dependencies>

引入里程碑仓库

由于还没有GA，所以要引入里程碑仓库

<repositories>
 <repository>
 <id>spring-milestones</id>
 <name>Spring Milestones</name>
 <url>https://repo.spring.io/milestone</url>
 </repository>
 <repository>
 <id>spring-snapshots</id>
 <name>Spring Snapshots</name>
 <url>https://repo.spring.io/snapshot</url>
 <snapshots>
 <enabled>true</enabled>
 </snapshots>
 </repository>
</repositories>

引入插件

由于使用了dsl动态生成，还要引入注解处理器

<build>
 <plugins>
 <plugin>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-maven-plugin</artifactId>
 <configuration>
 <excludes>
 <exclude>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 </exclude>
 </excludes>
 </configuration>
 </plugin>
 <!-- added for QueryDsl-->
 <plugin>
 <groupId>com.mysema.maven</groupId>
 <artifactId>apt-maven-plugin</artifactId>
 <version>1.1.3</version>
 <executions>
 <execution>
 <goals>
 <goal>process</goal>
 </goals>
 <configuration>
 <outputDirectory>target/generated-sources/java</outputDirectory>
 <processor>com.querydsl.apt.jpa.JPAAnnotationProcessor</processor>
 </configuration>
 </execution>
 </executions>
 </plugin>
 </plugins>
</build>

SpringBoot 配置

编写配置文件

spring:
 datasource:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: root
 password: "****"
 url: jdbc:mysql://localhost:3306/sakila?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
 jpa:
 hibernate:
 ddl-auto: none
 show-sql: true
 database-platform: org.hibernate.dialect.MySQL55Dialect
 properties:
 hibernate:
 format_sql: true
 use_sql_comments: true
 graphql:
 path: /graphql
 schema:
 locations: classpath:graphql/
 fileExtensions: .graphqls, .gqls
 printer:
 enabled: true
server:
 port: 8080

实体类

package top.fjy8018.graphsqldemo.entity;

import lombok.Data;

import javax.persistence.*;
import java.sql.Timestamp;
import java.util.Objects;

/**
 * 演员表实体类
 *
 * @author F嘉阳
 * @date 2021/11/5 10:34
 */
@Data
@Entity
@Table(name = "actor", schema = "sakila")
public class ActorEntity {
 @Id
 @Column(name = "actor_id", nullable = false)
 private Integer actorId;

 @Column(name = "first_name", nullable = false, length = 45)
 private String firstName;

 @Column(name = "last_name", nullable = false, length = 45)
 private String lastName;

 @Column(name = "last_update", nullable = false)
 private Date lastUpdate;

}

DAO

DAO直接继承dsl相关处理器，可自动配对增删改查方法，相比第三方库更加简便

package top.fjy8018.graphsqldemo.repository;

import org.springframework.data.querydsl.QuerydslPredicateExecutor;
import org.springframework.data.repository.CrudRepository;
import org.springframework.graphql.data.GraphQlRepository;
import top.fjy8018.graphsqldemo.entity.ActorEntity;

/**
 * 演员表DAO
 *
 * @author F嘉阳
 * @date 2021/11/5 10:35
 */
@GraphQlRepository
public interface ActorEntityRepository extends
 CrudRepository<ActorEntity, Integer>, QuerydslPredicateExecutor<ActorEntity> {
}

Graphql资源定义

在resources/graphql下定义接口和实体文件

接口声明和实体定义schema.graphqls

type Query {
 findOneActor(actorId : ID!): ActorEntity
 actorList: [ActorEntity]
}

type ActorEntity {
 actorId: ID!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

接口定义acotrEntity.graphql

query findOneActor($id: ID!) {
 findOneActor(actorId: $id) {
 actorId
 firstName
 lastName
 lastUpdate
 }
}

query actorList {
 actorList{
 actorId
 }
}

启动测试

Spring-Graphql默认不包含可视化界面，此处使用postman进行测试

唯一查询

列表查询

总结

使用Spring官方组件好处在于和Spring生态集成度很高，如果本身就采用Jpa方式进行业务开发，迁移更加方便，需要开发的代码也很少。

样例源码地址

FJiayang/graphql-demo

GraphQL实践1——集成JPA与MySQL

Fri, 05 Nov 2021 17:41:00 +0800

介绍

官方定义：

GraphQL 既是一种用于 API 的查询语言也是一个满足你数据查询的运行时。 GraphQL 对你的 API 中的数据提供了一套易于理解的完整描述，使得客户端能够准确地获得它需要的数据，而且没有任何冗余，也让 API 更容易地随着时间推移而演进，还能用于构建强大的开发者工具。

中文官网：

GraphQL — A query language for your API

集成方案选择

GraphQL经过近几年的发展，SpringBoot官方、第三方都提供了starter

SpringBoot官方提供的starter

Spring GraphQL

第三方提供的starter

graphql-spring-boot

两者活跃度都还不错，但考虑到SpringBoot官方的依赖库还未GA，而且使用起来与Spring家族耦合过大，最终决定采用第三方的starter

集成过程

数据库配置

此处采用MySQL数据库，数据内容采用MySQL官方样例sakila数据，表结构和数据地址如下

sakila database下载地址

导入后如下

工程配置

引入最新的SpringBoot框架，依赖如下

<dependencies>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 </dependency>

 <dependency>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 <optional>true</optional>
 </dependency>

 <!-- https://mvnrepository.com/artifact/com.graphql-java-kickstart/graphql-spring-boot-starter -->
 <dependency>
 <groupId>com.graphql-java-kickstart</groupId>
 <artifactId>graphql-spring-boot-starter</artifactId>
 <version>12.0.0</version>
 </dependency>

 <dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <scope>runtime</scope>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-data-jpa</artifactId>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-test</artifactId>
 <scope>test</scope>
 </dependency>
</dependencies>

增加数据库配置

spring:
 datasource:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: root
 password: "****"
 url: jdbc:mysql://localhost:3306/sakila?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
 jpa:
 hibernate:
 ddl-auto: update
 show-sql: true
 database-platform: org.hibernate.dialect.MySQL55Dialect
 properties:
 hibernate:
 format_sql: true
 use_sql_comments: true
server:
 port: 8080
graphql:
 tools:
 schema-location-pattern: graphql/*.graphqls
 graphiql:
 # 开启可交互页面，用于调试
 enabled: true

实体映射

新增实体映射，此处只映射Actor表作为样例

package top.fjy8018.graphsqldemo.entity;

import lombok.Data;

import javax.persistence.*;
import java.sql.Timestamp;
import java.util.Objects;

/**
 * 演员表实体映射
 *
 * @author F嘉阳
 * @date 2021/11/5 10:34
 */
@Data
@Entity
@Table(name = "actor", schema = "sakila")
public class ActorEntity {
 @Id
 @Column(name = "actor_id", nullable = false)
 private Integer actorId;

 @Column(name = "first_name", nullable = false, length = 45)
 private String firstName;

 @Column(name = "last_name", nullable = false, length = 45)
 private String lastName;

 @Column(name = "last_update", nullable = false)
 private Timestamp lastUpdate;
}

新增DAO

package top.fjy8018.graphsqldemo.repository;

import org.springframework.data.jpa.repository.JpaRepository;
import top.fjy8018.graphsqldemo.entity.ActorEntity;

/**
 * 演员表DAO
 *
 * @author F嘉阳
 * @date 2021/11/5 10:35
 */
public interface ActorEntityRepository extends JpaRepository<ActorEntity, Integer> {
}

新增查询接口配置

在resources/graphql下新增文件actor.graphqls

type Query {
 actorList: [ActorEntity]
 findOneActor(id : ID!): ActorEntity
}

type ActorEntity {
 actorId: ID!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

新增查询解析器

package top.fjy8018.graphsqldemo.resolver;

import graphql.kickstart.tools.GraphQLQueryResolver;
import lombok.AllArgsConstructor;
import org.springframework.stereotype.Component;
import top.fjy8018.graphsqldemo.entity.ActorEntity;
import top.fjy8018.graphsqldemo.repository.ActorEntityRepository;

import java.util.Collection;

/**
 * 演员表GraphQL查询解析器
 *
 * @author F嘉阳
 * @date 2021/11/5 16:27
 */
@Component
@AllArgsConstructor
public class ActorGraphQLQueryResolver implements GraphQLQueryResolver {

 private final ActorEntityRepository actorRepository;

 public Collection<ActorEntity> actorList() {
 return actorRepository.findAll();
 }

 public ActorEntity findOneActor(Integer id) {
 return actorRepository.findById(id).orElse(null);
 }
}

启动验证

启动工程后，浏览器打开http://localhost:8080/graphiql

列表查询

由于采用交互式页面，输入也会有语法提示，比较方便

唯一查询

优缺点分析

优点

接口定义清晰，配置即接口
前端按需获取字段，减少数据传输量
接口整合、演进更加便利

缺点

减少字段个数对后端实现没有变化，并不能提升数据库性能

样例源码地址

FJiayang/graphql-demo

Kubesphere落地DevOps

Sat, 19 Jun 2021 22:16:00 +0800

创建企业空间

创建一个企业空间和一个帐户 (project-admin)，必须邀请该帐户至该企业空间并赋予 workspace-self-provisioner 角色。

进入企业空间，切换到DevOps工程，点击创建

创建工程

安装helm

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

找不到环境变量报错

安装helm遇到报错

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
 % Total % Received % Xferd Average Speed Time Time Time Current
unset -f pathmunge
 Dload Upload Total Spent Left Speed
100 11248 100 11248 0 0 61543 0 --:--:-- --:--:-- --:--:-- 61802
Helm v3.6.0 is available. Changing from version v3.2.1.
Downloading https://get.helm.sh/helm-v3.6.0-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
helm not found. Is /usr/local/bin on your $PATH?
Failed to install helm
 For support, go to https://github.com/helm/helm.

则需要配置环境变量

$ vi /etc/profile
# 添加配置
export PATH=/usr/local/bin:$PATH

$ source /etc/profile

验证

$ helm version
version.BuildInfo{Version:"v3.6.0", GitCommit:"7f2df6467771a75f5646b7f12afb408590ed1755", GitTreeState:"clean", GoVersion:"go1.16.3"}

安装SonarQube

执行以下命令安装 SonarQube 服务器

$ helm upgrade --install sonarqube sonarqube --repo https://charts.kubesphere.io/main -n kubesphere-devops-system --create-namespace --set service.type=NodePort

输出

执行以下命令以获取 SonarQube NodePort

$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
$ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo http://$NODE_IP:$NODE_PORT

输出

此时可在kubesphere控制台中看到sonar服务

配置 SonarQube 服务器

访问地址，点击右上角的 Log in，然后使用默认帐户 admin/admin 登录。

创建 SonarQube 管理员令牌 (Token)

点击右上角字母 A，然后从菜单中选择 My Account 以转到 Profile 页面。

生成token

创建 Webhook 服务器

执行以下命令获取 SonarQube Webhook 的地址。

$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services ks-jenkins)
$ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo http://$NODE_IP:$NODE_PORT/sonarqube-webhook/

输出

依次点击 Administration、Configuration 和 Webhooks 创建一个 Webhook。

点击 Create

由于webhook都通过内部网络调用，故可以使用内部网络地址

将 SonarQube 配置添加到 ks-installer

执行以下命令编辑 ks-installer

$ kubectl edit cc -n kubesphere-system ks-installer

搜寻至 devops。添加字段 sonarqube 并在其下方指定 externalSonarUrl 和 externalSonarToken。

 devops:
 enabled: true
 jenkinsJavaOpts_MaxRAM: 2g
 jenkinsJavaOpts_Xms: 512m
 jenkinsJavaOpts_Xmx: 512m
 jenkinsMemoryLim: 2Gi
 jenkinsMemoryReq: 1500Mi
 jenkinsVolumeSize: 8Gi
 sonarqube:
 externalSonarUrl: http://10.170.0.5:32108
 externalSonarToken: c6a6e2e15a9d52e30b744df0eef4c639bbea88a1

完成操作后保存此文件。

将 SonarQube 服务器添加至 Jenkins

执行以下命令获取 Jenkins 的地址。

$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services ks-jenkins)
$ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo http://$NODE_IP:$NODE_PORT

输出

使用地址 http://{$Public IP}:30180 访问 Jenkins。安装 KubeSphere 时，默认情况下也会安装 Jenkins 仪表板。此外，Jenkins 还配置有 KubeSphere LDAP，这意味着您可以直接使用 KubeSphere 帐户（例如 admin/P@88w0rd）登录 Jenkins。

使用kubesphere admin账户登录

前往系统管理下的 Manage Credentials 并点击 Stores scoped to Jenkins 下的 Jenkins，再点击全局凭据 (unrestricted)

点击左侧导航栏的添加凭据，用 SonarQube 管理员令牌添加凭证

点击左侧的系统管理，向下翻页找到并点击系统配置。

搜寻到 SonarQube servers，然后点击 Add SonarQube。

添加sonar服务器地址和名称，点击应用完成操作。

将 sonarqubeURL 添加到 KubeSphere 控制台

指定 sonarqubeURL，以便可以直接从 KubeSphere 控制台访问 SonarQube。

执行以下命令：

$ kubectl edit cm -n kubesphere-system ks-console-config

搜寻到 client，添加 devops 字段并指定 sonarqubeURL。

 client:
 version:
 kubesphere: v3.1.0
 kubernetes: v1.20.4
 openpitrix: v0.3.5
 enableKubeConfig: true
 devops:
 sonarqubeURL: http://34.92.90.181:32108

结果

重启服务

执行以下命令。

$ kubectl -n kubesphere-system rollout restart deploy ks-apiserver
$ kubectl -n kubesphere-system rollout restart deploy ks-console

为新工程创建 SonarQube Token

创建一个 SonarQube 令牌，以便您的流水线可以在运行时与 SonarQube 通信。

在 SonarQube 控制台上，点击 Create new project。

设置工程名称

生成token

创建令牌后，点击 Continue。

分别选择 Java 和 Maven。复制下图所示绿色框中的序列号，如果要在流水线中使用，则需要在凭证中添加此序列号。

创建流水线工程

创建凭证

凭证 ID	类型	用途
dockerhub-id	帐户凭证	Docker Hub
demo-kubeconfig	kubeconfig	Kubernetes

demo-kubeconfig

dockerhub-id

创建一个凭证 ID (sonar-token)

列表中看到已创建的三个凭证。

创建流水线

在流水线页面点击创建，为了使用图形化流水线搭建，不在此处引用代码仓库，直接点击下一步

保持默认配置，点击创建

创建成功

进入流水线，点击编辑

此时选择CICD模板

可视化编辑流水线

编辑后运行

流水线支持交互式操作

在 KubeSphere 控制台查看结果

流水线执行成功后，即可查看到代码质量标签

sonar控制台也可以查看对应结果

jenkins视图

blue ocean

Kubesphere运维Kubernetes集群

Sun, 23 May 2021 22:58:00 +0800

背景

使用原生Kubernetes-dashboard和Grafana+Prometheus管理较大规模集群比较困难，有如下缺陷

每次执行命令都要登陆服务器
容器日志采集方案、监控方案搭建比较繁琐
添加节点不方便
管理存储插件、秘钥、配置不方便

解决方案

目前国内流行的k8s管理平台主要有rancher、kubesphere、kubeboard

rancher主打原生集群管理，侧重运维人员
kubesphere主打开发运维工作流，侧重研发人员
kubeboard与kubesphere类似，侧重微服务管理，但功能上较少，对多节点管理需要收费

经过评估，最终决定采用kubesphere进行集群管理

Rancher — 多云混合云多集群Kubernetes管理平台

[post url=“https://kubesphere.io/zh/” title=“Kubesphere” intro=“面向云原生应用的容器混合云” cover=“https://kubesphere.io/images/logo.svg” /]

Kuboard — 快速在 Kubernetes 上落地微服务

安装步骤

目标是面对小规模集群管理，节点数最多为5个，故不考虑高可用集群配置，采用单主节点+4个worker节点配置

采用纯净CentOS 7 Linux系统，需要做以下前置准备

依赖包安装

安装官方要求的依赖包，由于计划安装NFS存储插件，还需要安装nfs工具包

$ yum install -y socat conntrack ebtables ipset nfs-utils

配置DNS解析

将各节点DNS配置为私有DNS服务器

$ tee /etc/resolv.conf <<-'EOF'
# Generated by NetworkManager
nameserver 192.168.1.13
EOF

私有DNS搭建可以参考文章

[[利用Docker自建Dns服务器]]

按照官方教程安装多节点kubernetes

强烈建议先用KubeKey安装纯净k8s集群+控制面板，关闭所有插件。等安装完毕再一个个插件打开，防止因为内存不够而无法打开kubesphere

安装步骤参考官方文档一键安装方式

多节点安装 — Kubesphere

启动成功

启动成功后按照官方指南打开插件

启用可插拔组件 — Kubesphere

所有插件打开后如下

使用

节点状态

存储卷管理

监控告警

日志采集

日志查询

操作审计

工具箱

企业空间权限控制

内置helm应用商店，支持上线审批，在线编辑等

支持关联外部仓库

应用流量治理

链路追踪

灰度发布

费用统计

更多的使用场景包括：DevOps、边缘节点、多集群管理还在探索中

SpringCloudStream函数式编程模型

Sun, 21 Feb 2021 12:22:00 +0800

背景

SpringCloud 3.1版本之后，@EnableBinding、@Output等StreamApi注解都标记为废弃

/**
 * Indicates that an output binding target will be created by the framework.
 *
 * @author Dave Syer
 * @author Marius Bogoevici
 * @author Artem Bilan
 *
 * @deprecated as of 3.1 in favor of functional programming model
 */

@Qualifier
@Target({ ElementType.FIELD, ElementType.METHOD, ElementType.ANNOTATION_TYPE,
		ElementType.PARAMETER })
@Retention(RetentionPolicy.RUNTIME)
@Inherited
@Documented
@Deprecated
public @interface Output {

	/**
	 * Specify the binding target name; used as a bean name for binding target and as a
	 * destination name by default.
	 * @return the binding target name
	 */
	String value() default "";

}

/**
 * Enables the binding of targets annotated with {@link Input} and {@link Output} to a
 * broker, according to the list of interfaces passed as value to the annotation.
 *
 * @author Dave Syer
 * @author Marius Bogoevici
 * @author David Turanski
 * @author Soby Chacko
 *
 * @deprecated as of 3.1 in favor of functional programming model
 */
@Target({ ElementType.TYPE, ElementType.ANNOTATION_TYPE })
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@Configuration
@Import({ BindingBeansRegistrar.class, BinderFactoryAutoConfiguration.class })
@EnableIntegration
@Deprecated
public @interface EnableBinding {

	/**
	 * A list of interfaces having methods annotated with {@link Input} and/or
	 * {@link Output} to indicate binding targets.
	 * @return list of interfaces
	 */
	Class<?>[] value() default {};

}

Producing and Consuming Messages — SpringCloud 官方文档

@EnableBinding @deprecated as of 3.1 in favor of functional programming model — stackoverflow

The type EnableBinding is deprecated — stackoverflow

因此，所有采用命令式编程的API，在升级到新版本时，官方推荐使用更加简单的函数式编程API

完整代码样例

FJiayang/spring-cloud-stream-rabbit-example — GitHub项目地址

升级方式

命令式编程模型

以简单多Topic消息为例，原本的编程方式如下

生产者

@SpringBootApplication
public class ProducerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ProducerApplication.class, args);
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 17:57
 */
@RestController
@EnableBinding(MySource.class)
public class Producer {

 @Autowired
 private MySource channel;

 @RequestMapping("/send")
 public String send() {
 channel.output().send(MessageBuilder.withPayload(new Date()).build());
 return "success";
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 18:01
 */
public interface MySource {
 String OUTPUT = "output";

 @Output(MySource.OUTPUT)
 MessageChannel output();
}

配置文件

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin
 cloud:
 stream:
 bindings:
 output:
 destination: my-test-channel
server:
 port: 8082

消费者

@SpringBootApplication
public class ConsumerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ConsumerApplication.class, args);
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 18:10
 */
@EnableBinding(MySink.class)
public class Consumer {
 @StreamListener(MySink.INPUT)
 public void receive(Message<String> message) {
 System.out.println("接收到MQ消息:" + message.getPayload());
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 18:07
 */
public interface MySink {
 String INPUT = "input";

 @Input(MySink.INPUT)
 SubscribableChannel input();
}

配置文件

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin
 cloud:
 stream:
 bindings:
 input:
 destination: my-test-channel
server:
 port: 8081

函数式编程模型

从命令式升级到函数式编程，首先需要升级SpringCloud版本至3.1以上，此处升级到最新版2020.0.1

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>

 <groupId>top.fjy8018</groupId>
 <artifactId>cloud-stream</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>pom</packaging>

 <name>cloud-stream</name>
 <description>Demo project for Spring Boot</description>

 <modules>
 <module>producer</module>
 <module>consumer</module>
 </modules>

 <parent>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-parent</artifactId>
 <version>2.4.2</version>
 <relativePath/> <!-- lookup parent from repository -->
 </parent>

 <properties>
 <spring-cloud.version>2020.0.1</spring-cloud.version>
 </properties>

 <dependencyManagement>
 <dependencies>
 <dependency>
 <groupId>org.springframework.cloud</groupId>
 <artifactId>spring-cloud-dependencies</artifactId>
 <version>${spring-cloud.version}</version>
 <type>pom</type>
 <scope>import</scope>
 </dependency>
 </dependencies>
 </dependencyManagement>
</project>

目前国内有清晰的升级指引的还比较少，目前只能参考官方文档材料

Programming Model — SpringCloud官方文档

模型1——系统定时驱动的消息队列

官方文档描述

Suppliers (Sources)
Function and Consumer are pretty straightforward when it comes to how their invocation is triggered. They are triggered based on data (events) sent to the destination they are bound to. In other words, they are classic event-driven components.

However, Supplier is in its own category when it comes to triggering. Since it is, by definition, the source (the origin) of the data, it does not subscribe to any in-bound destination and, therefore, has to be triggered by some other mechanism(s). There is also a question of Supplier implementation, which could be imperative or reactive and which directly relates to the triggering of such suppliers.

Consider the following sample:
@SpringBootApplication
public static class SupplierConfiguration {

	@Bean
	public Supplier<String> stringSupplier() {
		return () -> "Hello from Supplier";
	}
}
The preceding Supplier bean produces a string whenever its get() method is invoked. However, who invokes this method and how often? The framework provides a default polling mechanism (answering the question of “Who?”) that will trigger the invocation of the supplier and by default it will do so every second (answering the question of “How often?”). In other words, the above configuration produces a single message every second and each message is sent to an output destination that is exposed by the binder.

其大意为，若采用默认的函数式编程模型，会每秒钟发送一条消息到消费者，定时发送参数可以配置

按照文档升级后如下

生产者

@SpringBootApplication
public class ProducerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ProducerApplication.class, args);
 }

 @Bean
 public Supplier<Date> source1() {
 return () -> new Date();
 }
}

函数式编程不需要在代码中定义监听队列，只需要按照约定写在配置文件中即可

官方文档描述

Functional binding names
Unlike the explicit naming required by annotation-based support (legacy) used in the previous versions of spring-cloud-stream, the functional programming model defaults to a simple convention when it comes to binding names, thus greatly simplifying application configuration. Let’s look at the first example:
@SpringBootApplication
public class SampleApplication {

	@Bean
	public Function<String, String> uppercase() {
	 return value -> value.toUpperCase();
	}
}
In the preceding example we have an application with a single function which acts as message handler. As a Function it has an input and output. The naming convention used to name input and output bindings is as follows:

input - <functionName> + -in- + <index>

output - <functionName> + -out- + <index>

The in and out corresponds to the type of binding (such as input or output). The index is the index of the input or output binding. It is always 0 for typical single input/output function, so it’s only relevant for Functions with multiple input and output arguments.

So if for example you would want to map the input of this function to a remote destination (e.g., topic, queue etc) called “my-topic” you would do so with the following property:
--spring.cloud.stream.bindings.uppercase-in-0.destination=my-topic
Note how uppercase-in-0 is used as a segment in property name. The same goes for uppercase-out-0.

配置文件

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin

 cloud:
 stream:
 bindings:
 source1-out-0:
 destination: test1
 function:
 definition: source1

server:
 port: 8083

由此可见，整体配置简化了很多，多为规约大于配置的方式进行定义，特别注意function配置的位置，是在stream配置的同级

消费者

@SpringBootApplication
public class ConsumerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ConsumerApplication.class, args);
 }

 @Bean
 public Consumer<Date> sink1() {
 return System.out::println;
 }
}

配置文件同理

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin

 cloud:
 stream:
 bindings:
 sink1-in-0:
 destination: test1
 function:
 definition: sink1


server:
 port: 8081

运行结果

生产者每隔一秒发送一次消息

模型2——`StreamBridge`

但实际生产中，更多是由业务场景触发，故不能使用该模式进行配置，SpringCloud还提供了另外一种StreamBridge模式

官方文档介绍

Sending arbitrary data to an output (e.g. Foreign event-driven sources)
There are cases where the actual source of data may be coming from the external (foreign) system that is not a binder. For example, the source of the data may be a classic REST endpoint. How do we bridge such source with the functional mechanism used by spring-cloud-stream?

Spring Cloud Stream provides two mechanisms, so let’s look at them in more details

Here, for both samples we’ll use a standard MVC endpoint method called delegateToSupplier bound to the root web context, delegating incoming requests to stream via two different mechanisms - imperative (via StreamBridge) and reactive (via EmitterProcessor).

大意是，实际的数据源可以是外部事件驱动，例如通过Rest接口

官方样例

Using StreamBridge
@SpringBootApplication
@Controller
public class WebSourceApplication {

	public static void main(String[] args) {
		SpringApplication.run(WebSourceApplication.class, "--spring.cloud.stream.source=toStream");
	}

	@Autowired
	private StreamBridge streamBridge;

	@RequestMapping
	@ResponseStatus(HttpStatus.ACCEPTED)
	public void delegateToSupplier(@RequestBody String body) {
		System.out.println("Sending " + body);
		streamBridge.send("toStream-out-0", body);
	}
}
Here we autowire a StreamBridge bean which allows us to send data to an output binding effectively bridging non-stream application with spring-cloud-stream. Note that preceding example does not have any source functions defined (e.g., Supplier bean) leaving the framework with no trigger to create source bindings, which would be typical for cases where configuration contains function beans. So to trigger the creation of source binding we use spring.cloud.stream.source property where you can declare the name of your sources. The provided name will be used as a trigger to create a source binding. So in the preceding example the name of the output binding will be toStream-out-0 which is consistent with the binding naming convention used by functions (see Binding and Binding names). You can use ; to signify multiple sources (e.g., --spring.cloud.stream.source=foo;bar)

参考官方文档改造后

生产者

这里直接贴出多主题的消费模式，实际生产中多数也是订阅多个主题

@SpringBootApplication
public class ProducerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ProducerApplication.class, args);
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 17:57
 */
@RestController
public class Producer {

 @Autowired
 private StreamBridge streamBridge;

 @RequestMapping("/send1")
 public String send1() {
 streamBridge.send("source1-out-0", new Date());
 return "success1";
 }

 @RequestMapping("/send2")
 public String send2() {
 streamBridge.send("source2-out-0", LocalDateTime.now().format(DateTimeFormatter.ISO_LOCAL_DATE_TIME));
 return "success2";
 }

}

配置文件

spring:
 rabbitmq:
 host: 192.168.133.128
 username: dev-user
 password: devpassword

 cloud:
 stream:
 bindings:
 source1-out-0:
 destination: test2
 source2-out-0:
 destination: test3
 function:
 definition: source1;source2


server:
 port: 8083

消费者

@SpringBootApplication
public class ConsumerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ConsumerApplication.class, args);
 }

 @Bean
 public Consumer<Date> sink1() {
 return System.out::println;
 }

 @Bean
 public Consumer<String> sink2() {
 return System.out::println;
 }
}

配置文件

spring:
 rabbitmq:
 host: 192.168.133.128
 username: dev-user
 password: devpassword

 cloud:
 stream:
 bindings:
 sink1-in-0:
 destination: test2
 sink2-in-0:
 destination: test3
 function:
 definition: sink1;sink2

server:
 port: 8081

运行结果

触发/send1

触发/send2

可见消息被正确的路由

总结

整体来看，函数式编程比命令式编程更加简洁，而且加上SpringCloud规约大于配置的特点，可以明显减少配置的代码量，是未来发展的趋势。

RocketMQ学习笔记6——集群实战

Wed, 23 Dec 2020 19:15:00 +0800

组建高性能MQ服务

双主双从模式
保障无单点故障问题的高可用集群

集群拓扑图

IP	资源	服务
10.139.12.52	1C2G20GB	rocketmq-nameserver1、rocketmq-master1
10.139.12.53	1C2G20GB	rocketmq-nameserver2、rocketmq-master2
10.139.12.54	1C2G20GB	rocketmq-nameserver3、rocketmq-master1-slave
10.139.12.55	1C2G20GB	rocketmq-nameserver4、rocketmqmaster2-slave

配置文件最优设置解析

修改host文件映射

10.139.12.52 rocketmq-nameserver1 rocketmq-master1
10.139.12.53 rocketmq-nameserver2 rocketmq-master2
10.139.12.54 rocketmq-nameserver3 rocketmq-master1-slave
10.139.12.55 rocketmq-nameserver4 rocketmq-master2-slave

将4个节点的hosts均改为如上内容，并将4.4.0版本包放入环境中解压

创建存储目录

$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index

建立软连接

$ cd ~/rocketmq-test
$ ln -s rocketmq-all-4.4.0-bin-release rocketmq

修改配置文件

解压后默认的配置文件如下

[user@301 ~/rocketmq-test/rocketmq/conf]$ll
total 36
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync
drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave
-rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf
-rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml
-rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml
-rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml
-rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml
-rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml

若想保障生产环境高可用，推荐使用双主双从同步双写模式，可以让一个主节点对应多个从节点，但从节点越多，同步效率越慢

本次采用双主双从同步双写模式，一个主节点对应一个从节点

[user@301 ~/rocketmq-test/rocketmq/conf/2m-2s-sync]$ll
total 16
-rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-a.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties
-rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-b.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties

用途如下

IP	资源	服务	配置文件
10.139.12.52	1C2G20GB	rocketmq-nameserver1、rocketmq-master1	broker-a.properties
10.139.12.53	1C2G20GB	rocketmq-nameserver2、rocketmq-master2	broker-b.properties
10.139.12.54	1C2G20GB	rocketmq-nameserver3、rocketmq-slave1	broker-a-s.properties
10.139.12.55	1C2G20GB	rocketmq-nameserver4、rocketmq-slave2	broker-b-s.properties

`broker-a.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=false
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=false
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间，最大100%
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-b.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-b
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=false
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=false
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间，最大100%
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-a-s.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-b-s.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对
brokerName=broker-b
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

分发配置文件

$ scp broker-* ocmuser@rocketmq-master2-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync
$ scp broker-* ocmuser@rocketmq-master2:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync
$ scp broker-* ocmuser@rocketmq-master1-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync

修改日志配置文件

$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq/logs
$ cd /home/ocmuser/rocketmq-test/rocketmq/conf && sed -i 's#${user.home}#/home/ocmuser/rocketmq-test/#g' *.xml

修改脚本启动参数

/home/ocmuser/rocketmq-test/rocketmq/bin/runbroker.sh内容如下，至少保障1g内存

#======================================================================================
# 开发环境JVM Configuration
#======================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"

/home/ocmuser/rocketmq-test/rocketmq/bin/runserver.sh内容如下

#======================================================================================
# JVM Configuration
#======================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

启动 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin
$ nohup sh mqnamesrv &

启动 Master1:BrokerServerA

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master2:BrokerServerB

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master1-Slave:BrokerServerC

$ cd cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a-s.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master2-Slave:BrokerServerD

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b-s.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

查看进程和日志，可见部署完成

角色为同步双写，异步刷盘，4个节点通信正常

控制台监控验证

修改连接配置

rocketmq.config.namesrvAddr=10.139.12.51:9876;10.139.12.52:9876;10.139.12.53:9876;10.139.12.54:9876

连接正常

查看各节点角色和状态

服务停止

首先关闭 4 个 BrokerServer，再关闭 4 个 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ sh mqshutdown broker 
$ sh mqshutdown namesrv

双主双从环境构建与调优

为取得可靠性与效率的平衡，建议采用同步双写，异步刷盘的模式

RocketMQ学习笔记5——原理解析

Wed, 23 Dec 2020 19:13:00 +0800

消息存储结构

CommitLog：消息主体以及元数据的存储主体，存储Producer端写入的消息主体内容,消息内容不是定长的。消息主要是顺序写入日志文件，当文件满了，写入下一个文件；其日志信息科用于数据恢复，作为MQ数据保障兜底策略。
ConsumerQueue：消息消费队列，存储消息在commit log 中所在的这个位置，也可以说是一个偏移量.引入的目的主要是提高消息消费的性能。
IndexFile：IndexFile（索引文件）提供了一种可以通过key或时间区间来查询消息的方法。
每一个topic下的每一个message queue都对应一个ConsumerQueue文件，具体存储位置通过配置文件配置。
RocketMQ采用的是混合型的存储结构，即为Broker单个实例下所有的队列（ConsumerQueue）共用一个日志数据文件（即为CommitLog）来存储。
刷盘采用顺序写随机读的模式。
消费端采用“0拷贝”模式提升性能。

同步\异步刷盘

消息存储：内存+磁盘存储，两种刷盘方式

异步刷盘

能够充分利用OS的PageCache的优势，只要消息写入PageCache（内存）即可将成功的ACK（成功信息）返回给Producer端。
消息堆积到一定量才触发刷盘。
消息刷盘采用后台异步线程提交的方式进行，降低了读写延迟，提高了MQ的性能和吞吐量。

同步刷盘

只有在消息真正持久化至磁盘后RocketMQ的Broker端才会真正返回给Producer端一个成功的ACK响应。
同步刷盘对MQ消息可靠性来说是一种不错的保障，但是性能上会有较大影响，一般适用于金融业务应用该模式较多。

同步\异步复制

针对集群模式设计，同一组Broker有Master-Slave角色
推荐采用同步双写，异步复制的模式保持性能与可靠性的平衡

同步复制

Master与Slave都成功才返回ACK（成功信息）给Producer

异步复制

Master节点刷盘成功后立即返回ACK（成功信息）给Producer
再通过异步线程将信息同步给其他Slave节点
高性能，但存在数据丢失风险

同步双写

每个Master配置一个Slave，有多对Master-Slave，HA采用同步双写方式，即只有主备都写成功，才向应用返回成功，这种模式的优缺点如下：

优点：数据与服务都无单点故障，Master宕机情况下，消息无延迟，服务可用性与数据可用性都非常高；
缺点：性能比异步复制模式略低（大约低10%左右），发送单个消息的RT会略高，且目前版本在主节点宕机后，备机不能自动切换为主机。

高可用机制

Master-Slave搭配实现高可用机制
通过BrokerId区分节点
Master支持读写，Slave只读
当Master繁忙或者不可用时，可以自动切换到Slave读取消息

最佳实践

创建Topic时，将Topic的Message Queue创建到多个Broker组中，使得逻辑上高可用
由于RocketMQ不支持主从切换，故可以考虑使用监控程序监听主节点的有效性，若连接超时，则将Slave节点的配置改为Master并重启，升级为主节点，实现高可用。

NameServer协调者

NameServer是一个非常简单的Topic路由注册中心，其角色类似Dubbo中的zookeeper，支持Broker的动态注册与发现，是整个集群的状态服务器。
主要包括两个功能：
- Broker管理，NameServer接受Broker集群的注册信息并且保存下来作为路由信息的基本数据。然后提供心跳检测机制，检查Broker是否还存活；
- 路由信息管理，每个NameServer将保存关于Broker集群的整个路由信息和用于客户端查询的队列信息。然后Producer和Conumser通过NameServer就可以知道整个Broker集群的路由信息，从而进行消息的投递和消费。
NameServer通常也是集群的方式部署，各实例间相互不进行信息通讯。Broker是向每一台NameServer注册自己的路由信息，所以每一个NameServer实例上面都保存一份完整的路由信息。
当某个NameServer因某种原因下线了，Broker仍然可以向其它NameServer同步其路由信息，Producer,Consumer仍然可以动态感知Broker的路由的信息。

为什么不用zookeeper

RocketMQ 3.x以前使用zookeeper实现，后来改为nameserver实现
zookeeper提供主从选举，广播原子性等特性在MQ中用不上，功能冗余
zk使用主从自动切换提供高可用保障，nameserver使用多副本节点保障高可用
nameserver只提供路由等基本状态信息维护，代码量小，更加轻量，便于维护。

NameServer信息维护源码

package org.apache.rocketmq.namesrv.routeinfo;

public class RouteInfoManager {
 private final ReadWriteLock lock = new ReentrantReadWriteLock();
 private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;
 private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
 private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
 private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
 private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;
}

RocketMQ学习笔记4——消费者核心

Wed, 23 Dec 2020 19:11:00 +0800

配置参数详解

PushConsumer核心参数

ConsumeFromWhere源码，支持三种参数配置，默认是CONSUME_FROM_LAST_OFFSET

package org.apache.rocketmq.common.consumer;

public enum ConsumeFromWhere {
 /**
 * 第一次启动时从队列最后开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_LAST_OFFSET,

 @Deprecated
 CONSUME_FROM_LAST_OFFSET_AND_FROM_MIN_WHEN_BOOT_FIRST,
 @Deprecated
 CONSUME_FROM_MIN_OFFSET,
 @Deprecated
 CONSUME_FROM_MAX_OFFSET,
 /**
 * 从队列最开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_FIRST_OFFSET,
 /**
 * 指定具体时间点开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_TIMESTAMP,
}

AllocateMessageQueueStrategy接口用于集群模式下消息策略，默认配置为平均消息分配器AllocateMessageQueueAveragely

默认配置源码

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Constructor specifying consumer group and enabled msg trace flag.
 *
 * @param consumerGroup Consumer group.
 * @param enableMsgTrace Switch flag instance for message trace.
 */
 public DefaultMQPushConsumer(final String consumerGroup, boolean enableMsgTrace) {
 this(null, consumerGroup, null, new AllocateMessageQueueAveragely(), enableMsgTrace, null);
 }
}

subscription订阅参数，传递两个值，一个是主题，一个是表达式tag过滤

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Subscription relationship
 */
 private Map<String /* topic */, String /* sub expression */> subscription = new HashMap<String, String>();
}

样例：

consumer.subscribe("test_model_topic2", "TagA||TagB");

consumer.subscribe("test_model_topic2", "TagA");

consumer.subscribe("test_model_topic2", "*");

offsetStore配置核心偏移量，有两个实现，用于支持远程存储和本地存储，即支持集群和单机模式

consumeThreadMin与consumeThreadMax线程池自动调整参数，默认为20

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Minimum consumer thread number
 */
 private int consumeThreadMin = 20;

 /**
 * Max consumer thread number
 */
 private int consumeThreadMax = 20;
}

流控参数consumeConcurrentlyMaxSpan与pullThresholdForQueue

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Concurrently max span offset.it has no effect on sequential consumption
 * 单个队列并行消费的最大跨度
 */
 private int consumeConcurrentlyMaxSpan = 2000;

 /**
 * Flow control threshold on queue level, each message queue will cache at most 1000 messages by default,
 * Consider the {@code pullBatchSize}, the instantaneous value may exceed the limit
 * 一个队列最大的消息消费个数
 */
 private int pullThresholdForQueue = 1000;
}

消息拉取参数pullInterval与pullBatchSize

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Message pull Interval
 * 消息拉取的时间间隔
 */
 private long pullInterval = 0;

 /**
 * Batch pull size
 * 一次拉取的数据量
 */
 private int pullBatchSize = 32;
}

consumeMessageBatchMaxSize一次消息最大拉取的数据数量配置

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Batch consumption size
 * 一次消息最大拉取的数据数量
 */
 private int consumeMessageBatchMaxSize = 1;
}

MessageListener用于消息监听配置，包括并发、顺序读取等

MessageModel配置消息模式，广播或者集群，默认为集群模式

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Message model defines the way how messages are delivered to each consumer clients.
 * </p>
 *
 * RocketMQ supports two message models: clustering and broadcasting. If clustering is set, consumer clients with
 * the same {@link #consumerGroup} would only consume shards of the messages subscribed, which achieves load
 * balances; Conversely, if the broadcasting is set, each consumer client will consume all subscribed messages
 * separately.
 * </p>
 *
 * This field defaults to clustering.
 */
 private MessageModel messageModel = MessageModel.CLUSTERING;
}

集群与广播模式

集群模式

MessageModel源码支持两种模式，集群和广播

package org.apache.rocketmq.common.protocol.heartbeat;

/**
 * Message model
 */
public enum MessageModel {
 /**
 * broadcast
 */
 BROADCASTING("BROADCASTING"),
 /**
 * clustering（默认）
 */
 CLUSTERING("CLUSTERING");

 private String modeCN;

 MessageModel(String modeCN) {
 this.modeCN = modeCN;
 }

 public String getModeCN() {
 return modeCN;
 }
}

集群模式下GroupName用于把多个Consumer组织到一起，所以同一个集群，GroupName必须相同
相同的GroupName的Consumer只消费所订阅消息的一部分，即支持负载均衡
优势：集群模式下增加节点能动态增加负载均衡
一条消息只会被一个Consumer消费

样例

消息消费与奇偶性无关，随机负载均衡，并且负载均衡存在一定的差值，并非100%平分
为提高附在均衡度，可以用2个consume监听4个队列，或者3个consume监听6个队列
Producer只能保证平均分配到各个队列，不能保证被平均的消费

生产者

public class Producer {
	
 public static void main(String[] args) throws MQClientException, InterruptedException {
 	String group_name = "test_model_producer_name";
 DefaultMQProducer producer = new DefaultMQProducer(group_name);
 producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 producer.start();
 
 for (int i = 0; i < 10; i++) {
 try {
 	String tag = (i % 2 ==0) ? "TagA":"TagB";
 Message msg = new Message("test_model_topic2",// topic
 		tag,// tag
 ("信息内容" + i).getBytes()// body
 );
 SendResult sendResult = producer.send(msg);
 System.out.println(sendResult);
 }
 catch (Exception e) {
 e.printStackTrace();
 Thread.sleep(1000);
 }
 }
 
 producer.shutdown();
 }
}

消费者1

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.CLUSTERING);
//			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					//if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					//}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
		
	}
}

消费者2

public class Consumer2 {

	public Consumer2() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.CLUSTERING);
//			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					//if(tags.equals("TagB")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					//}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer2 c2 = new Consumer2();
		System.out.println("c2 start..");
		
	}
}

发送10条消息结果

消费者1结果

消费者2结果

注意，若使用集群模式，则tag表达式应该为*号，才能支持多consumer消费，若集群模式下指定tag表达式，则会导致消费结果不确定，原因如下

consumerA监听4条队列中的2条，consumerB监听4条队列中的另外2条。
Producer将包含TagA和TagB的消息均分到4条队列中。
consumerA消费TagA的消息，那么此时consumerA监听的2条队列中的TagB消息则永远不会被消费

样例

consumer1改成监听TagA，其他内容不变

consumer.subscribe("test_model_topic2", "TagA");

此时发送10条消息，consumer1监听的结果

此时将consumer2改成监听TagB并启动

consumer.subscribe("test_model_topic2", "TagB");

此时发送10条消息，consumer1监听的结果

consumer2监听结果为空，大量消息丢失

经过实测发现，只要集群模式下指定了tag，那么只有一个consumer能正常进行消费，其他consumer都为空

广播模式

同一个ConsumeGroup里的Consumer都消费订阅Topic全部信息
也就是一条消息会被每个Consume消费

样例

将consumer1和2改成广播模式，并监听所有tag

consumer.setMessageModel(MessageModel.BROADCASTING);
consumer.subscribe("test_model_topic2", "*");

发送10条消息consumer1消费结果

consumer2消费结果

注意广播模式下使用服务端tag过滤仍然有bug

将consumer1设置为只监听tagA消息

consumer.subscribe("test_model_topic2", "TagA");

consumer2设置为只监听tag2消息

consumer.subscribe("test_model_topic2", "TagB");

发送10条消息consumer1消费结果

consumer2消费结果

结果只有其中一个consumer能正常消费，与集群模式一样的问题

目前（4.4.0）若想要实现同一个topic的tag过滤，则只能让consumer监听所有消息并主动判断

样例：

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
	}
}

发送10条消息结果

消费者1结果

消费者2结果

对于consumer消费指定队列消息，推荐做法是将消息投递到专属的队列中，由同一个consumer监听，即相当于用groupname+topic唯一定位一个consumer，该方法集群与广播模式均适用

样例

此时consumer1变为test_model_consumer_name1

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name1";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "TagA");
//			consumer.setMessageModel(MessageModel.CLUSTERING);
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
	}
}

consumer2变为test_model_consumer_name2

public class Consumer2 {

	public Consumer2() {
		try {
			String group_name = "test_model_consumer_name2";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "TagB");
//			consumer.setMessageModel(MessageModel.CLUSTERING);
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagB")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer2 c2 = new Consumer2();
		System.out.println("c2 start..");
	}
}

发送10条消息结果

消费者1结果

消费者2结果

消费存储核心-OffSet

Offset是消费者消费进度的核心
Offset指某个topic下的一条消息在某个MessageQueue里的位置
通过Offset可以进行定位到这条消息
Offset的存储实现分为远程文件类型和本地文件类型两种

集群模式-`RemoteBrokerOffsetStore`

默认集群模式clustering，采用远程文件存储Offset
本质上因为多消费模式，每个Consumer消费所订阅主题的一部分

广播模式-`LocalFileOffsetStore`

广播模式下，由于每个Consumer都会收到消息且消费
各个Consumer之间没有任何干扰，独立线程消费
所以使用LocalFileOffsetStore，也就是把Offset存储到本地

PushConsumer长轮询模式分析

DefaultMQPushConsumer是使用长轮询模式进行实现的
通常主流的消息获取模式：Push消息推送模式与Pull消息拉取模式
Push模式消耗broker端性能，若消费端消费速度不一致，可能导致消息堆积
pull模式消耗消费端性能，消费端不知道消息达到时间
RocketMQ采用长轮询机制（消费端主动获取）

长轮询过程

客户端发起获取消息请求，假设一次请求超时时间为15s
服务端判断是否有消息，有则直接返回
服务端无消息，则等待5s（阈值），5s后再次检查是否有消息，有则返回，无则继续等待
15s后循环检查结束，若仍然无消息，则返回给客户端

优势：

主动权在客户端，当服务端消息堆积，不会全推送到客户端

缺点：

连接返回时间较长，占用客户端资源（请求阻塞，等待服务端返回）

源码分析

源码实现位置

PullRequestHoldService继承自Runnable接口

核心类PullRequestHoldService源码全貌

package org.apache.rocketmq.broker.longpolling;

public class PullRequestHoldService extends ServiceThread {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
 private static final String TOPIC_QUEUEID_SEPARATOR = "@";
 private final BrokerController brokerController;
 private final SystemClock systemClock = new SystemClock();
 private ConcurrentMap<String/* topic@queueId */, ManyPullRequest> pullRequestTable =
 new ConcurrentHashMap<String, ManyPullRequest>(1024);

 public PullRequestHoldService(final BrokerController brokerController) {
 this.brokerController = brokerController;
 }

 public void suspendPullRequest(final String topic, final int queueId, final PullRequest pullRequest) {
 String key = this.buildKey(topic, queueId);
 ManyPullRequest mpr = this.pullRequestTable.get(key);
 if (null == mpr) {
 mpr = new ManyPullRequest();
 ManyPullRequest prev = this.pullRequestTable.putIfAbsent(key, mpr);
 if (prev != null) {
 mpr = prev;
 }
 }

 mpr.addPullRequest(pullRequest);
 }

 private String buildKey(final String topic, final int queueId) {
 StringBuilder sb = new StringBuilder();
 sb.append(topic);
 sb.append(TOPIC_QUEUEID_SEPARATOR);
 sb.append(queueId);
 return sb.toString();
 }

 @Override
 public void run() {
 log.info("{} service started", this.getServiceName());
 while (!this.isStopped()) {
 try {
 // 判断是否启用长轮询
 if (this.brokerController.getBrokerConfig().isLongPollingEnable()) {
 // 等待5s
 this.waitForRunning(5 * 1000);
 } else {
 this.waitForRunning(this.brokerController.getBrokerConfig().getShortPollingTimeMills());
 }

 long beginLockTimestamp = this.systemClock.now();
 // 检查请求
 this.checkHoldRequest();
 long costTime = this.systemClock.now() - beginLockTimestamp;
 if (costTime > 5 * 1000) {
 log.info("[NOTIFYME] check hold request cost {} ms.", costTime);
 }
 } catch (Throwable e) {
 log.warn(this.getServiceName() + " service has exception. ", e);
 }
 }

 log.info("{} service end", this.getServiceName());
 }

 @Override
 public String getServiceName() {
 return PullRequestHoldService.class.getSimpleName();
 }

 private void checkHoldRequest() {
 // 遍历所有请求的channel
 for (String key : this.pullRequestTable.keySet()) {
 String[] kArray = key.split(TOPIC_QUEUEID_SEPARATOR);
 if (2 == kArray.length) {
 String topic = kArray[0];
 int queueId = Integer.parseInt(kArray[1]);
 final long offset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
 try {
 // 唤醒
 this.notifyMessageArriving(topic, queueId, offset);
 } catch (Throwable e) {
 log.error("check hold request failed. topic={}, queueId={}", topic, queueId, e);
 }
 }
 }
 }

 public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset) {
 notifyMessageArriving(topic, queueId, maxOffset, null, 0, null, null);
 }

 public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset, final Long tagsCode,
 long msgStoreTime, byte[] filterBitMap, Map<String, String> properties) {
 String key = this.buildKey(topic, queueId);
 // 取出等待的channel（请求）
 ManyPullRequest mpr = this.pullRequestTable.get(key);
 if (mpr != null) {
 List<PullRequest> requestList = mpr.cloneListAndClear();
 if (requestList != null) {
 List<PullRequest> replayList = new ArrayList<PullRequest>();
 // 循环所有的请求
 for (PullRequest request : requestList) {
 long newestOffset = maxOffset;
 if (newestOffset <= request.getPullFromThisOffset()) {
 newestOffset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
 }

 if (newestOffset > request.getPullFromThisOffset()) {
 boolean match = request.getMessageFilter().isMatchedByConsumeQueue(tagsCode,
 new ConsumeQueueExt.CqExtUnit(tagsCode, msgStoreTime, filterBitMap));
 // match by bit map, need eval again when properties is not null.
 if (match && properties != null) {
 match = request.getMessageFilter().isMatchedByCommitLog(null, properties);
 }
 // 如果匹配上
 if (match) {
 try {
 // 唤醒request
 this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
 request.getRequestCommand());
 } catch (Throwable e) {
 log.error("execute request when wakeup failed.", e);
 }
 continue;
 }
 }

 if (System.currentTimeMillis() >= (request.getSuspendTimestamp() + request.getTimeoutMillis())) {
 try {
 this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
 request.getRequestCommand());
 } catch (Throwable e) {
 log.error("execute request when wakeup failed.", e);
 }
 continue;
 }

 replayList.add(request);
 }

 if (!replayList.isEmpty()) {
 mpr.addPullRequest(replayList);
 }
 }
 }
 }
}

ManyPullRequest里存储的实际为PullRequest数组

public class ManyPullRequest {
 private final ArrayList<PullRequest> pullRequestList = new ArrayList<>();
	...
}

PullRequest实际存储NIO请求

public class PullRequest {
 private final RemotingCommand requestCommand;
 /** NIO channel */
 private final Channel clientChannel;
 private final long timeoutMillis;
 private final long suspendTimestamp;
 private final long pullFromThisOffset;
 private final SubscriptionData subscriptionData;
 private final MessageFilter messageFilter;
	...
}

`DefaultMQPullConsumer`消息拉取模式

实现类DefaultMQPullConsumer
主要逻辑
- 获取Message Queue并遍历
- 维护OffsetStore
- 根据不同消息状态做不同处理

生产者样例

public class Producer {
 public static void main(String[] args) throws MQClientException, InterruptedException {
 	
 	String group_name = "test_pull_producer_name";
 DefaultMQProducer producer = new DefaultMQProducer(group_name);
 producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 producer.start();
 
 for (int i = 0; i < 10; i++) {
 try {
 Message msg = new Message("test_pull_topic",// topic
 		"TagA",// tag
 		("Hello RocketMQ " + i).getBytes()// body
 );
 // 配置延迟
 msg.setDelayTimeLevel(2);
 
 SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
 // 外部方法的arg会传递到Object中
 @Override
 public MessageQueue select(List<MessageQueue> list, Message message, Object o) {
 Integer queueNumber = (Integer) o;
 return list.get(queueNumber);
 }
 },1);

 SendStatus status = sendResult.getSendStatus();
 System.out.println(status);

 Thread.sleep(1000);
 }
 catch (Exception e) {
 e.printStackTrace();
 Thread.sleep(3000);
 }
 }
 producer.shutdown();
 }
}

发送消息后的结果，在队列1有45条消息

使用普通Pull轮询模式

public class PullConsumer {
	//Map<key, value> key为指定的队列，value为这个队列拉取数据的最后位置
 private static final Map<MessageQueue, Long> offseTable = new HashMap<MessageQueue, Long>();
 
 public static void main(String[] args) throws MQClientException {
 	
 	String group_name = "test_pull_consumer_name";
 DefaultMQPullConsumer consumer = new DefaultMQPullConsumer(group_name);
 consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 consumer.start();
 System.err.println("consumer start");
 //	从TopicTest这个主题去获取所有的队列（默认会有4个队列）
 Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("test_pull_topic");
 //	遍历每一个队列，进行拉取数据
 for (MessageQueue mq : mqs) {
 System.out.println("Consume from the queue: " + mq);
 
 SINGLE_MQ: while (true) {
 try {
 	//	从queue中获取数据，从什么位置开始拉取数据 单次对多拉取32条记录
 PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
 System.out.println(pullResult);
 System.out.println(pullResult.getPullStatus());
 System.out.println();
 // 客户端自行维护Offset
 putMessageQueueOffset(mq, pullResult.getNextBeginOffset());
 switch (pullResult.getPullStatus()) {
	 case FOUND:
	 	List<MessageExt> list = pullResult.getMsgFoundList();
	 	for(MessageExt msg : list){
	 		System.out.println(new String(msg.getBody()));
	 	}
	 break;
	 case NO_MATCHED_MSG:
	 break;
	 case NO_NEW_MSG:
	 	System.out.println("没有新的数据啦...");
	 break SINGLE_MQ;
	 case OFFSET_ILLEGAL:
	 break;
	 default:
	 break;
 }
 }
 catch (Exception e) {
 e.printStackTrace();
 }
 }
 }
 consumer.shutdown();
 }
 
 
 private static void putMessageQueueOffset(MessageQueue mq, long offset) {
 offseTable.put(mq, offset);
 }
 
 
 private static long getMessageQueueOffset(MessageQueue mq) {
 Long offset = offseTable.get(mq);
 if (offset != null)
 return offset;
 return 0;
 }
}

消费结果，其会随机选择队列读取，导致服务端积压的消息得不到有效的消费

而且该模式需要客户端编码实现Offset存储

Pull定时任务实现样例

public class PullScheduleService {

 public static void main(String[] args) throws MQClientException {
 	
 	String group_name = "test_pull_consumer_name";
 	// 使用定时任务
 final MQPullConsumerScheduleService scheduleService = new MQPullConsumerScheduleService(group_name);
 
 scheduleService.getDefaultMQPullConsumer().setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 
 scheduleService.setMessageModel(MessageModel.CLUSTERING);
 
 scheduleService.registerPullTaskCallback("test_pull_topic", new PullTaskCallback() {
 
 @Override
 public void doPullTask(MessageQueue mq, PullTaskContext context) {
 MQPullConsumer consumer = context.getPullConsumer();
 System.err.println("-------------- queueId: " + mq.getQueueId() + "-------------");
 try {
 // 获取从哪里拉取
 long offset = consumer.fetchConsumeOffset(mq, false);
 if (offset < 0)
 offset = 0;
 
 PullResult pullResult = consumer.pull(mq, "*", offset, 32);
 switch (pullResult.getPullStatus()) {
 case FOUND:
 	List<MessageExt> list = pullResult.getMsgFoundList();
 	for(MessageExt msg : list){
 		//消费数据...
 		System.out.println(new String(msg.getBody()));
 	}
 break;
 case NO_MATCHED_MSG:
 break;
 case NO_NEW_MSG:
 case OFFSET_ILLEGAL:
 break;
 default:
 break;
 }
 // 客户端自动更新Offset，无需本地持久化维护
 consumer.updateConsumeOffset(mq, pullResult.getNextBeginOffset());
 // 设置再过3000ms后重新拉取
 context.setPullNextDelayTimeMillis(3000);
 
 }
 catch (Exception e) {
 e.printStackTrace();
 }
 }
 });
 
 scheduleService.start();
 }
}

运行结果，由于采用多线程读取所有队列，所以当服务端有消息时可以及时获取

实现原理

MQPullConsumerScheduleService中将PullTaskCallback方法作为Map存储
并使用ScheduledThreadPoolExecutor作为线程池，配置线程数为10
线程通过调度taskTable中的PullTaskImpl任务，实现方法回调

package org.apache.rocketmq.client.consumer;

public class MQPullConsumerScheduleService {
 private final InternalLogger log = ClientLogger.getLog();
 private final MessageQueueListener messageQueueListener = new MessageQueueListenerImpl();
 private final ConcurrentMap<MessageQueue, PullTaskImpl> taskTable =
 new ConcurrentHashMap<MessageQueue, PullTaskImpl>();
 private DefaultMQPullConsumer defaultMQPullConsumer;
 private int pullThreadNums = 20;
 private ConcurrentMap<String /* topic */, PullTaskCallback> callbackTable =
 new ConcurrentHashMap<String, PullTaskCallback>();
 private ScheduledThreadPoolExecutor scheduledThreadPoolExecutor;

 public MQPullConsumerScheduleService(final String consumerGroup) {
 this.defaultMQPullConsumer = new DefaultMQPullConsumer(consumerGroup);
 this.defaultMQPullConsumer.setMessageModel(MessageModel.CLUSTERING);
 }

 public void putTask(String topic, Set<MessageQueue> mqNewSet) {
 Iterator<Entry<MessageQueue, PullTaskImpl>> it = this.taskTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<MessageQueue, PullTaskImpl> next = it.next();
 if (next.getKey().getTopic().equals(topic)) {
 if (!mqNewSet.contains(next.getKey())) {
 next.getValue().setCancelled(true);
 it.remove();
 }
 }
 }

 for (MessageQueue mq : mqNewSet) {
 if (!this.taskTable.containsKey(mq)) {
 // 需要调度的任务
 PullTaskImpl command = new PullTaskImpl(mq);
 this.taskTable.put(mq, command);
 // 使用多线程调度
 this.scheduledThreadPoolExecutor.schedule(command, 0, TimeUnit.MILLISECONDS);

 }
 }
 }

 public void start() throws MQClientException {
 final String group = this.defaultMQPullConsumer.getConsumerGroup();
 this.scheduledThreadPoolExecutor = new ScheduledThreadPoolExecutor(
 this.pullThreadNums,
 new ThreadFactoryImpl("PullMsgThread-" + group)
 );

 this.defaultMQPullConsumer.setMessageQueueListener(this.messageQueueListener);

 this.defaultMQPullConsumer.start();

 log.info("MQPullConsumerScheduleService start OK, {} {}",
 this.defaultMQPullConsumer.getConsumerGroup(), this.callbackTable);
 }

 public void registerPullTaskCallback(final String topic, final PullTaskCallback callback) {
 // 放入回调任务集合
 this.callbackTable.put(topic, callback);
 this.defaultMQPullConsumer.registerMessageQueueListener(topic, null);
 }

 class MessageQueueListenerImpl implements MessageQueueListener {
 @Override
 public void messageQueueChanged(String topic, Set<MessageQueue> mqAll, Set<MessageQueue> mqDivided) {
 MessageModel messageModel =
 MQPullConsumerScheduleService.this.defaultMQPullConsumer.getMessageModel();
 switch (messageModel) {
 case BROADCASTING:
 MQPullConsumerScheduleService.this.putTask(topic, mqAll);
 break;
 case CLUSTERING:
 MQPullConsumerScheduleService.this.putTask(topic, mqDivided);
 break;
 default:
 break;
 }
 }
 }

 class PullTaskImpl implements Runnable {
 private final MessageQueue messageQueue;
 private volatile boolean cancelled = false;

 public PullTaskImpl(final MessageQueue messageQueue) {
 this.messageQueue = messageQueue;
 }

 @Override
 public void run() {
 String topic = this.messageQueue.getTopic();
 if (!this.isCancelled()) {
 // 获取回调任务
 PullTaskCallback pullTaskCallback =
 MQPullConsumerScheduleService.this.callbackTable.get(topic);
 if (pullTaskCallback != null) {
 final PullTaskContext context = new PullTaskContext();
 context.setPullConsumer(MQPullConsumerScheduleService.this.defaultMQPullConsumer);
 try {
 // 执行客户端实现的回调任务
 pullTaskCallback.doPullTask(this.messageQueue, context);
 } catch (Throwable e) {
 context.setPullNextDelayTimeMillis(1000);
 log.error("doPullTask Exception", e);
 }

 if (!this.isCancelled()) {
 MQPullConsumerScheduleService.this.scheduledThreadPoolExecutor.schedule(this,
 context.getPullNextDelayTimeMillis(), TimeUnit.MILLISECONDS);
 } else {
 log.warn("The Pull Task is cancelled after doPullTask, {}", messageQueue);
 }
 } else {
 log.warn("Pull Task Callback not exist , {}", topic);
 }
 } else {
 log.warn("The Pull Task is cancelled, {}", messageQueue);
 }
 }
 }
}

RocketMQ学习笔记3——生产者核心

Wed, 23 Dec 2020 19:09:00 +0800

配置参数

producerGroup：组名
createTopicKey：创建Topic（不允许使用，应当由架构设计定义）
defaultTopicQueueNums：创建topic时默认的队列数量（默认为4）
sendMsgTimeout：发送消息时的超时时间（默认值：3000，单位：毫秒）
compressMsgBodyOverHowmuch ：消息压缩字节（默认4096）
retryTimesWhenSendFailed ：同步模式下，在返回发送失败之前，内部尝试重新发送消息的最大次数。（默认值：2）
retryTimesWhenSendAsyncFailed：异步模式下，在发送失败之前，内部尝试重新发送消息的最大次数。（默认值：2）
retryAnotherBrokerWhenNotStoreOK：同步模式下，消息保存失败时是否重试其他broker 。（默认值：false ）
maxMessageSize：最大消息限制。（默认值：1024 * 1024 * 4，单位：字节）

主从同步机制

主从同步
同步信息：数据内容+元数据信息
元数据（配置信息）同步：Broker角色识别，为Slave则启动同步任务
- 通过Netty传输
- 实质为定时任务
消息同步（通过Socket）：HAService、HAconnection、WaitNotifyObject
- 通过原生Socket传输，性能高，数据包支持容量大

元数据同步源码分析

元信息同步工程源码位于broker中

发起定时任务源码BrokerController

package org.apache.rocketmq.broker;

public class BrokerController {
 public boolean initialize() throws CloneNotSupportedException {
 ...
 if (result) {
 ...
 // 进行角色判断
 if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) {
 if (this.messageStoreConfig.getHaMasterAddress() != null && this.messageStoreConfig.getHaMasterAddress().length() >= 6) {
 this.messageStore.updateHaMasterAddress(this.messageStoreConfig.getHaMasterAddress());
 this.updateMasterHAServerAddrPeriodically = false;
 } else {
 this.updateMasterHAServerAddrPeriodically = true;
 }
 // 启动定时任务进行元信息同步
 this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

 @Override
 public void run() {
 try {
 BrokerController.this.slaveSynchronize.syncAll();
 } catch (Throwable e) {
 log.error("ScheduledTask syncAll slave exception", e);
 }
 }
 }, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);
 } else {
 this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

 @Override
 public void run() {
 try {
 BrokerController.this.printMasterAndSlaveDiff();
 } catch (Throwable e) {
 log.error("schedule printMasterAndSlaveDiff error.", e);
 }
 }
 }, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);
 }
 }
 ...
 }
}

syncAll方法实现

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 public void syncAll() {
 // 同步Topic配置信息
 this.syncTopicConfig();
 // 同步消费者偏移量
 this.syncConsumerOffset();
 // 同步延迟偏移量
 this.syncDelayOffset();
 // 同步订阅组配置信息
 this.syncSubscriptionGroupConfig();
 }
 ...
}

同步Topic配置信息源码

syncTopicConfig方法实现如下

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 private void syncTopicConfig() {
 String masterAddrBak = this.masterAddr;
 if (masterAddrBak != null) {
 try {
 TopicConfigSerializeWrapper topicWrapper =
 this.brokerController.getBrokerOuterAPI().getAllTopicConfig(masterAddrBak);
 if (!this.brokerController.getTopicConfigManager().getDataVersion()
 .equals(topicWrapper.getDataVersion())) {

 this.brokerController.getTopicConfigManager().getDataVersion()
 .assignNewOne(topicWrapper.getDataVersion());
 this.brokerController.getTopicConfigManager().getTopicConfigTable().clear();
 this.brokerController.getTopicConfigManager().getTopicConfigTable()
 .putAll(topicWrapper.getTopicConfigTable());
 this.brokerController.getTopicConfigManager().persist();

 log.info("Update slave topic config from master, {}", masterAddrBak);
 }
 } catch (Exception e) {
 log.error("SyncTopicConfig Exception, {}", masterAddrBak, e);
 }
 }
 }
 ...
}

其中getAllTopicConfig方法实现如下

package org.apache.rocketmq.broker.out;

public class BrokerOuterAPI {
 ...
 public TopicConfigSerializeWrapper getAllTopicConfig(
 final String addr) throws RemotingConnectException, RemotingSendRequestException,
 RemotingTimeoutException, InterruptedException, MQBrokerException {
 RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_TOPIC_CONFIG, null);
 // 发起远程同步调用
 RemotingCommand response = this.remotingClient.invokeSync(MixAll.brokerVIPChannel(true, addr), request, 3000);
 assert response != null;
 switch (response.getCode()) {
 case ResponseCode.SUCCESS: {
 return TopicConfigSerializeWrapper.decode(response.getBody(), TopicConfigSerializeWrapper.class);
 }
 default:
 break;
 }

 throw new MQBrokerException(response.getCode(), response.getRemark());
 }
 ...
}

invokeSync通过调用封装过的Netty客户端实现

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingClient extends NettyRemotingAbstract implements RemotingClient {
 ...
 /**
 * 同步请求
 * @param addr 请求地址
 * @param request 请求命令
 * @param timeoutMillis 超时时间
 */
 @Override
 public RemotingCommand invokeSync(String addr, final RemotingCommand request, long timeoutMillis)
 throws InterruptedException, RemotingConnectException, RemotingSendRequestException, RemotingTimeoutException {
 long beginStartTime = System.currentTimeMillis();
 final Channel channel = this.getAndCreateChannel(addr);
 if (channel != null && channel.isActive()) {
 try {
 doBeforeRpcHooks(addr, request);
 // 算出极限耗时代价
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTime) {
 // 如果同步等待时长 < 极限耗时 说明请求不成立，抛出异常
 throw new RemotingTimeoutException("invokeSync call timeout");
 }
 // 同步请求实现
 RemotingCommand response = this.invokeSyncImpl(channel, request, timeoutMillis - costTime);
 doAfterRpcHooks(RemotingHelper.parseChannelRemoteAddr(channel), request, response);
 return response;
 } catch (RemotingSendRequestException e) {
 log.warn("invokeSync: send request exception, so close the channel[{}]", addr);
 this.closeChannel(addr, channel);
 throw e;
 } catch (RemotingTimeoutException e) {
 if (nettyClientConfig.isClientCloseSocketIfTimeout()) {
 this.closeChannel(addr, channel);
 log.warn("invokeSync: close socket because of timeout, {}ms, {}", timeoutMillis, addr);
 }
 log.warn("invokeSync: wait response timeout exception, the channel[{}]", addr);
 throw e;
 }
 } else {
 this.closeChannel(addr, channel);
 throw new RemotingConnectException(addr);
 }
 }
 ...
}

invokeSyncImpl实现逻辑

package org.apache.rocketmq.remoting.netty;

public abstract class NettyRemotingAbstract {
 ...
 public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
 final long timeoutMillis)
 throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
 final int opaque = request.getOpaque();

 try {
 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
 this.responseTable.put(opaque, responseFuture);
 final SocketAddress addr = channel.remoteAddress();
 // 使用netty channel
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 // 轮训监听
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 } else {
 responseFuture.setSendRequestOK(false);
 }
 // 异常情况
 responseTable.remove(opaque);
 responseFuture.setCause(f.cause());
 responseFuture.putResponse(null);
 log.warn("send a request command to channel <" + addr + "> failed.");
 }
 });

 RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
 if (null == responseCommand) {
 if (responseFuture.isSendRequestOK()) {
 throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
 responseFuture.getCause());
 } else {
 throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
 }
 }

 return responseCommand;
 } finally {
 this.responseTable.remove(opaque);
 }
 }
	...
}

同步消费者偏移量源码

其中syncConsumerOffset方法实现如下

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 private void syncConsumerOffset() {
 String masterAddrBak = this.masterAddr;
 if (masterAddrBak != null) {
 try {
 ConsumerOffsetSerializeWrapper offsetWrapper =
 this.brokerController.getBrokerOuterAPI().getAllConsumerOffset(masterAddrBak);
 this.brokerController.getConsumerOffsetManager().getOffsetTable()
 .putAll(offsetWrapper.getOffsetTable());
 this.brokerController.getConsumerOffsetManager().persist();
 log.info("Update slave consumer offset from master, {}", masterAddrBak);
 } catch (Exception e) {
 log.error("SyncConsumerOffset Exception, {}", masterAddrBak, e);
 }
 }
 }
 ...
}

getAllConsumerOffset实现逻辑

package org.apache.rocketmq.broker.out;

public class BrokerOuterAPI {
 ...
 public ConsumerOffsetSerializeWrapper getAllConsumerOffset(
 final String addr) throws InterruptedException, RemotingTimeoutException,
 RemotingSendRequestException, RemotingConnectException, MQBrokerException {
 // 封装CMD指令向主节点发起同步offset信息请求，同步时间上限为3000ms
 RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_CONSUMER_OFFSET, null);
 RemotingCommand response = this.remotingClient.invokeSync(addr, request, 3000);
 assert response != null;
 switch (response.getCode()) {
 case ResponseCode.SUCCESS: {
 return ConsumerOffsetSerializeWrapper.decode(response.getBody(), ConsumerOffsetSerializeWrapper.class);
 }
 default:
 break;
 }

 throw new MQBrokerException(response.getCode(), response.getRemark());
 }
 ...
}

封装的CMD指令

package org.apache.rocketmq.common.protocol;

public class RequestCode {
 ...
 public static final int NOTIFY_CONSUMER_IDS_CHANGED = 40;

 public static final int LOCK_BATCH_MQ = 41;

 public static final int UNLOCK_BATCH_MQ = 42;
 public static final int GET_ALL_CONSUMER_OFFSET = 43;

 public static final int GET_ALL_DELAY_OFFSET = 45;
 ...
}

远程命令对象构建，构建后通过remotingClient发送即可

package org.apache.rocketmq.remoting.protocol;

public class RemotingCommand {
 ...
	public static RemotingCommand createRequestCommand(int code, CommandCustomHeader customHeader) {
 RemotingCommand cmd = new RemotingCommand();
 cmd.setCode(code);
 cmd.customHeader = customHeader;
 setCmdVersion(cmd);
 return cmd;
 }
 ...
}

消息同步源码

消息同步工程源码位于store中

HAConnection实现消息读写

HAservice包含节点连接实现HAClient

逻辑

Master节点：

AcceptSocketService：接收Slave节点连接
HAConnection：
- ReadSocketService：读取来自Slave节点的数据
- WriteSocketService：写入Slave节点的数据

Slave节点：

HAService：
- HAClient：对Master节点连接，读写数据

通信协议：Master节点与Slave节点通信协议较简单，主要如下

对象	用途	第几位	字段	数据类型	字节数	说明
`Slave`=>`Master`	上报`CommitLog`已经同步的物理位置	0	`maxPhyOffset`	Long	8	`CommitLog`最大物理位置
`Master`=>`Slave`	传输新的`CommitLog`数据	0	`fromPhyOffset`	Long	8	传输`CommitLog`开始的物理位置
		1	`size`	Int	4	传输`CommitLog`数据长度
		2	`body`	Bytes		传输`CommitLog`数据

连接Master源码

package org.apache.rocketmq.store.ha;

public class HAService {
 ...
 class HAClient extends ServiceThread {
 /**
 * 客户端连接Master，使用NIO函数，提高效率
 * @return
 * @throws ClosedChannelException
 */
 private boolean connectMaster() throws ClosedChannelException {
 if (null == socketChannel) {
 String addr = this.masterAddress.get();
 if (addr != null) {

 SocketAddress socketAddress = RemotingUtil.string2SocketAddress(addr);
 if (socketAddress != null) {
 this.socketChannel = RemotingUtil.connect(socketAddress);
 if (this.socketChannel != null) {
 this.socketChannel.register(this.selector, SelectionKey.OP_READ);
 }
 }
 }

 this.currentReportedOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();

 this.lastWriteTimestamp = System.currentTimeMillis();
 }

 return this.socketChannel != null;
 }
 
 /**
 * Client(Slave)主循环，实现不断从Master传输CommitLog数据
 * 上传Master自己本地的CommitLog已经同步物理位置
 */
 @Override
 public void run() {
 log.info(this.getServiceName() + " service started");

 while (!this.isStopped()) {
 try {
 if (this.connectMaster()) {

 if (this.isTimeToReportOffset()) {
 // 满足上报间隔，上报到Master进度
 // 固定间隔（默认为5s）向Master上报Slave本地CommitLog已经同步到的物理位置
 boolean result = this.reportSlaveMaxOffset(this.currentReportedOffset);
 if (!result) {
 this.closeMaster();
 }
 }

 this.selector.select(1000);
 // 处理Master传输给Slave的CommitLog数据
 boolean ok = this.processReadEvent();
 if (!ok) {
 this.closeMaster();
 }
 // 若进度有变化，上报到Master进度
 if (!reportSlaveMaxOffsetPlus()) {
 continue;
 }
 // Master超时未返回，关闭连接
 long interval =
 HAService.this.getDefaultMessageStore().getSystemClock().now()
 - this.lastWriteTimestamp;
 if (interval > HAService.this.getDefaultMessageStore().getMessageStoreConfig()
 .getHaHousekeepingInterval()) {
 log.warn("HAClient, housekeeping, found this connection[" + this.masterAddress
 + "] expired, " + interval);
 this.closeMaster();
 log.warn("HAClient, master not response some time, so close connection");
 }
 } else {
 this.waitForRunning(1000 * 5);
 }
 } catch (Exception e) {
 log.warn(this.getServiceName() + " service has exception. ", e);
 this.waitForRunning(1000 * 5);
 }
 }

 log.info(this.getServiceName() + " service end");
 }
 ...
 }
 ...
}

RemotingUtil使用原生Socket实现

package org.apache.rocketmq.remoting.common;

public class RemotingUtil {
 public static SocketAddress string2SocketAddress(final String addr) {
 String[] s = addr.split(":");
 InetSocketAddress isa = new InetSocketAddress(s[0], Integer.parseInt(s[1]));
 return isa;
 }
 
 public static SocketChannel connect(SocketAddress remote, final int timeoutMillis) {
 SocketChannel sc = null;
 try {
 // 打开channel
 sc = SocketChannel.open();
 // 设置同步阻塞，可能存在bug
 sc.configureBlocking(true);
 // 设置关闭Socket的延迟事件，那么当线程执行Socket的close方法时，会进入阻塞状态
 // 直到Socket发送完所有剩余数据，或者超过了方法设置的延迟时间，才从close方法返回
 sc.socket().setSoLinger(false, -1);
 // 禁止使用Nagle算法，使用小数据传输
 // Nagle算法通过将缓冲区的小封包自动相连，组成较大的包
 sc.socket().setTcpNoDelay(true);
 // 设置接收缓冲区
 sc.socket().setReceiveBufferSize(1024 * 64);
 // 设置发送缓冲区
 sc.socket().setSendBufferSize(1024 * 64);
 // 连接
 sc.socket().connect(remote, timeoutMillis);
 // bug？又设置非同步阻塞了
 sc.configureBlocking(false);
 return sc;
 } catch (Exception e) {
 if (sc != null) {
 try {
 sc.close();
 } catch (IOException e1) {
 e1.printStackTrace();
 }
 }
 }

 return null;
 }
 ...
}

核心读请求实现

package org.apache.rocketmq.store.ha;

public class HAService {
 ...
 class HAClient extends ServiceThread {
 private boolean processReadEvent() {
 int readSizeZeroTimes = 0;
 while (this.byteBufferRead.hasRemaining()) {
 try {
 int readSize = this.socketChannel.read(this.byteBufferRead);
 if (readSize > 0) {
 lastWriteTimestamp = HAService.this.defaultMessageStore.getSystemClock().now();
 readSizeZeroTimes = 0;
 boolean result = this.dispatchReadRequest();
 if (!result) {
 log.error("HAClient, dispatchReadRequest error");
 return false;
 }
 } else if (readSize == 0) {
 if (++readSizeZeroTimes >= 3) {
 break;
 }
 } else {
 log.info("HAClient, processReadEvent read socket < 0");
 return false;
 }
 } catch (IOException e) {
 log.info("HAClient, processReadEvent read socket exception", e);
 return false;
 }
 }

 return true;
 }
 
 /**
 * 核心读请求实现
 * 
 * 读取Master传输的CommitLog数据，并返回是否正确
 * 如果读取到数据，写入CommitLog
 * 异常原因：
 * 1. Master传输来的数据开始位置offset(masterPhyOffset)不等于Slave的CommitLog数最大offset(slavePhyOffset)
 * 2. 上报到Master进度失败
 * @return 是否正确
 */
 private boolean dispatchReadRequest() {
 final int msgHeaderSize = 8 + 4; // phyoffset + size
 int readSocketPos = this.byteBufferRead.position();

 while (true) {
 // 读取到请求数据
 int diff = this.byteBufferRead.position() - this.dispatchPostion;
 if (diff >= msgHeaderSize) {
 // 读取masterPhyOffset、bodySize
 long masterPhyOffset = this.byteBufferRead.getLong(this.dispatchPostion);
 int bodySize = this.byteBufferRead.getInt(this.dispatchPostion + 8);
 // 设置Slave节点CommitLog文件最大的offset位置
 long slavePhyOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();
 // 校验Master传输来的数据开始位置offset和slave的commitLog数最大offset是否相同
 if (slavePhyOffset != 0) {
 if (slavePhyOffset != masterPhyOffset) {
 log.error("master pushed offset not equal the max phy offset in slave, SLAVE: "
 + slavePhyOffset + " MASTER: " + masterPhyOffset);
 return false;
 }
 }
 // 读取消息
 if (diff >= (msgHeaderSize + bodySize)) {
 // 写入CommitLog
 byte[] bodyData = new byte[bodySize];
 this.byteBufferRead.position(this.dispatchPostion + msgHeaderSize);
 this.byteBufferRead.get(bodyData);

 HAService.this.defaultMessageStore.appendToCommitLog(masterPhyOffset, bodyData);
 // 设置处理到的位置
 this.byteBufferRead.position(readSocketPos);
 this.dispatchPostion += msgHeaderSize + bodySize;
 // 上报到Master进度
 if (!reportSlaveMaxOffsetPlus()) {
 return false;
 }
 // 继续读数据
 continue;
 }
 }
 // 空间写满，重新分配空间
 if (!this.byteBufferRead.hasRemaining()) {
 this.reallocateByteBuffer();
 }

 break;
 }

 return true;
 }
 ...
 }
 ...
}

同步\异步消息发送

实现分类

同步消息发送
- 发送：producer.send(Message msg)
- 核心实现：DefaultMQProducerImpl
异步消息发送
- 发送：producer.send(Message msg,SendCallback sendCallback)
- 核心实现：DefaultMQProducerImpl

实例

public class Producer {

	public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
		
		DefaultMQProducer producer = new DefaultMQProducer("test_quick_producer_name");
		
	producer.setNamesrvAddr(Const.NAMESRV_ADDR_MASTER_SLAVE);
		
		producer.start();
		
		for(int i = 0 ; i <5; i ++) {
			//	1.	创建消息
			Message message = new Message("test_quick_topic",	//	主题
					"TagA", //	标签
					"key" + i,	// 	用户自定义的key ,唯一的标识
					("Hello RocketMQ" + i).getBytes());	//	消息内容实体（byte[]）
			// 异步发送消息
			producer.send(message, new SendCallback() {
				@Override
				public void onSuccess(SendResult sendResult) {
					System.err.println("msgId: " + sendResult.getMsgId() + ", status: " + sendResult.getSendStatus());
					producer.shutdown();
				}
				@Override
				public void onException(Throwable e) {
					e.printStackTrace();
					System.err.println("------发送失败");
				}
			});
		}	
	}
}

同步消息发送源码分析

发送消息方法

package org.apache.rocketmq.client.producer;

public class DefaultMQProducer extends ClientConfig implements MQProducer {
 ...
 @Override
 public SendResult send(Message msg,
 long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 return this.defaultMQProducerImpl.send(msg, timeout);
 }
}

模式枚举

package org.apache.rocketmq.client.impl;

public enum CommunicationMode {
 SYNC,
 ASYNC,
 // 单向，不需要响应
 ONEWAY,
}

消息发送实现类

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public SendResult send(Message msg,
 long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 return this.sendDefaultImpl(msg, CommunicationMode.SYNC, null, timeout);
 }
 
 private SendResult sendDefaultImpl(
 Message msg,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final long timeout
 ) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 // 校验运行状态
 this.makeSureStateOK();
 // 检查消息格式
 Validators.checkMessage(msg, this.defaultMQProducer);
 // 回调ID，用户标记是否同一次消息
 final long invokeID = random.nextLong();
 // 开始发送消息的时间
 long beginTimestampFirst = System.currentTimeMillis();
 long beginTimestampPrev = beginTimestampFirst;
 long endTimestamp = beginTimestampFirst;
 // 查找topic
 TopicPublishInfo topicPublishInfo = this.tryToFindTopicPublishInfo(msg.getTopic());
 if (topicPublishInfo != null && topicPublishInfo.ok()) {
 boolean callTimeout = false;
 // 要发送的消息队列
 MessageQueue mq = null;
 Exception exception = null;
 SendResult sendResult = null;
 // 重试次数，从配置参数中读取
 int timesTotal = communicationMode == CommunicationMode.SYNC ? 1 + this.defaultMQProducer.getRetryTimesWhenSendFailed() : 1;
 int times = 0;
 // 存储每次发送消息选择的broker名称
 String[] brokersSent = new String[timesTotal];
 for (; times < timesTotal; times++) {
 // 最后一次消息发送的brokerName进行存储
 String lastBrokerName = null == mq ? null : mq.getBrokerName();
 // 消息队列选择策略
 MessageQueue mqSelected = this.selectOneMessageQueue(topicPublishInfo, lastBrokerName);
 if (mqSelected != null) {
 mq = mqSelected;
 brokersSent[times] = mq.getBrokerName();
 try {
 beginTimestampPrev = System.currentTimeMillis();
 // 计算代价
 long costTime = beginTimestampPrev - beginTimestampFirst;
 if (timeout < costTime) {
 callTimeout = true;
 break;
 }
 // 消息发送实现
 sendResult = this.sendKernelImpl(msg, mq, communicationMode, sendCallback, topicPublishInfo, timeout - costTime);
 endTimestamp = System.currentTimeMillis();
 // 更新broker可用信息（容错）
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
 switch (communicationMode) {
 case ASYNC:
 return null;
 case ONEWAY:
 return null;
 case SYNC:
 // 同步发送等待返回结果
 if (sendResult.getSendStatus() != SendStatus.SEND_OK) {
 if (this.defaultMQProducer.isRetryAnotherBrokerWhenNotStoreOK()) {
 continue;
 }
 }

 return sendResult;
 default:
 break;
 }
 } catch (RemotingException e) {
 endTimestamp = System.currentTimeMillis();
 // 抛异常则更新broker容错信息
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 continue;
 } catch (MQClientException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 continue;
 } catch (MQBrokerException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 // 抛出如下异常则进行重试
 switch (e.getResponseCode()) {
 case ResponseCode.TOPIC_NOT_EXIST:
 case ResponseCode.SERVICE_NOT_AVAILABLE:
 case ResponseCode.SYSTEM_ERROR:
 case ResponseCode.NO_PERMISSION:
 case ResponseCode.NO_BUYER_ID:
 case ResponseCode.NOT_IN_CURRENT_UNIT:
 continue;
 default:
 if (sendResult != null) {
 return sendResult;
 }

 throw e;
 }
 } catch (InterruptedException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
 log.warn(String.format("sendKernelImpl exception, throw exception, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());

 log.warn("sendKernelImpl exception", e);
 log.warn(msg.toString());
 throw e;
 }
 } else {
 break;
 }
 }

 if (sendResult != null) {
 return sendResult;
 }

 String info = String.format("Send [%d] times, still failed, cost [%d]ms, Topic: %s, BrokersSent: %s",
 times,
 System.currentTimeMillis() - beginTimestampFirst,
 msg.getTopic(),
 Arrays.toString(brokersSent));

 info += FAQUrl.suggestTodo(FAQUrl.SEND_MSG_FAILED);

 MQClientException mqClientException = new MQClientException(info, exception);
 if (callTimeout) {
 throw new RemotingTooMuchRequestException("sendDefaultImpl call timeout");
 }

 if (exception instanceof MQBrokerException) {
 mqClientException.setResponseCode(((MQBrokerException) exception).getResponseCode());
 } else if (exception instanceof RemotingConnectException) {
 mqClientException.setResponseCode(ClientErrorCode.CONNECT_BROKER_EXCEPTION);
 } else if (exception instanceof RemotingTimeoutException) {
 mqClientException.setResponseCode(ClientErrorCode.ACCESS_BROKER_TIMEOUT);
 } else if (exception instanceof MQClientException) {
 mqClientException.setResponseCode(ClientErrorCode.BROKER_NOT_EXIST_EXCEPTION);
 }

 throw mqClientException;
 }

 List<String> nsList = this.getmQClientFactory().getMQClientAPIImpl().getNameServerAddressList();
 if (null == nsList || nsList.isEmpty()) {
 throw new MQClientException(
 "No name server address, please set it." + FAQUrl.suggestTodo(FAQUrl.NAME_SERVER_ADDR_NOT_EXIST_URL), null).setResponseCode(ClientErrorCode.NO_NAME_SERVER_EXCEPTION);
 }

 throw new MQClientException("No route info of this topic, " + msg.getTopic() + FAQUrl.suggestTodo(FAQUrl.NO_TOPIC_ROUTE_INFO),
 null).setResponseCode(ClientErrorCode.NOT_FOUND_TOPIC_EXCEPTION);
 }
}

查找topic方法实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 private TopicPublishInfo tryToFindTopicPublishInfo(final String topic) {
 // 使用Map缓存
 TopicPublishInfo topicPublishInfo = this.topicPublishInfoTable.get(topic);
 if (null == topicPublishInfo || !topicPublishInfo.ok()) {
 // 当无可用topic发布信息时，主动向NameServer拉取一次
 this.topicPublishInfoTable.putIfAbsent(topic, new TopicPublishInfo());
 this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
 topicPublishInfo = this.topicPublishInfoTable.get(topic);
 }
 // 若可用
 if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) {
 return topicPublishInfo;
 } else {
 // 否则根据设置判断是否新建Topic
 this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, true, this.defaultMQProducer);
 topicPublishInfo = this.topicPublishInfoTable.get(topic);
 return topicPublishInfo;
 }
 }
}

客户端获取路由信息实现

package org.apache.rocketmq.client.impl.factory;

public class MQClientInstance {
 public boolean updateTopicRouteInfoFromNameServer(final String topic) {
 return updateTopicRouteInfoFromNameServer(topic, false, null);
 }
 
 public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault,
 DefaultMQProducer defaultMQProducer) {
 try {
 if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
 try {
 TopicRouteData topicRouteData;
 if (isDefault && defaultMQProducer != null) {
 topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(),
 1000 * 3);
 if (topicRouteData != null) {
 for (QueueData data : topicRouteData.getQueueDatas()) {
 int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
 data.setReadQueueNums(queueNums);
 data.setWriteQueueNums(queueNums);
 }
 }
 } else {
 topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, 1000 * 3);
 }
 if (topicRouteData != null) {
 TopicRouteData old = this.topicRouteTable.get(topic);
 boolean changed = topicRouteDataIsChange(old, topicRouteData);
 if (!changed) {
 changed = this.isNeedUpdateTopicRouteInfo(topic);
 } else {
 log.info("the topic[{}] route info changed, old[{}] ,new[{}]", topic, old, topicRouteData);
 }

 if (changed) {
 TopicRouteData cloneTopicRouteData = topicRouteData.cloneTopicRouteData();

 for (BrokerData bd : topicRouteData.getBrokerDatas()) {
 this.brokerAddrTable.put(bd.getBrokerName(), bd.getBrokerAddrs());
 }

 // Update Pub info
 {
 TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData);
 publishInfo.setHaveTopicRouterInfo(true);
 Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<String, MQProducerInner> entry = it.next();
 MQProducerInner impl = entry.getValue();
 if (impl != null) {
 impl.updateTopicPublishInfo(topic, publishInfo);
 }
 }
 }

 // Update sub info
 {
 Set<MessageQueue> subscribeInfo = topicRouteData2TopicSubscribeInfo(topic, topicRouteData);
 Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<String, MQConsumerInner> entry = it.next();
 MQConsumerInner impl = entry.getValue();
 if (impl != null) {
 impl.updateTopicSubscribeInfo(topic, subscribeInfo);
 }
 }
 }
 log.info("topicRouteTable.put. Topic = {}, TopicRouteData[{}]", topic, cloneTopicRouteData);
 this.topicRouteTable.put(topic, cloneTopicRouteData);
 return true;
 }
 } else {
 log.warn("updateTopicRouteInfoFromNameServer, getTopicRouteInfoFromNameServer return null, Topic: {}", topic);
 }
 } catch (Exception e) {
 if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX) && !topic.equals(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC)) {
 log.warn("updateTopicRouteInfoFromNameServer Exception", e);
 }
 } finally {
 this.lockNamesrv.unlock();
 }
 } else {
 log.warn("updateTopicRouteInfoFromNameServer tryLock timeout {}ms", LOCK_TIMEOUT_MILLIS);
 }
 } catch (InterruptedException e) {
 log.warn("updateTopicRouteInfoFromNameServer Exception", e);
 }

 return false;
 }
}

消息队列选择策略实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
 return this.mqFaultStrategy.selectOneMessageQueue(tpInfo, lastBrokerName);
 }
}

负载均衡消息选择策略实现

package org.apache.rocketmq.client.latency;

public class MQFaultStrategy {
 private final static InternalLogger log = ClientLogger.getLog();
 private final LatencyFaultTolerance<String> latencyFaultTolerance = new LatencyFaultToleranceImpl();
 // 发送消息延迟容错开关
 private boolean sendLatencyFaultEnable = false;
 // 消息发送时长的延迟级别数组
 private long[] latencyMax = {50L, 100L, 550L, 1000L, 2000L, 3000L, 15000L};
 // broker预估不可用时长级别数组
 private long[] notAvailableDuration = {0L, 0L, 30000L, 60000L, 120000L, 180000L, 600000L};
 /**
 * 负载均衡消息选择策略
 */
 public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
 if (this.sendLatencyFaultEnable) {
 try {
 // 获取可用队列
 int index = tpInfo.getSendWhichQueue().getAndIncrement();
 for (int i = 0; i < tpInfo.getMessageQueueList().size(); i++) {
 int pos = Math.abs(index++) % tpInfo.getMessageQueueList().size();
 if (pos < 0)
 pos = 0;
 MessageQueue mq = tpInfo.getMessageQueueList().get(pos);
 if (latencyFaultTolerance.isAvailable(mq.getBrokerName())) {
 if (null == lastBrokerName || mq.getBrokerName().equals(lastBrokerName))
 return mq;
 }
 }
 // 选择最优队列
 final String notBestBroker = latencyFaultTolerance.pickOneAtLeast();
 int writeQueueNums = tpInfo.getQueueIdByBroker(notBestBroker);
 if (writeQueueNums > 0) {
 final MessageQueue mq = tpInfo.selectOneMessageQueue();
 if (notBestBroker != null) {
 mq.setBrokerName(notBestBroker);
 mq.setQueueId(tpInfo.getSendWhichQueue().getAndIncrement() % writeQueueNums);
 }
 return mq;
 } else {
 latencyFaultTolerance.remove(notBestBroker);
 }
 } catch (Exception e) {
 log.error("Error occurred when selecting message queue", e);
 }

 return tpInfo.selectOneMessageQueue();
 }

 return tpInfo.selectOneMessageQueue(lastBrokerName);
 }
}

更新broker可用信息（容错）实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) {
 this.mqFaultStrategy.updateFaultItem(brokerName, currentLatency, isolation);
 }
}

与负载均衡消息选择策略位于相同类中

package org.apache.rocketmq.client.latency;

public class MQFaultStrategy {
 	 public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) {
 if (this.sendLatencyFaultEnable) {
 long duration = computeNotAvailableDuration(isolation ? 30000 : currentLatency);
 this.latencyFaultTolerance.updateFaultItem(brokerName, currentLatency, duration);
 }
 } 
}

消息发送实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 private SendResult sendKernelImpl(final Message msg,
 final MessageQueue mq,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final TopicPublishInfo topicPublishInfo,
 final long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 long beginStartTime = System.currentTimeMillis();
 // 获得地址
 String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
 if (null == brokerAddr) {
 tryToFindTopicPublishInfo(mq.getTopic());
 brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
 }

 SendMessageContext context = null;
 if (brokerAddr != null) {
 brokerAddr = MixAll.brokerVIPChannel(this.defaultMQProducer.isSendMessageWithVIPChannel(), brokerAddr);
 // 记录消息内容，因为下面逻辑可能改变消息内容，例如消息压缩
 byte[] prevBody = msg.getBody();
 try {
 //for MessageBatch,ID has been set in the generating process
 if (!(msg instanceof MessageBatch)) {
 MessageClientIDSetter.setUniqID(msg);
 }

 int sysFlag = 0;
 boolean msgBodyCompressed = false;
 if (this.tryToCompressMessage(msg)) {
 sysFlag |= MessageSysFlag.COMPRESSED_FLAG;
 msgBodyCompressed = true;
 }

 final String tranMsg = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
 if (tranMsg != null && Boolean.parseBoolean(tranMsg)) {
 sysFlag |= MessageSysFlag.TRANSACTION_PREPARED_TYPE;
 }
 // 消息发送前校验
 if (hasCheckForbiddenHook()) {
 CheckForbiddenContext checkForbiddenContext = new CheckForbiddenContext();
 checkForbiddenContext.setNameSrvAddr(this.defaultMQProducer.getNamesrvAddr());
 checkForbiddenContext.setGroup(this.defaultMQProducer.getProducerGroup());
 checkForbiddenContext.setCommunicationMode(communicationMode);
 checkForbiddenContext.setBrokerAddr(brokerAddr);
 checkForbiddenContext.setMessage(msg);
 checkForbiddenContext.setMq(mq);
 checkForbiddenContext.setUnitMode(this.isUnitMode());
 this.executeCheckForbiddenHook(checkForbiddenContext);
 }
 // 消息发送前回调处理
 if (this.hasSendMessageHook()) {
 context = new SendMessageContext();
 context.setProducer(this);
 context.setProducerGroup(this.defaultMQProducer.getProducerGroup());
 context.setCommunicationMode(communicationMode);
 context.setBornHost(this.defaultMQProducer.getClientIP());
 context.setBrokerAddr(brokerAddr);
 context.setMessage(msg);
 context.setMq(mq);
 String isTrans = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
 if (isTrans != null && isTrans.equals("true")) {
 context.setMsgType(MessageType.Trans_Msg_Half);
 }

 if (msg.getProperty("__STARTDELIVERTIME") != null || msg.getProperty(MessageConst.PROPERTY_DELAY_TIME_LEVEL) != null) {
 context.setMsgType(MessageType.Delay_Msg);
 }
 this.executeSendMessageHookBefore(context);
 }
 // 构建消息发送请求头
 SendMessageRequestHeader requestHeader = new SendMessageRequestHeader();
 requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup());
 requestHeader.setTopic(msg.getTopic());
 requestHeader.setDefaultTopic(this.defaultMQProducer.getCreateTopicKey());
 requestHeader.setDefaultTopicQueueNums(this.defaultMQProducer.getDefaultTopicQueueNums());
 requestHeader.setQueueId(mq.getQueueId());
 requestHeader.setSysFlag(sysFlag);
 requestHeader.setBornTimestamp(System.currentTimeMillis());
 requestHeader.setFlag(msg.getFlag());
 requestHeader.setProperties(MessageDecoder.messageProperties2String(msg.getProperties()));
 requestHeader.setReconsumeTimes(0);
 requestHeader.setUnitMode(this.isUnitMode());
 requestHeader.setBatch(msg instanceof MessageBatch);
 if (requestHeader.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
 String reconsumeTimes = MessageAccessor.getReconsumeTime(msg);
 if (reconsumeTimes != null) {
 requestHeader.setReconsumeTimes(Integer.valueOf(reconsumeTimes));
 MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_RECONSUME_TIME);
 }

 String maxReconsumeTimes = MessageAccessor.getMaxReconsumeTimes(msg);
 if (maxReconsumeTimes != null) {
 requestHeader.setMaxReconsumeTimes(Integer.valueOf(maxReconsumeTimes));
 MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_MAX_RECONSUME_TIMES);
 }
 }

 SendResult sendResult = null;
 // 消息发送逻辑
 switch (communicationMode) {
 // 异步发送
 case ASYNC:
 Message tmpMessage = msg;
 if (msgBodyCompressed) {
 //If msg body was compressed, msgbody should be reset using prevBody.
 //Clone new message using commpressed message body and recover origin massage.
 //Fix bug:https://github.com/apache/rocketmq-externals/issues/66
 tmpMessage = MessageAccessor.cloneMessage(msg);
 msg.setBody(prevBody);
 }
 long costTimeAsync = System.currentTimeMillis() - beginStartTime;
 if (timeout < costTimeAsync) {
 throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
 }
 sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
 brokerAddr,
 mq.getBrokerName(),
 tmpMessage,
 requestHeader,
 timeout - costTimeAsync,
 communicationMode,
 sendCallback,
 topicPublishInfo,
 this.mQClientFactory,
 this.defaultMQProducer.getRetryTimesWhenSendAsyncFailed(),
 context,
 this);
 break;
 case ONEWAY:
 // 同步发送
 case SYNC:
 long costTimeSync = System.currentTimeMillis() - beginStartTime;
 if (timeout < costTimeSync) {
 throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
 }
 sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
 brokerAddr,
 mq.getBrokerName(),
 msg,
 requestHeader,
 timeout - costTimeSync,
 communicationMode,
 context,
 this);
 break;
 default:
 assert false;
 break;
 }

 if (this.hasSendMessageHook()) {
 // 发送结果放入上下文后返回
 context.setSendResult(sendResult);
 this.executeSendMessageHookAfter(context);
 }

 return sendResult;
 } catch (RemotingException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } catch (MQBrokerException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } catch (InterruptedException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } finally {
 msg.setBody(prevBody);
 }
 }

 throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
 }
}

使用MQClientAPIImpl真正发起Netty网络请求

package org.apache.rocketmq.client.impl;

public class MQClientAPIImpl {
 ...
 public SendResult sendMessage(
 final String addr,
 final String brokerName,
 final Message msg,
 final SendMessageRequestHeader requestHeader,
 final long timeoutMillis,
 final CommunicationMode communicationMode,
 final SendMessageContext context,
 final DefaultMQProducerImpl producer
 ) throws RemotingException, MQBrokerException, InterruptedException {
 return sendMessage(addr, brokerName, msg, requestHeader, timeoutMillis, communicationMode, null, null, null, 0, context, producer);
 }

 /**
 * 封装 RemotingCommand 对象后真正发起网络请求
 */
 public SendResult sendMessage(
 final String addr,
 final String brokerName,
 final Message msg,
 final SendMessageRequestHeader requestHeader,
 final long timeoutMillis,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final TopicPublishInfo topicPublishInfo,
 final MQClientInstance instance,
 final int retryTimesWhenSendFailed,
 final SendMessageContext context,
 final DefaultMQProducerImpl producer
 ) throws RemotingException, MQBrokerException, InterruptedException {
 long beginStartTime = System.currentTimeMillis();
 RemotingCommand request = null;
 if (sendSmartMsg || msg instanceof MessageBatch) {
 SendMessageRequestHeaderV2 requestHeaderV2 = SendMessageRequestHeaderV2.createSendMessageRequestHeaderV2(requestHeader);
 request = RemotingCommand.createRequestCommand(msg instanceof MessageBatch ? RequestCode.SEND_BATCH_MESSAGE : RequestCode.SEND_MESSAGE_V2, requestHeaderV2);
 } else {
 request = RemotingCommand.createRequestCommand(RequestCode.SEND_MESSAGE, requestHeader);
 }
 // 设置请求体
 request.setBody(msg.getBody());

 switch (communicationMode) {
 case ONEWAY:
 this.remotingClient.invokeOneway(addr, request, timeoutMillis);
 return null;
 case ASYNC:
 final AtomicInteger times = new AtomicInteger();
 long costTimeAsync = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTimeAsync) {
 throw new RemotingTooMuchRequestException("sendMessage call timeout");
 }
 this.sendMessageAsync(addr, brokerName, msg, timeoutMillis - costTimeAsync, request, sendCallback, topicPublishInfo, instance,
 retryTimesWhenSendFailed, times, context, producer);
 return null;
 case SYNC:
 long costTimeSync = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTimeSync) {
 throw new RemotingTooMuchRequestException("sendMessage call timeout");
 }
 return this.sendMessageSync(addr, brokerName, msg, timeoutMillis - costTimeSync, request);
 default:
 assert false;
 break;
 }

 return null;
 }

 private SendResult sendMessageSync(
 final String addr,
 final String brokerName,
 final Message msg,
 final long timeoutMillis,
 final RemotingCommand request
 ) throws RemotingException, MQBrokerException, InterruptedException {
 // 同步发网络请求，返回响应结果
 RemotingCommand response = this.remotingClient.invokeSync(addr, request, timeoutMillis);
 assert response != null;
 return this.processSendResponse(brokerName, msg, response);
 }
}

底层Netty通信

类图

`RemotingClient`源码

package org.apache.rocketmq.remoting;

public interface RemotingClient extends RemotingService {

 /**
 * 更新协调服务地址列表
 * @param addrs
 */
 void updateNameServerAddressList(final List<String> addrs);

 /**
 * 更新协调服务地址列表
 * @return
 */
 List<String> getNameServerAddressList();

 /**
 * 同步消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 RemotingCommand invokeSync(final String addr, final RemotingCommand request,
 final long timeoutMillis) throws InterruptedException, RemotingConnectException,
 RemotingSendRequestException, RemotingTimeoutException;

 /**
 * 异步消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeAsync(final String addr, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback) throws InterruptedException, RemotingConnectException,
 RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException;

 /**
 * 单次消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeOneway(final String addr, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingConnectException, RemotingTooMuchRequestException,
 RemotingTimeoutException, RemotingSendRequestException;

 /**
 *用于注册一些底层通信服务，比如进行元数据信息的同步工作，commitlog的同步等
 * @param requestCode 底层业务通信规则码
 * @param processor 注册器（单线程）
 * @param executor 线程池，用于执行注册器的业务逻辑
 */
 void registerProcessor(final int requestCode, final NettyRequestProcessor processor,
 final ExecutorService executor);

 /**
 * 回调函数执行线程设置，用于生产者发送消息后的回调函数
 * @param callbackExecutor
 */
 void setCallbackExecutor(final ExecutorService callbackExecutor);

 /**
 * 获取回调池：NameServer地址
 * @return
 */
 ExecutorService getCallbackExecutor();
 
 /**
 * 用于远程服务地址判断是否可以进行通信
 * @param addr
 * @return
 */
 boolean isChannelWritable(final String addr);
}

registerProcessor方法在实现类中用于初始化时进行注册逻辑

package org.apache.rocketmq.client.impl;

public class MQClientAPIImpl {
 /**
 * Netty 服务启动时初始化工作
 * @param nettyClientConfig 配置信息
 * @param clientRemotingProcessor
 * @param rpcHook
 * @param clientConfig
 */
 public MQClientAPIImpl(final NettyClientConfig nettyClientConfig,
 final ClientRemotingProcessor clientRemotingProcessor,
 RPCHook rpcHook, final ClientConfig clientConfig) {
 this.clientConfig = clientConfig;
 topAddressing = new TopAddressing(MixAll.getWSAddr(), clientConfig.getUnitName());
 this.remotingClient = new NettyRemotingClient(nettyClientConfig, null);
 this.clientRemotingProcessor = clientRemotingProcessor;

 this.remotingClient.registerRPCHook(rpcHook);
 this.remotingClient.registerProcessor(RequestCode.CHECK_TRANSACTION_STATE, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.NOTIFY_CONSUMER_IDS_CHANGED, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.RESET_CONSUMER_CLIENT_OFFSET, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.GET_CONSUMER_STATUS_FROM_CLIENT, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.GET_CONSUMER_RUNNING_INFO, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.CONSUME_MESSAGE_DIRECTLY, this.clientRemotingProcessor, null);
 }
 ...
}

setCallbackExecutor方法用于回调函数执行线程设置，用于生产者发送消息后的回调线程池，例如DefaultMQProducerImpl中的使用方式

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public void setCallbackExecutor(final ExecutorService callbackExecutor) {
	 this.mQClientFactory.getMQClientAPIImpl().getRemotingClient().setCallbackExecutor(callbackExecutor);
 }
 
 /**
 * It will be removed at 4.4.0 cause for exception handling and the wrong Semantics of timeout.
 * A new one will be provided in next version
 * @param msg
 * @param sendCallback
 * @param timeout the <code>sendCallback</code> will be invoked at most time
 * @throws RejectedExecutionException
 */
 @Deprecated
 public void send(final Message msg, final SendCallback sendCallback, final long timeout)
 throws MQClientException, RemotingException, InterruptedException {
 final long beginStartTime = System.currentTimeMillis();
 // 获取回调线程池
 ExecutorService executor = this.getCallbackExecutor();
 try {
 executor.submit(new Runnable() {
 @Override
 public void run() {
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeout > costTime) {
 try {
 sendDefaultImpl(msg, CommunicationMode.ASYNC, sendCallback, timeout - costTime);
 } catch (Exception e) {
 sendCallback.onException(e);
 }
 } else {
 sendCallback.onException(
 new RemotingTooMuchRequestException("DEFAULT ASYNC send call timeout"));
 }
 }

 });
 } catch (RejectedExecutionException e) {
 throw new MQClientException("executor rejected ", e);
 }

 }
	...
}

`RemotingServer`源码

package org.apache.rocketmq.remoting;

public interface RemotingServer extends RemotingService {

 /**
 * 用于注册一些底层通信服务，比如进行元数据信息的同步工作，commitlog的同步等
 * @param requestCode 底层业务通信规则码
 * @param processor 注册器（单线程）
 * @param executor 线程池，用于执行注册器的业务逻辑
 */
 void registerProcessor(final int requestCode, final NettyRequestProcessor processor,
 final ExecutorService executor);

 /**
 * 默认注册方法：比如注册：{@link org.apache.rocketmq.broker.processor.AdminBrokerProcessor}管理Broker工作线程或者namesrv有关的流程器，
 * 比如同步地址、路由topic等信息
 * @param processor
 * @param executor
 */
 void registerDefaultProcessor(final NettyRequestProcessor processor, final ExecutorService executor);

 /**
 * 获取端口
 * @return
 */
 int localListenPort();

 /**
 * 根据业务规则获取Pair<执行器，线程池>
 * @param requestCode 业务规则码
 * @return
 */
 Pair<NettyRequestProcessor, ExecutorService> getProcessorPair(final int requestCode);

 /**
 * 同步消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 RemotingCommand invokeSync(final Channel channel, final RemotingCommand request,
 final long timeoutMillis) throws InterruptedException, RemotingSendRequestException,
 RemotingTimeoutException;

 /**
 * 异步消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeAsync(final Channel channel, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback) throws InterruptedException,
 RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException;

 /**
 * 单向消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeOneway(final Channel channel, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException,
 RemotingSendRequestException;

}

`NettyRemotingServer`源码

其唯一实现类NettyRemotingServer，主要实现Netty编程模型

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingServer extends NettyRemotingAbstract implements RemotingServer {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);
 /**
 * NettyServer辅助类
 */
 private final ServerBootstrap serverBootstrap;
 /**
 * selector组（用于注册并监听event）
 */
 private final EventLoopGroup eventLoopGroupSelector;
 /**
 * 主线程组（用于接收TCP连接）
 */
 private final EventLoopGroup eventLoopGroupBoss;
 /**
 * NettyServer配置
 */
 private final NettyServerConfig nettyServerConfig;

 /**
 * 主要使用在registerProcessor方法中
 */
 private final ExecutorService publicExecutor;

 /**
 * channel事件监听
 */
 private final ChannelEventListener channelEventListener;

 public NettyRemotingServer(final NettyServerConfig nettyServerConfig,
 final ChannelEventListener channelEventListener) {
 super(nettyServerConfig.getServerOnewaySemaphoreValue(), nettyServerConfig.getServerAsyncSemaphoreValue());
 this.serverBootstrap = new ServerBootstrap();
 this.nettyServerConfig = nettyServerConfig;
 this.channelEventListener = channelEventListener;

 int publicThreadNums = nettyServerConfig.getServerCallbackExecutorThreads();
 if (publicThreadNums <= 0) {
 publicThreadNums = 4;
 }

 this.publicExecutor = Executors.newFixedThreadPool(publicThreadNums, new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyServerPublicExecutor_" + this.threadIndex.incrementAndGet());
 }
 });

 this.eventLoopGroupBoss = new NioEventLoopGroup(1, new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyBoss_%d", this.threadIndex.incrementAndGet()));
 }
 });

 // 如果是Linux平台，并且开启了native epoll，就用EpollEventLoopGroup，这个也就是用JNI，调用c写的epoll；否则就用java NIO的NioEventLoopGroup
 if (useEpoll()) {
 this.eventLoopGroupSelector = new EpollEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);
 private int threadTotal = nettyServerConfig.getServerSelectorThreads();

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyServerEPOLLSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
 }
 });
 } else {
 this.eventLoopGroupSelector = new NioEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);
 private int threadTotal = nettyServerConfig.getServerSelectorThreads();

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyServerNIOSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
 }
 });
 }

 loadSslContext();
 }
 
 @Override
 public void start() {
 // 根据配置创建工作线程组
 this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
 nettyServerConfig.getServerWorkerThreads(),
 new ThreadFactory() {

 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyServerCodecThread_" + this.threadIndex.incrementAndGet());
 }
 });
 // Netty编程模型
 ServerBootstrap childHandler =
 this.serverBootstrap.group(this.eventLoopGroupBoss, this.eventLoopGroupSelector)
 .channel(useEpoll() ? EpollServerSocketChannel.class : NioServerSocketChannel.class)
 // sync + accept = backlog
 .option(ChannelOption.SO_BACKLOG, 1024)
 // 允许重复使用本地地址和端口
 .option(ChannelOption.SO_REUSEADDR, true)
 .option(ChannelOption.SO_KEEPALIVE, false)
 .childOption(ChannelOption.TCP_NODELAY, true)
 .childOption(ChannelOption.SO_SNDBUF, nettyServerConfig.getServerSocketSndBufSize())
 .childOption(ChannelOption.SO_RCVBUF, nettyServerConfig.getServerSocketRcvBufSize())
 .localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort()))
 .childHandler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) throws Exception {
 ch.pipeline()
 .addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME,
 new HandshakeHandler(TlsSystemConfig.tlsMode))
 .addLast(defaultEventExecutorGroup,
 new NettyEncoder(),
 new NettyDecoder(),
 new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()),
 new NettyConnectManageHandler(),
 new NettyServerHandler()
 );
 }
 });

 if (nettyServerConfig.isServerPooledByteBufAllocatorEnable()) {
 childHandler.childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
 }

 try {
 ChannelFuture sync = this.serverBootstrap.bind().sync();
 InetSocketAddress addr = (InetSocketAddress) sync.channel().localAddress();
 this.port = addr.getPort();
 } catch (InterruptedException e1) {
 throw new RuntimeException("this.serverBootstrap.bind().sync() InterruptedException", e1);
 }

 if (this.channelEventListener != null) {
 this.nettyEventExecutor.start();
 }

 this.timer.scheduleAtFixedRate(new TimerTask() {

 @Override
 public void run() {
 try {
 NettyRemotingServer.this.scanResponseTable();
 } catch (Throwable e) {
 log.error("scanResponseTable exception", e);
 }
 }
 }, 1000 * 3, 1000);
 }
 ...
}

`NettyRemotingAbstract`源码

NettyRemotingAbstract具体实现三种发送方法

package org.apache.rocketmq.remoting.netty;

public abstract class NettyRemotingAbstract {
 /**
 * 异步调用具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
 long beginStartTime = System.currentTimeMillis();
 final int opaque = request.getOpaque();
 boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
 if (acquired) {
 final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync);
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTime) {
 throw new RemotingTooMuchRequestException("invokeAsyncImpl call timeout");
 }

 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis - costTime, invokeCallback, once);
 this.responseTable.put(opaque, responseFuture);
 try {
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 }
 requestFail(opaque);
 log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel));
 }
 });
 } catch (Exception e) {
 responseFuture.release();
 log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e);
 throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
 }
 } else {
 if (timeoutMillis <= 0) {
 throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast");
 } else {
 String info =
 String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
 timeoutMillis,
 this.semaphoreAsync.getQueueLength(),
 this.semaphoreAsync.availablePermits()
 );
 log.warn(info);
 throw new RemotingTimeoutException(info);
 }
 }
 }
 /**
 * 单向发送具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 public void invokeOnewayImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
 request.markOnewayRPC();
 boolean acquired = this.semaphoreOneway.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
 if (acquired) {
 final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreOneway);
 try {
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 once.release();
 if (!f.isSuccess()) {
 log.warn("send a request command to channel <" + channel.remoteAddress() + "> failed.");
 }
 }
 });
 } catch (Exception e) {
 once.release();
 log.warn("write send a request command to channel <" + channel.remoteAddress() + "> failed.");
 throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
 }
 } else {
 if (timeoutMillis <= 0) {
 throw new RemotingTooMuchRequestException("invokeOnewayImpl invoke too fast");
 } else {
 String info = String.format(
 "invokeOnewayImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
 timeoutMillis,
 this.semaphoreOneway.getQueueLength(),
 this.semaphoreOneway.availablePermits()
 );
 log.warn(info);
 throw new RemotingTimeoutException(info);
 }
 }
 }
 /**
 * 同步调用具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
 final long timeoutMillis)
 throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
 final int opaque = request.getOpaque();

 try {
 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
 this.responseTable.put(opaque, responseFuture);
 final SocketAddress addr = channel.remoteAddress();
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 } else {
 responseFuture.setSendRequestOK(false);
 }

 responseTable.remove(opaque);
 responseFuture.setCause(f.cause());
 responseFuture.putResponse(null);
 log.warn("send a request command to channel <" + addr + "> failed.");
 }
 });

 RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
 if (null == responseCommand) {
 if (responseFuture.isSendRequestOK()) {
 throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
 responseFuture.getCause());
 } else {
 throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
 }
 }

 return responseCommand;
 } finally {
 this.responseTable.remove(opaque);
 }
 }
 ...
}

Netty编码解码源码解析——`RemotingCommand`源码

netty消息数据包结构

RemotingCommand中编码、解码方法体现了其数据包格式和用法

package org.apache.rocketmq.remoting.protocol;

public class RemotingCommand {
 
 public ByteBuffer encode() {
 // 1> header length size
 int length = 4;

 // 2> header data length
 byte[] headerData = this.headerEncode();
 length += headerData.length;

 // 3> body data length
 if (this.body != null) {
 length += body.length;
 }

 ByteBuffer result = ByteBuffer.allocate(4 + length);

 // length
 result.putInt(length);

 // header length
 result.put(markProtocolType(headerData.length, serializeTypeCurrentRPC));

 // header data
 result.put(headerData);

 // body data;
 if (this.body != null) {
 result.put(this.body);
 }

 result.flip();

 return result;
 }
 public static RemotingCommand decode(final ByteBuffer byteBuffer) {
 int length = byteBuffer.limit();
 int oriHeaderLen = byteBuffer.getInt();
 int headerLength = getHeaderLength(oriHeaderLen);

 byte[] headerData = new byte[headerLength];
 // 此处使用的是java nio buffer
 byteBuffer.get(headerData);
 // header封装，长度+类型 对应数据包结构第二位
 RemotingCommand cmd = headerDecode(headerData, getProtocolType(oriHeaderLen));

 int bodyLength = length - 4 - headerLength;
 byte[] bodyData = null;
 if (bodyLength > 0) {
 bodyData = new byte[bodyLength];
 byteBuffer.get(bodyData);
 }
 cmd.body = bodyData;

 return cmd;
 }
 ...
}

其支持的序列化类型有两个

SerializeType序列化类型枚举

package org.apache.rocketmq.remoting.protocol;

public enum SerializeType {
 JSON((byte) 0),
 ROCKETMQ((byte) 1);
 ...
}

在NettyRemotingClient中的Netty编程模型也有编码、解码配置，其使用的是netty buffer

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingClient extends NettyRemotingAbstract implements RemotingClient {
 @Override
 public void start() {
 this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
 nettyClientConfig.getClientWorkerThreads(),
 new ThreadFactory() {

 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyClientWorkerThread_" + this.threadIndex.incrementAndGet());
 }
 });

 Bootstrap handler = this.bootstrap.group(this.eventLoopGroupWorker).channel(NioSocketChannel.class)
 .option(ChannelOption.TCP_NODELAY, true)
 .option(ChannelOption.SO_KEEPALIVE, false)
 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, nettyClientConfig.getConnectTimeoutMillis())
 .option(ChannelOption.SO_SNDBUF, nettyClientConfig.getClientSocketSndBufSize())
 .option(ChannelOption.SO_RCVBUF, nettyClientConfig.getClientSocketRcvBufSize())
 .handler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) throws Exception {
 ChannelPipeline pipeline = ch.pipeline();
 if (nettyClientConfig.isUseTLS()) {
 if (null != sslContext) {
 pipeline.addFirst(defaultEventExecutorGroup, "sslHandler", sslContext.newHandler(ch.alloc()));
 log.info("Prepend SSL handler");
 } else {
 log.warn("Connections are insecure as SSLContext is null!");
 }
 }
 pipeline.addLast(
 // 使用工作线程
 defaultEventExecutorGroup,
 // 编码
 new NettyEncoder(),
 // 解码
 new NettyDecoder(),
 new IdleStateHandler(0, 0, nettyClientConfig.getClientChannelMaxIdleTimeSeconds()),
 // 连接管理器
 new NettyConnectManageHandler(),
 new NettyClientHandler());
 }
 });

 this.timer.scheduleAtFixedRate(new TimerTask() {
 @Override
 public void run() {
 try {
 NettyRemotingClient.this.scanResponseTable();
 } catch (Throwable e) {
 log.error("scanResponseTable exception", e);
 }
 }
 }, 1000 * 3, 1000);

 if (this.channelEventListener != null) {
 this.nettyEventExecutor.start();
 }
 }
}

NettyEncoder源码

package org.apache.rocketmq.remoting.netty;

import io.netty.buffer.ByteBuf;
import java.nio.ByteBuffer;

public class NettyEncoder extends MessageToByteEncoder<RemotingCommand> {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);

 @Override
 public void encode(ChannelHandlerContext ctx, RemotingCommand remotingCommand, ByteBuf out)
 throws Exception {
 try {
 ByteBuffer header = remotingCommand.encodeHeader();
 out.writeBytes(header);
 byte[] body = remotingCommand.getBody();
 if (body != null) {
 out.writeBytes(body);
 }
 } catch (Exception e) {
 log.error("encode exception, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e);
 if (remotingCommand != null) {
 log.error(remotingCommand.toString());
 }
 RemotingUtil.closeChannel(ctx.channel());
 }
 }
}

NettyDecoder源码

其继承LengthFieldBasedFrameDecoder，支持灵活配置编解码参数

package org.apache.rocketmq.remoting.netty;

import io.netty.buffer.ByteBuf;
import io.netty.handler.codec.LengthFieldBasedFrameDecoder;
import java.nio.ByteBuffer;

public class NettyDecoder extends LengthFieldBasedFrameDecoder {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);

 private static final int FRAME_MAX_LENGTH =
 Integer.parseInt(System.getProperty("com.rocketmq.remoting.frameMaxLength", "16777216"));

 public NettyDecoder() {
 super(FRAME_MAX_LENGTH, 0, 4, 0, 4);
 }

 @Override
 public Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception {
 ByteBuf frame = null;
 try {
 frame = (ByteBuf) super.decode(ctx, in);
 if (null == frame) {
 return null;
 }

 ByteBuffer byteBuffer = frame.nioBuffer();

 return RemotingCommand.decode(byteBuffer);
 } catch (Exception e) {
 log.error("decode exception, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e);
 RemotingUtil.closeChannel(ctx.channel());
 } finally {
 if (null != frame) {
 frame.release();
 }
 }

 return null;
 }
}

消息返回状态详解

SEND_OK：消息发送成功。
FLUSH_DISK_TIMEOUT：消息发送成功，但消息刷盘失败，此时服务器宕机则消息丢失
FLUSH_SLAVE_TIMEOUT：主从同步消息时超时，从节点已经收到消息，若此时从节点宕机，则从节点消息丢失
SLAVE_NOT_AVAILABLE：从节点不可用，此时主节点宕机，则消息丢失

对于可靠消息投递，后三种情况均应该考虑补偿。

延迟消息投递

场景：消息发送到Broker后，要特定的时间才会被Consumer消费

特点：目前只支持固定精度的定时消息

固定精度消息原因：支持任意精度消息时，对消息持久化、排序产生较大性能开销

使用方法：对message设置setDelayTimeLevel级别即可

核心类：MessageStoreConfig和ScheduleMessageService

MessageStoreConfig源码

package org.apache.rocketmq.store.config;

public class MessageStoreConfig {
 // 延时投递级别，固定精度
 private String messageDelayLevel = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h";
 ...
}

自定义消息发送规则

场景：把消息发送到指定的队列（Message Queue）

处理类：MessageQueueSelector

用法：调用send(Message msg, MessageQueueSelector selector, Object arg)方法即可

实现代码片段，指定发送到id为1的队列中

SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
 // 外部方法的arg会传递到Object中
 @Override
 public MessageQueue select(List<MessageQueue> list, Message message, Object o) {
 Integer queueNumber = (Integer) o;
 return list.get(queueNumber);
 }
},1);

RocketMQ学习笔记2——入门

Wed, 23 Dec 2020 19:06:00 +0800

生产者模型使用

步骤

创建生产者对象DefaultMQProducer
设置NamesrvAddr
启动生产者服务
创建消息并发送

简单消息发送

创建测试主题

常量定义

public class Const {

	public static final String NAMESRV_ADDR_SINGLE = "10.139.12.52:9876";
}

public class Producer {

	public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
		
		DefaultMQProducer producer = new DefaultMQProducer("test_quick_producer_name");
		
		producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
		producer.start();
		
		for(int i = 0 ; i <5; i ++) {
			//	1.	创建消息
			Message message = new Message("test_quick_topic",	//	主题
					"TagA", //	标签
					"key" + i,	// 	用户自定义的key ,唯一的标识
					("Hello RocketMQ" + i).getBytes());	//	消息内容实体（byte[]）
			SendResult sr = producer.send(message, new MessageQueueSelector() {
				@Override
				public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
					Integer queueNumber = (Integer)arg;
					return mqs.get(queueNumber);
				}
			}, 2);
			System.err.println(sr);
			System.err.println("消息发出: " + sr);
		}
		producer.shutdown();
	}
}

运行结果

SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4]
10:45:15.129 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10911] result: true
10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10909] result: true
10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:9876] result: true

消费者模型使用

步骤

创建消费者对象DefaultMQPushConsumer
设置NamesrvAddr及消费位置ConsumeFromWhere
进行主题订阅Subscribe
注册监听并消费registerMessageListener

简单消息消费

public class Consumer {

	public static void main(String[] args) throws MQClientException {
		// 采用Push方式
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test_quick_consumer_name");
		
		consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
		// 匹配所有Tag
		consumer.subscribe("test_quick_topic", "*");
		
		consumer.registerMessageListener(new MessageListenerConcurrently() {
			
			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
				MessageExt me = msgs.get(0);
				try {
					String topic = me.getTopic();
					String tags = me.getTags();
					String keys = me.getKeys();
					
					String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET);
					System.err.println("topic: " + topic + ",tags: " + tags + ", keys: " + keys + ",body: " + msgBody);
				} catch (Exception e) {
					e.printStackTrace();
 // 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
					return ConsumeConcurrentlyStatus.RECONSUME_LATER;
				}
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		consumer.start();
		System.err.println("consumer start...");
	}
}

模拟异常重试

public class Consumer {

	public static void main(String[] args) throws MQClientException {
		// 采用Push方式
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test_quick_consumer_name");
		
		consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
		consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
		// 匹配所有Tag
		consumer.subscribe("test_quick_topic", "*");
		
		consumer.registerMessageListener(new MessageListenerConcurrently() {
			
			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
				MessageExt me = msgs.get(0);
				try {
					String topic = me.getTopic();
					String tags = me.getTags();
					String keys = me.getKeys();
					// 模拟异常
					if(keys.equals("key1")) {
						System.err.println("消息消费失败..");
						int a = 1/0;
					}
					
					String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET);
					System.err.println("topic: " + topic + ",tags: " + tags + ", keys: " + keys + ",body: " + msgBody);
				} catch (Exception e) {
					e.printStackTrace();
					// 重发次数
					int recousumeTimes = me.getReconsumeTimes();
					System.err.println("recousumeTimes: " + recousumeTimes);
					// 超过3次不再重发
					if(recousumeTimes == 3) {
						//		记录日志....
						// 	做补偿处理
						return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
					}
					// 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
					return ConsumeConcurrentlyStatus.RECONSUME_LATER;
				}
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		consumer.start();
		System.err.println("consumer start...");
	}
}

日志输出

consumer start...
topic: test_quick_topic,tags: TagA, keys: key0,body: Hello RocketMQ0
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 0
topic: test_quick_topic,tags: TagA, keys: key2,body: Hello RocketMQ2
topic: test_quick_topic,tags: TagA, keys: key3,body: Hello RocketMQ3
topic: test_quick_topic,tags: TagA, keys: key4,body: Hello RocketMQ4
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 1
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 2
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 3

四种集群环境

集群环境拓扑图

分类

单点模式（不推荐）
主从模式
双主模式（不推荐）
双主双从、多主多从模式

主从集群环境构建

优势：

主从模式可以保障消息的即时性与可靠性
当投递一条消息后关闭主节点，从节点可以继续提供消费者进行消费，但不能接受消息
主节点上线后会进行消费进度offset同步

基础环境

选择2台服务器进行部署验证

IP	IP
GY11-SyOCM33-101	10.139.12.52	主
GY11-SyOCM33-201	10.139.12.53	从

53节点按照单点部署时操作好对应目录和日志配置

修改主节点配置

52节点修改主节点配置文件broker-a.properties如下

修改了namesrvAddr

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876;10.139.12.53:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

修改从节点配置

修改从节点配置文件broker-a-s.properties如下

拷贝主节点配置后修改id和brokerRole

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876;10.139.12.53:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

主从配置同步到53节点即可

启动NameSrv

启动Namesrv（主从节点均执行）

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin
$ nohup sh mqnamesrv &

启动Broker

主节点启动

$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &

从节点启动

$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a-s.properties >/dev/null 2>&1 &

启动控制台

修改控制台配置地址参数为

rocketmq.config.namesrvAddr=10.139.12.52:9876;10.139.12.53:9876

数据高可用机制故障演练

投递一条消息

关闭主节点，剩余一个slave

启动消费者，消费正常

主节点上线，同步消息位点（offset）使消费端不会重复消费

RocketMQ学习笔记1——初探

Wed, 23 Dec 2020 19:04:00 +0800

RocketMQ整体认知

优势

支持集群模型、负载均衡、水平扩展能力
亿级别的消息堆积能力
采用零拷贝的原理、顺序写盘、随机读
丰富的API使用
代码优秀，底层通信框架采用Netty NIO框架
NameServer代替Zookeeper
强调集群无单点，可扩展，任意一点高可用，水平可扩展
消息失败重试机制，消息可查询
开源社区活跃，成熟度高（经过双十一考验）

核心概念模型

基本概念

消息生产者（Producer）：负责生产消息，一般由业务系统负责生产消息。
消息消费者（Consumer）：负责消费消息，一般是后台系统负责异步消费。
推动式消费（Push Consumer）： Consumer消费的一种类型，该模式下Broker收到数据后会主动推送给消费端。
拉取式消费（Pull Consumer）： Consumer消费的一种类型，应用通常主动调用Consumer的拉消息方法从Broker服务器拉消息。
生产者组（Producer Group）：同一类Producer的集合，这类Producer发送同一类消息且发送逻辑一致。
消费者组（Consumer Group）：同一类Consumer的集合，这类Consumer通常消费同一类消息且消费逻辑一致。
Broker：MQ消息服务（中转角色，用于消息存储于生产消息转发）

源码结构

源码分析以4.4.0为例

源码地址： https://github.com/apache/rocketmq

结构：

结构解析：

rocketmq-common：公用数据结构
rocketmq-distribution：编译模块，编译输出
rocketmq-filter：进行Broker过滤的不感兴趣的消息传输，减小带宽压力
rocketmq-logappender、rocketmq-logging：日志服务
rocketmq-namesrv：Namesrv服务，用于服务协调
rocketmq-openmessaging：对外提供服务
rocketmq-remoting：远程调用接口，封装Netty底层通信
rocketmq-srvutil：提供一些公用的工具方法，比如命令行解析参数
rocketmq-store：消息存储
rocketmq-test、rocketmq-example
rocketmq-tools：管理工具，比如有名的mqadmin工具

编译部署&环境搭建

选择1台服务器进行部署验证

IP	IP
GY11-SyOCM33-101	10.139.12.52

编译部署

在源码目录执行一下命令编译工程

$ mvn -Prelease-all -DskipTests clean package

编译成功后在rocketmq\distribution\target目录下生成apache-rocketmq.tar.gz包，使用该包解压与从官网下载一致，其他步骤参考二进制部署。

二进制包部署（单点）

此处从官网下载4.4.0版本并上传至服务器解压

下载地址

解压到~/rocketmq-test/rocketmq-all-4.4.0-bin-release/如下

$ ll
total 36
drwxr-xr-x 2 ocmuser gpocm 83 Jan 17 2019 benchmark
drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 bin
drwxr-xr-x 5 ocmuser gpocm 196 Jan 17 2019 conf
drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 lib
-rw-r--r-- 1 ocmuser gpocm 17336 Jan 17 2019 LICENSE
-rw-r--r-- 1 ocmuser gpocm 1337 Jan 17 2019 NOTICE
-rw-r--r-- 1 ocmuser gpocm 2481 Jan 17 2019 README.md

创建存储目录

$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index

RocketMQ 配置文件

进入提供好的样例配置文件目录，其中包含双主双从异步复制、双主双从同步复制和双主集群配置文件，此处选择双主双从异步复制作为配置模板

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf]$ll
total 36
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync
drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave
-rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf
-rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml
-rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml
-rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml
-rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml
-rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml

选择broker-a配置文件

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async]$ll
total 16
-rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-a.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties
-rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-b.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties

删除原有配置，加入如下配置（单点）

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

修改日志配置文件

$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs 
$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf && sed -i 's#${user.home}#/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release#g' *.xml

修改启动脚本参数

将默认8g的内存配置改成1g

$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runbroker.sh

runbroker.sh

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"
...

同理

$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runserver.sh

runserver.sh

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"
...

启动 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ nohup sh mqnamesrv &

检查进程

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin]$jps
65322 NamesrvStartup
65534 Jps

启动 BrokerServer A

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &

$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/namesrv.log

停止与数据清理

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ sh mqshutdown broker 
$ sh mqshutdown namesrv 
# --等待停止 
$ rm -rf /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index 
# --按照上面步骤重启 NameServer 与 BrokerServer

SpringBoot 控制台讲解

克隆工程：https://github.com/apache/rocketmq-externals.git

进入控制台配置文件rocketmq-externals\rocketmq-console\src\main\resources\application.properties修改配置

server.contextPath=
server.port=9090

## SSL setting
#server.ssl.key-store=classpath:rmqcngkeystore.jks
#server.ssl.key-store-password=rocketmq
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=rmqcngkey

#spring.application.index=true
spring.application.name=rocketmq-console
spring.http.encoding.charset=UTF-8
spring.http.encoding.enabled=true
spring.http.encoding.force=true
logging.config=classpath:logback.xml
#if this value is empty,use env value rocketmq.config.namesrvAddr NAMESRV_ADDR | now, you can set it in ops page.default localhost:9876
rocketmq.config.namesrvAddr=10.139.12.52:9876
#if you use rocketmq version < 3.5.8, rocketmq.config.isVIPChannel should be false.default true
rocketmq.config.isVIPChannel=
#rocketmq-console's data path:dashboard/monitor
rocketmq.config.dataPath=./data
#set it false if you don't want use dashboard.default true
rocketmq.config.enableDashBoardCollect=true
#set the message track trace topic if you don't want use the default one

SpringBoot源码分析2——走向自动装配

Thu, 10 Dec 2020 17:28:00 +0800

Spring Framework 手动装配

Spring模式注解装配

定义:一种用于声明在应用中扮演“组件”角色的注解
举例:@Component、@Service、@Configuration等
装配:< context: component-scan>或@Componentscan

官方仓库说明

Stereotype Annotations
A *stereotype annotation* is an annotation that is used to declare the role that a component plays within the application. For example, the @Repository annotation in the Spring Framework is a marker for any class that fulfills the role or stereotype of a repository (also known as Data Access Object or DAO).

@Component is a generic stereotype for any Spring-managed component. Any component annotated with @Component is a candidate for component scanning. Similarly, any component annotated with an annotation that is itself meta-annotated with @Component is also a candidate for component scanning. For example, @Service is meta-annotated with @Component.

Core Spring provides several stereotype annotations out of the box, including but not limited to: @Component, @Service, @Repository, @Controller, @RestController, and @Configuration. @Repository, @Service, etc. are specializations of @Component.

模式注解是一种用于声明在应用中扮演“组件”角色的注解。如 Spring Framework 中的 @Repository 标注在任何类上，用于扮演仓储角色的模式注解。

@Component 作为一种由 Spring 容器托管的通用模式组件，任何被 @Component 标准的组件均为组件扫描的候选对象。类似地，凡是被 @Component 元标注（meta-annotated）的注解，如 @Service ，当任何组件标注它时，也被视作组件扫描的候选对象（派生性）

模式注解举例

Spring Framework 注解	场景说明	起始版本
`@Repository`	数据仓储模式注解	2.0
`@Component`	通用组件模式注解	2.5
`@Service`	服务模式注解	2.5
`@Controller`	Web控制器模式注解	2.5
`@Configuration`	配置类模式注解	3.0

装配方式

<context:component-scan> 方式

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:context="http://www.springframework.org/schema/context"
 xsi:schemaLocation="http://www.springframework.org/schema/beans 
http://www.springframework.org/schema/beans/spring-beans.xsd
 http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-
context.xsd">
 
 <!-- 激活注解驱动特性 -->
 <context:annotation-config />
 
 <!-- 找寻被 @Component 或者其派生 Annotation 标记的类（Class），将它们注册为 Spring Bean -->
 <context:component-scan base-package="top.fjy8018.spring.boot" />
 
</beans>

@ComponentScan 方式

@ComponentScan(basePackages = "top.fjy8018.spring.boot")
public class SpringConfiguration {
 ...
}

Spring Framework手动装配

自定义模式注解

`@Component` “派生性”

package top.fjy8018.springboot.annotation;

import org.springframework.stereotype.Component;
import org.springframework.stereotype.Repository;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 10:36
 */
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Repository
public @interface FirstLevelRepository {

 /**
 * 保证签名一致性
 */
 String value() default "";
}

三个级别并无明显区别，体现注解派生性，即使用@FirstLevelRepository和使用@Component效果一致

派生关系

@Component
- @Repository
  - @FirstLevelRepository

package org.springframework.stereotype;

/**
 * @since 2.5
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Indexed
public @interface Component {

 /**
 * Component的签名
 */
 String value() default "";

}

Repository中元标注了@Component

package org.springframework.stereotype;

/**
 * @since 2.0
 * @see Component
 */
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Component
public @interface Repository {

 /**
 * 签名与Component一致
 */
 @AliasFor(annotation = Component.class)
 String value() default "";
}

`@Component` “层次性”

package top.fjy8018.springboot.annotation;

import org.springframework.stereotype.Component;
import org.springframework.stereotype.Repository;

import java.lang.annotation.*;

/**
 * Component层次性
 * 顶级 {@link Component}
 * 一级 {@link Repository @Repository}
 * 二级 {@link FirstLevelRepository}
 * 三级 {@link SecondLevelRepository}
 *
 * @author F嘉阳
 * @date 2018-09-09 10:54
 */
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@FirstLevelRepository
public @interface SecondLevelRepository {

 /**
 * 层次性、派生性都必须保证签名一致
 */
 String value() default "";
}

@Component
- @Repository
  - FirstLevelRepository
    - SecondLevelRepository

注解目标类，value 为bean的名称，此处不管用@FirstLevelRepository还是用SecondLevelRepository，效果都和使用@Component一致

package top.fjy8018.springboot.repository;

import top.fjy8018.springboot.annotation.FirstLevelRepository;

/**
 * @author F嘉阳
 * @date 2018-09-09 10:40
 */
//@Component("myRepository")
@FirstLevelRepository(value = "myRepository")
//@SecondLevelRepository(value = "myRepository")
public class RepositoryDemo {
}

仓储引导类，使用@ComponentScan指定扫描Bean路径

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.ComponentScan;
import top.fjy8018.springboot.repository.RepositoryDemo;

/**
 * @author F嘉阳
 * @date 2018-09-09 10:43
 */
@ComponentScan(basePackages = "top.fjy8018.springboot.repository")
public class RepositoryBootstrapDemo {

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(RepositoryBootstrapDemo.class)
 .web(WebApplicationType.NONE)
 .run(args);

 // 通过bean名称获取bean
 RepositoryDemo myRepository = applicationContext.getBean("myRepository", RepositoryDemo.class);

 // myRepository是否存在，若不存在则报错
 System.out.println("myRepository是否存在：" + myRepository);

 // 关闭上下文
 applicationContext.close();
 }
}

执行结果

SpringBoot源码中也有体现该层次关系

@SpringBootApplication
- @SpringBootConfiguration
  - @Configuration
    - @Component
- @EnableAutoConfiguration
  - @AutoConfigurationPackage

Spring @Enable 模块装配

Spring Framework 3.1 开始支持“@Enable 模块驱动”。所谓“模块”是指具备相同领域的功能组件集合，组合所形成一个独立的单元。比如 Web MVC 模块、AspectJ代理模块、Caching（缓存）模块、JMX（Java 管理扩展）模块、Async（异步处理）模块等。

举例:@EnableWebmvc、@EnableAutoConfiguration等
实现:注解方式、编程方式

@Enable 注解模块举例

框架实现	`@Enable` 注解模块	激活模块
Spring Framework	`@EnableWebMvc`	Web
	`@EnableTransactionManagement`	事务管理模块
	`@EnableCaching`	Caching 模块
	`@EnableMBeanExport`	JMX 模块
	`@EnableAsync`	异步处理模块
	`EnableWebFlux`	Web Flux 模块
	`@EnableAspectJAutoProxy`	AspectJ 代理模块

Spring Boot	`@EnableAutoConfiguration`	自动装配模块
	`@EnableManagementContext`	Actuator 管理模块
	`@EnableConfigurationProperties`	配置属性绑定模块
	`@EnableOAuth2Sso`	OAuth2 单点登录模块

Spring Cloud	`@EnableEurekaServer`	Eureka服务器模块
	`@EnableConfigServer`	配置服务器模块
	`@EnableFeignClients`	Feign客户端模块
	`@EnableZuulProxy`	服务网关
	`@EnableCircuitBreaker`	服务熔断模块

实现方式

注解驱动方式

Spring源码，使用@Import注入配置Bean

package org.springframework.web.servlet.config.annotation;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Import(DelegatingWebMvcConfiguration.class)
public @interface EnableWebMvc {}

接口编程方式

Spring源码——激活缓存注解

package org.springframework.cache.annotation;

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Import(CachingConfigurationSelector.class)
public @interface EnableCaching {...}

CachingConfigurationSelector类图

ImportSelector接口源码，该接口在3.1引入，目的在于增加灵活度

selectImports方法返回数组，即可以死是多个配置Bean

package org.springframework.context.annotation;

/**
 * @since 3.1
 */
public interface ImportSelector {
 
 String[] selectImports(AnnotationMetadata importingClassMetadata);
}

@Enable注解方式自定义实现

激活HelloWorld模块，使用@Import(HelloWorldConfiguration.class)直接注入HelloWorldConfiguration模块

package top.fjy8018.springboot.annotation;

import org.springframework.context.annotation.Import;
import top.fjy8018.springboot.configuration.HelloWorldConfiguration;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:35
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Import(HelloWorldConfiguration.class)
public @interface EnableHelloWorld {}

目标类，其注解@Configuration在Spring3.0时引入

package top.fjy8018.springboot.configuration;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:31
 */
@Configuration
public class HelloWorldConfiguration {

 /**
 * 方法名称即Bean名称
 * @return Hello World!
 */
 @Bean
 public String helloWorld() {
 return "Hello world!";
 }
}

package org.springframework.context.annotation;

/**
 * @since 3.0
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Component
public @interface Configuration {
 @AliasFor(annotation = Component.class)
 String value() default "";
}

@Enable接口编程方式自定义实现

参考Spring包名路径创建自定义Selector，通过HelloWorldImportSelector实现HelloWorldConfiguration的配置Bean导入

package top.fjy8018.springboot.annotation;

import org.springframework.context.annotation.ImportSelector;
import org.springframework.core.type.AnnotationMetadata;
import top.fjy8018.springboot.configuration.HelloWorldConfiguration;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:28
 */
public class HelloWorldImportSelector implements ImportSelector {

 /**
 * @param importingClassMetadata 元数据信息
 * @return 一个或多个配置Bean
 */
 @Override
 public String[] selectImports(AnnotationMetadata importingClassMetadata) {
 return new String[]{HelloWorldConfiguration.class.getName()};
 }
}

激活模块方式变化，使用@Import(HelloWorldImportSelector.class)注入HelloWorldImportSelector模块，在Selector里加载并返回 HelloWorldConfiguration 模块

package top.fjy8018.springboot.annotation;

import org.springframework.context.annotation.Import;
import top.fjy8018.springboot.configuration.HelloWorldConfiguration;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:35
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
//@Import(HelloWorldConfiguration.class)
@Import(HelloWorldImportSelector.class)
public @interface EnableHelloWorld {
}

实现逻辑

graph LR
A(EnableHelloWorld) --> B(HelloWorldImportSelector) 
 B-->C(HelloWorldConfiguration)
 C --> D(helloWorld)

此方法比直接注解驱动更有弹性，可增加switch条件返回不同的模块，源码样例CachingConfigurationSelector

package org.springframework.cache.annotation;
public class CachingConfigurationSelector extends AdviceModeImportSelector<EnableCaching> {
 
 @Override
 public String[] selectImports(AdviceMode adviceMode) {
 switch (adviceMode) {
 case PROXY:
 return getProxyImports();
 case ASPECTJ:
 return getAspectJImports();
 default:
 return null;
 }
 }
 ...
}

Spring条件装配

从 Spring Framework 3.1 开始，允许在 Bean 装配时增加前置条件判断

Spring 注解	场景说明	起始版本
`@Profile`	配置化条件装配	3.1
`@Conditional`	编程条件装配	4.0

条件注解实现方式

配置方式 - `@Profile`

在spring4后实现方式发生变化，依赖@Conditional和ProfileCondition实现

package org.springframework.context.annotation;

/**
* @since 3.1
*/
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(ProfileCondition.class)
public @interface Profile {...}

package org.springframework.context.annotation;

/**
 * @since 4.0
 */
class ProfileCondition implements Condition {...}

通过上下文对象和注解元信息判断

package org.springframework.context.annotation;

/**
 * @since 4.0
 */
@FunctionalInterface
public interface Condition {
 boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata);
}

编程方式 - `@Conditional`

ConditionalOnClass用于判断标识类在当前classpath下是否存在，该注解使用@Conditional搭配OnClassCondition实现

package org.springframework.boot.autoconfigure.condition;

/**
 * {@link Conditional} that only matches when the specified classes are on the classpath.
 */
@Target({ ElementType.TYPE, ElementType.METHOD })
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(OnClassCondition.class)
public @interface ConditionalOnClass {...}

package org.springframework.boot.autoconfigure.condition;

/**
 * {@link Condition} that checks if properties are defined in environment.
 * @since 1.1.0
 * @see ConditionalOnProperty
 */
@Order(Ordered.HIGHEST_PRECEDENCE + 40)
class OnPropertyCondition extends SpringBootCondition {
 @Override
 public ConditionOutcome getMatchOutcome(ConditionContext context,
 AnnotatedTypeMetadata metadata) {
 // 获取元信息
 List<AnnotationAttributes> allAnnotationAttributes = annotationAttributesFromMultiValueMap(
 metadata.getAllAnnotationAttributes(
 ConditionalOnProperty.class.getName()));
 List<ConditionMessage> noMatch = new ArrayList<>();
 List<ConditionMessage> match = new ArrayList<>();
 for (AnnotationAttributes annotationAttributes : allAnnotationAttributes) {
 ConditionOutcome outcome = determineOutcome(annotationAttributes,
 context.getEnvironment());
 (outcome.isMatch() ? match : noMatch).add(outcome.getConditionMessage());
 }
 if (!noMatch.isEmpty()) {
 return ConditionOutcome.noMatch(ConditionMessage.of(noMatch));
 }
 return ConditionOutcome.match(ConditionMessage.of(match));
 }
 ...
}

OnPropertyCondition为抽象类，实际仍然是Condition接口的实现

package org.springframework.boot.autoconfigure.condition;

public abstract class SpringBootCondition implements Condition {...}

ConditionalOnProperty用于判断配置在上下文中是否存在

package org.springframework.boot.autoconfigure.condition;

/**
 * @since 1.1.0
 */
@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.TYPE, ElementType.METHOD })
@Documented
@Conditional(OnPropertyCondition.class)
public @interface ConditionalOnProperty {...}

自定义条件装配

目标服务

package top.fjy8018.springboot.service;

/**
 * 计算模块
 * @author F嘉阳
 * @date 2018-09-09 11:56
 */
public interface CalculateService {

 /**
 * 多整数求和
 * @param values 多个整数
 * @return 累加值
 */
 Integer sum(Integer... values);
}

基于配置方式

服务实现类，分别使用 for循环实现和Lambda实现，均标注@Service作为Bean自动注入

指定@Profile("Java7")，当条件为Java7时才装配

package top.fjy8018.springboot.service.impl;

import org.springframework.context.annotation.Profile;
import org.springframework.stereotype.Service;
import top.fjy8018.springboot.service.CalculateService;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:57
 */
@Profile("Java7")
@Service
public class Java7CalculateServiceImpl implements CalculateService {

 /**
 * 多整数求和
 *
 * @param values 多个整数
 * @return 累加值
 */
 @Override
 public Integer sum(Integer... values) {
 System.out.println("计算模块java7 for循环实现");
 int sum = 0;
 for (int i = 0; i < values.length; i++) {
 sum += values[i];
 }
 return sum;
 }
}

指定@Profile("Java8")，当条件为Java7时才装配

package top.fjy8018.springboot.service.impl;

import org.springframework.context.annotation.Profile;
import org.springframework.stereotype.Service;
import top.fjy8018.springboot.service.CalculateService;

import java.util.stream.Stream;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:57
 */
@Profile("Java8")
@Service
public class Java8CalculateServiceImpl implements CalculateService {

 /**
 * 多整数求和
 * @param values 多个整数
 * @return 累加值
 */
 @Override
 public Integer sum(Integer... values) {
 System.out.println("计算模块java8 Lambda实现");
 return Stream.of(values).reduce(0, Integer::sum);
 }
}

启动类，通过scanBasePackages指定报名扫描，该方法与@ComponentScan(basePackages = "xxx")效果一致

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import top.fjy8018.springboot.service.CalculateService;

/**
 * @author F嘉阳
 * @date 2018-09-09 12:06
 */
@SpringBootApplication(scanBasePackages = "top.fjy8018.springboot.service")
public class CalculateServiceDemo {

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(CalculateServiceDemo.class)
 .profiles("Java8")
 .web(WebApplicationType.NONE)
 .run(args);

 // 直接用类获取bean
 CalculateService calculateService = applicationContext.getBean(CalculateService.class);

 // calculateService Bean 是否存在，若不添加Profile则必有报错
 Integer[] values = new Integer[10];
 for (int i = 1; i <= values.length; i++) {
 values[i - 1] = i;
 }
 System.out.println("calculateService Bean 求和: " + calculateService.sum(values));

 // 关闭上下文
 applicationContext.close();
 }
}

执行结果——java8

执行结果——java7

若不指定配置则报错

基于编程方式

系统属性条件判断实现类

package top.fjy8018.springboot.condition;

import org.springframework.context.annotation.Condition;
import org.springframework.context.annotation.ConditionContext;
import org.springframework.core.type.AnnotatedTypeMetadata;

import java.util.Map;

/**
 * @author F嘉阳
 * @date 2018-09-09 17:15
 */
public class OnSystemPropertyCondition implements Condition {

 /**
 * 条件匹配规则，此处可实现复杂逻辑判断
 * @param context 上下文
 * @param metadata 元信息
 * @return 是否匹配
 */
 @Override
 public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {

 // 通过类名称取元信息
 Map<String, Object> annotationAttributes = metadata.getAnnotationAttributes(ConditionalOnSystemProperty.class.getName());

 String propertyName = String.valueOf(annotationAttributes.get("name"));
 String propertyValue = String.valueOf(annotationAttributes.get("value"));

 // 取系统属性
 String systemPropertyValue = System.getProperty(propertyName);

 return propertyValue.equals(systemPropertyValue);
 }

Java 系统属性条件判断类装配，系统属性通过Java API获取java.lang.System#getProperty(java.lang.String)方法获取的属性值

package top.fjy8018.springboot.condition;

import org.springframework.context.annotation.Conditional;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 17:14
 */
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(OnSystemPropertyCondition.class)
public @interface ConditionalOnSystemProperty {

 /**
 * Java 系统属性名称
 * @return 系统属性名称
 */
 String name();

 /**
 * 系统属性值
 * @return 属性值
 */
 String value();
}

启动类，helloWorld仅在系统条件满足时自动装配，注解若作用于类上语义相同

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.Bean;
import top.fjy8018.springboot.condition.ConditionalOnSystemProperty;

/**
 * 系统属性条件引导类
 *
 * @author F嘉阳
 * @date 2018-09-09 17:26
 */
public class SystemPropertyConditionBootstrap {

 @Bean
 @ConditionalOnSystemProperty(name = "user.name", value = "11860")
 public String helloWorld() {
 return "Hello SpringBoot2.0";
 }

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(SystemPropertyConditionBootstrap.class)
 .web(WebApplicationType.NONE)
 .run(args);

 // 通过类型和方法名获取Bean
 String helloWorldBean = applicationContext.getBean("helloWorld", String.class);
 // 若系统参数条件不匹配则报找不到Bean错误，以此实现编程方式条件加载
 System.out.println("ConditionalOnSystemProperty 加载内容：" + helloWorldBean);

 applicationContext.close();
 }
}

执行结果

条件不满足则报错

Spring Boot 自动装配

在 Spring Boot 场景下，基于规约大于配置的原则，实现 Spring 组件自动装配的目的（0配置）。

装配: 模式注解、@Eηable模块、条件装配、工厂加载机制
实现: 激活自动装配、实现自动装配、配置自动装配实现

底层装配技术

Spring 模式注解装配
Spring @Enable 模块装配
Spring 条件装配装配
Spring 工厂加载机制
- 实现类： SpringFactoriesLoader
- 配置资源： META-INF/spring.factories

package org.springframework.core.io.support;

/**
 * @since 3.2
 */
public abstract class SpringFactoriesLoader {

 public static final String FACTORIES_RESOURCE_LOCATION = "META-INF/spring.factories";
 
 public static <T> List<T> loadFactories(Class<T> factoryClass, @Nullable ClassLoader classLoader) {
 Assert.notNull(factoryClass, "'factoryClass' must not be null");
 // 获取类加载器
 ClassLoader classLoaderToUse = classLoader;
 if (classLoaderToUse == null) {
 classLoaderToUse = SpringFactoriesLoader.class.getClassLoader();
 }
 List<String> factoryNames = loadFactoryNames(factoryClass, classLoaderToUse);
 if (logger.isTraceEnabled()) {
 logger.trace("Loaded [" + factoryClass.getName() + "] names: " + factoryNames);
 }
 List<T> result = new ArrayList<>(factoryNames.size());
 // 迭代装配
 for (String factoryName : factoryNames) {
 result.add(instantiateFactory(factoryName, factoryClass, classLoaderToUse));
 }
 AnnotationAwareOrderComparator.sort(result);
 return result;
 }
 ...
}

自动装配举例

参考 META-INF/spring.factories

实现方法

激活自动装配 - @EnableAutoConfiguration
实现自动装配 - XXXAutoConfiguration
配置自动装配实现 - META-INF/spring.factories

目的在于使用META-INF/spring.factories文件配置替代@ComponentScan手动装配注解，从而实现自动装配

而Spring大多数自动装配都由starter模块负责

自定义自动装配

设计逻辑

HelloWorldAutoConfiguration

条件判断： user.name == "Mercy"
模式注解： @Configuration
@Enable 模块：流程图

引导类

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;

/**
 * @author F嘉阳
 * @date 2018-09-10 17:13
 */
@EnableAutoConfiguration
public class HelloWorldAutoConfigurationBootstrap {

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(HelloWorldAutoConfigurationBootstrap.class)
 .web(WebApplicationType.NONE)
 .run(args);

 // 通过bean名称获取helloWorld Bean
 String helloWorldBean = applicationContext.getBean("helloWorld", String.class);

 // helloWorld Bean 是否存在
 System.out.println("helloWorld Bean 内容: " + helloWorldBean);

 // 关闭上下文
 applicationContext.close();
 }
}

配置Bean，使用模式注解标注 @Configuration，该类实际上实现了条件装配、模式注解、自动装配

package top.fjy8018.springboot.configuration;

import org.springframework.context.annotation.Configuration;
import top.fjy8018.springboot.annotation.EnableHelloWorld;
import top.fjy8018.springboot.condition.ConditionalOnSystemProperty;

/**
 * @author F嘉阳
 * @date 2018-09-10 17:23
 */
@Configuration
@EnableHelloWorld
@ConditionalOnSystemProperty(name = "user.name", value = "11860")
public class HelloWorldAutoConfiguration {}

创建spring.factories，每行全类名顶格开始，防止在某些版本下产生不稳定的BUG

# 自动装配
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
top.fjy8018.springboot.configuration.HelloWorldAutoConfiguration

执行结果与之前一致

SpringBoot源码分析1——总览

Thu, 10 Dec 2020 17:11:00 +0800

SpringBoot 2.0 特点

组件自动装配

规约大于配置,专注核心业务

模式注解、@ Enable模块、条件装配、加载机制

外部化配置

一次构建、按需调配,到处运行

Environment抽象、生命周期、破坏性变更

嵌入式容器

内置容器、无需部署、独立运行

Servlet Web容器、 Reactive Web容器

Spring Boot Starter

简化依赖、按需装配、自我包含

依赖管理、装配条件、装配顺序

Production-Ready

一站式运维、生态无缝整合

健康检査、数据指标、@ Endpoint管控

SpringBoot 与Java EE规范

Web: Servlet( JSR-315, JSR-340）
SQL: JDBC(SR-221)
数据校验: Bean validation(JSR303、JSR-349)
缓存: Java Caching API(JSR-107)
Web Sockets: Java api for Web Socket(JSR-356
Web services: JAX-WS( JSR-224）
Java管理:JMX(JSR3)
消息:JMS(JSR-914)

核心特性介绍

三大特性

组件自动装配: Web mvc、 Web flux、JDBC等
嵌入式Web容器: Tomcat、 Jetty以及 Undertow
生产准备特性:指标、健康检査、外部化配置等

组件自动装配

激活： @EnableAutoConfiguration 配置： /META-INF/spring.factories 实质为key-value形式实现： XXXAutoConfiguration

官方文档样例

import org.springframework.boot.*;
import org.springframework.boot.autoconfigure.*;
import org.springframework.web.bind.annotation.*;

@RestController
@EnableAutoConfiguration
public class Example {

 @RequestMapping("/")
 String home() {
 return "Hello World!";
 }

 public static void main(String[] args) throws Exception {
 SpringApplication.run(Example.class, args);
 }

}

实际上@SpringBootApplication已经包含该注解

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@SpringBootConfiguration
@EnableAutoConfiguration
@ComponentScan(excludeFilters = {
 @Filter(type = FilterType.CUSTOM, classes = TypeExcludeFilter.class),
 @Filter(type = FilterType.CUSTOM, classes = AutoConfigurationExcludeFilter.class) })
public @interface SpringBootApplication {
}

在IDEA中使用ctrl+shift+n可查看Spring源码中包含大量spring.factories文件

样例文件：

org/springframework/boot/spring-boot-autoconfigure/2.0.4.RELEASE/spring-boot-autoconfigure-2.0.4.RELEASE.jar!/META-INF/spring.factories

# Initializers
org.springframework.context.ApplicationContextInitializer=\
org.springframework.boot.autoconfigure.SharedMetadataReaderFactoryContextInitializer,\
org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener

# Application Listeners
org.springframework.context.ApplicationListener=\
org.springframework.boot.autoconfigure.BackgroundPreinitializer

# Auto Configuration Import Listeners
org.springframework.boot.autoconfigure.AutoConfigurationImportListener=\
org.springframework.boot.autoconfigure.condition.ConditionEvaluationReportAutoConfigurationImportListener

# Auto Configuration Import Filters
org.springframework.boot.autoconfigure.AutoConfigurationImportFilter=\
org.springframework.boot.autoconfigure.condition.OnClassCondition

# 自动装配配置，自动装载Bean列表
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
org.springframework.boot.autoconfigure.admin.SpringApplicationAdminJmxAutoConfiguration,\
org.springframework.boot.autoconfigure.web.servlet.WebMvcAutoConfiguration,\
...

WebMvcAutoConfiguration源码，其包含大量条件装配，若未检测到该类的依赖，则不会被装配

@Configuration
@ConditionalOnWebApplication(type = Type.SERVLET)
@ConditionalOnClass({ Servlet.class, DispatcherServlet.class, WebMvcConfigurer.class })
@ConditionalOnMissingBean(WebMvcConfigurationSupport.class)
@AutoConfigureOrder(Ordered.HIGHEST_PRECEDENCE + 10)
@AutoConfigureAfter({ DispatcherServletAutoConfiguration.class,
 ValidationAutoConfiguration.class })
public class WebMvcAutoConfiguration {
...

嵌入式Web容器

Web servlet: Tomcat、 Jetty和 Undertow
Web reactive Netty Web Server

生产准备特性

指标:/ actuator/ metrics
健康检查:/ actuator/ health
外部化配置:/ actuator/ configprops

Web应用介绍

传统 Servlet应用概览

Servlet组件: Servlet、 Filter、 Listener
Servlet注册: Servlet注解、 Spring Bean、 RegistrationBean
异步非阻塞:异步 Servlet、非阻塞 Servlet

传统 Servelt 应用

加入springboot应用通用依赖前缀多为spring-boot-starter-xxx

加入Web依赖

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
</dependency>

Maven依赖变化

由此可看出默认的Web容器为Tomcat（内嵌Servlet容器）

Servlet 组件

Servlet
- 实现
  - @WebServlet
  - HttpServlet
  - 注册
- URL 映射
  - @WebServlet(urlPatterns = "/my/servlet")
- 注册 @ServletComponentScan(basePackages = "top.fjy8018.springboot.web.servlet")
Filter
Listener

Servlet 注册

Servlet 注解

@ServletComponentScan +
- @WebServlet
- @WebFilter
- @WebListener

Spring Bean

@Bean +
- Servlet
- Filter
- Listener
RegistrationBean
- ServletRegistrationBean
- FilterRegistrationBean
- ServletListenerRegistrationBean

Servlet 实现

参考源码包位置，由于出现了Reactive Web容器，SpringBoot 1.x的包路径与2.x有所不同，2.x包路径变更为org.springframework.boot.autoconfigure.web.servlet

HttpServlet

package top.fjy8018.springboot.web.servlet;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

/**
 * 结合servlet规范注册
 *
 * @author F嘉阳
 * @date 2018-09-08 19:17
 */
@WebServlet(urlPatterns = "/myservlet")
public class MyServlet extends HttpServlet {

 @Override
 protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
 resp.getWriter().print("hello world");
 }
}

Servlet 3.0规范要求必须实现HttpServlet
其内部提供多种方法实现（ctrl+F12）
urlPatterns = "/myservlet"指明浏览器映射

注册

在主类中注册（包扫描）

package top.fjy8018.springboot;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.servlet.ServletComponentScan;

@SpringBootApplication
@ServletComponentScan(basePackages = "top.fjy8018.springboot.web.servlet")
public class OverviewApplication {

 public static void main(String[] args) {
 SpringApplication.run(OverviewApplication.class, args);
 }
}

启动后从控制台可看出映射已经成功

异步非阻塞 Servlet

异步 Servlet

javax.servlet.ServletRequest#startAsync()
javax.servlet.AsyncContext

非阻塞 Servlet

javax.servlet.ServletInputStream#setReadListener
- javax.servlet.ReadListener
javax.servlet.ServletOutputStream#setWriteListener
- javax.servlet.WriteListener

异步非阻塞Servlet实现

package top.fjy8018.springboot.web.servlet;

import javax.servlet.AsyncContext;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

/**
 * 结合servlet规范注册
 * asyncSupported指定异步servlet，默认false
 *
 * @author F嘉阳
 * @date 2018-09-08 19:17
 */
@WebServlet(urlPatterns = "/myservlet", asyncSupported = true)
public class MyServlet extends HttpServlet {

 @Override
 protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
 AsyncContext asyncContext = req.startAsync();

 // 执行任务
 asyncContext.start(() -> {
 try {
 resp.getWriter().print("hello world");

 // 触发完成，必须步骤，否则客户端发生超时
 asyncContext.complete();
 } catch (IOException e) {
 e.printStackTrace();
 }
 });
 }
}

asyncSupported = true 启用异步支持，否则报500错误

Spring Web MVC 应用介绍

Web mvo视图:模板引擎、内容协商、异常处理等
Web mVc rest:资源服务、资源跨域、服务发现等

Web MVC 视图

ViewResolver
View

源码

package org.springframework.web.servlet;

import java.util.Locale;
import org.springframework.lang.Nullable;

public interface ViewResolver {
 @Nullable
 View resolveViewName(String viewName, Locale locale) throws Exception;

}

通过指定名称和路径加载视图

视图渲染View

package org.springframework.web.servlet;

public interface View {
 ...

 void render(@Nullable Map<String, ?> model, HttpServletRequest request, HttpServletResponse response)
 throws Exception;

}

提供接口供不同的模板引擎实现ViewResolver

模板引擎

Thymeleaf
Freemarker
JSP

若同时有多种模板引擎，则需要内容协商

内容协商

ContentNegotiationConfigurer
ContentNegotiationStrategy
ContentNegotiatingViewResolver

异常处理

@ExceptionHandler
HandlerExceptionResolver
- ExceptionHandlerExceptionResolver
BasicErrorController (Spring Boot默认白页404)

源码

@Controller
@RequestMapping("${server.error.path:${error.path:/error}}")
public class BasicErrorController extends AbstractErrorController {
 
 @RequestMapping(produces = "text/html")
 public ModelAndView errorHtml(HttpServletRequest request,
 HttpServletResponse response) {
 HttpStatus status = getStatus(request);
 Map<String, Object> model = Collections.unmodifiableMap(getErrorAttributes(
 request, isIncludeStackTrace(request, MediaType.TEXT_HTML)));
 response.setStatus(status.value());
 ModelAndView modelAndView = resolveErrorView(request, response, status, model);
 return (modelAndView != null) ? modelAndView : new ModelAndView("error", model);
 }
 
 @RequestMapping
 @ResponseBody
 public ResponseEntity<Map<String, Object>> error(HttpServletRequest request) {
 Map<String, Object> body = getErrorAttributes(request,
 isIncludeStackTrace(request, MediaType.ALL));
 HttpStatus status = getStatus(request);
 return new ResponseEntity<>(body, status);
 }
}

其根据浏览器类型提供两种方式渲染，分别为html和json

{
 "timestamp": "2019-03-13T04:47:18.445+0000",
 "status": 404,
 "error": "Not Found",
 "message": "Not Found",
 "path": "/hello"
}

Web MVC REST

资源服务

@RequestMapping
@GetMapping
@ResponseBody
@RequestBody

从源码分析可知@GetMapping是从@RequestMapping变化而来，通过委派（别名）方式实现，并且在SpringMVC 4.3后引入

package org.springframework.web.bind.annotation;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import org.springframework.core.annotation.AliasFor;

/**
 * Annotation for mapping HTTP {@code GET} requests onto specific handler
 * methods.
 *
 * <p>Specifically, {@code @GetMapping} is a <em>composed annotation</em> that
 * acts as a shortcut for {@code @RequestMapping(method = RequestMethod.GET)}.
 *
 *
 * @author Sam Brannen
 * @since 4.3
 * @see PostMapping
 * @see PutMapping
 * @see DeleteMapping
 * @see PatchMapping
 * @see RequestMapping
 */
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@RequestMapping(method = RequestMethod.GET)
public @interface GetMapping {
 /**
 * 委派目标 {@link RequestMapping#name}.
 */
 @AliasFor(annotation = RequestMapping.class)
 String name() default "";
 ...
}

资源跨域

CrossOrigin
WebMvcConfigurer#addCorsMappings
传统解决方案
- IFrame
- JSONP

源码

package org.springframework.web.servlet.config.annotation;

/**
 * @since 3.1
 */
public interface WebMvcConfigurer {
 /**
 * 跨域配置
 * @since 4.2
 */
 default void addCorsMappings(CorsRegistry registry) {
 }
}

注解实现方式

package org.springframework.web.bind.annotation;

/**
 * @since 4.2
 */
@Target({ ElementType.METHOD, ElementType.TYPE })
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface CrossOrigin {...}

两者均来自springframework，属于不同的实现维度（接口编程 / 注解驱动）

服务发现

HATEOS

核心组件

DispatcherServlet——Servlet实现请求转发到映射URL
HandlerMapping——处理映射URL方法
HandlerAdapter——方法处理
ViewResolver——交由视图处理
…

Spring WebFlux （Spring 5+）

Reactor 基础

Java Lambda —— Reactive Stream实现（JDK 9 flow api、RxJava）
Mono
Flux

Web Flux 核心

Web MVC 注解兼容

@Controller
@RequestMapping
@ResponseBody
@RequestBody
…

函数式声明

RouterFunction

异步非阻塞

Servlet 3.1 +
Netty Reactor

使用场景

页面渲染
REST 应用

性能测试

国外博客

结论：Webflux 性能不会显著提升，但Webflux提升吞吐量

Web Server 应用

切换 Web server
自定义 Servlet Web server
自定义 Reactive Web server

切换 Web Server

切换其他 Servlet 容器

Tomcat -> Jetty

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 <exclusions>
 <!-- tomcat加载优先级高于jetty，若不加排除，则仍然加载tomcat -->
 <exclusion>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-tomcat</artifactId>
 </exclusion>
 </exclusions>
</dependency>
<!-- 使用 Jetty 容器 -->
<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-jetty</artifactId>
</dependency>

替换 Servlet 容器

WebFlux

<!--使用webflux必须注释web依赖，当web和flux共存时，web优先级更高-->
<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

注释后相关类和的注解也已经失效，需要删除

@SpringBootApplication
//@ServletComponentScan(basePackages = "top.fjy8018.springboot.web.servlet")
public class SpringbootApplication {

 public static void main(String[] args) {
 SpringApplication.run(SpringbootApplication.class, args);
 }
}

依赖变化

启动后变为Netty容器

自定义 Servlet Web Server

WebServerFactoryCustomizer

SpringBoot 2.0 新增支持接口

package org.springframework.boot.web.server;

/**
 * @since 2.0.0
 * @see WebServerFactoryCustomizerBeanPostProcessor
 */
@FunctionalInterface
public interface WebServerFactoryCustomizer<T extends WebServerFactory> {
 
 void customize(T factory);
}

该接口为主接口，其下有大量实现，IDEA通过ctrl+alt+B或者ctrl+alt+左键点击查看

其中TomcatServletWebServerFactoryCustomizer就是实现类之一，但在启用Webflux后该类无效

package org.springframework.boot.autoconfigure.web.servlet;

/**
 * {@link WebServerFactoryCustomizer} to apply {@link ServerProperties} to Tomcat web
 * servers.
 *
 * @author Brian Clozel
 * @author Phillip Webb
 * @since 2.0.0
 */
public class TomcatServletWebServerFactoryCustomizer
 implements WebServerFactoryCustomizer<TomcatServletWebServerFactory>, Ordered {
 @Override
 public void customize(TomcatServletWebServerFactory factory) {
 ServerProperties.Tomcat tomcatProperties = this.serverProperties.getTomcat();
 if (!ObjectUtils.isEmpty(tomcatProperties.getAdditionalTldSkipPatterns())) {
 factory.getTldSkipPatterns()
 .addAll(tomcatProperties.getAdditionalTldSkipPatterns());
 }
 if (tomcatProperties.getRedirectContextRoot() != null) {
 customizeRedirectContextRoot(factory,
 tomcatProperties.getRedirectContextRoot());
 }
 if (tomcatProperties.getUseRelativeRedirects() != null) {
 customizeUseRelativeRedirects(factory,
 tomcatProperties.getUseRelativeRedirects());
 }
 }
 ...
}

自定义 Reactive Web Server

ReactiveWebServerFactoryCustomizer

其源码显示提供了最基础的实现，包括端口、SSL等，允许自行扩展实现

package org.springframework.boot.autoconfigure.web.reactive;

/**
 * {@link WebServerFactoryCustomizer} to apply {@link ServerProperties} to reactive
 * servers.
 * @since 2.0.0
 */
public class ReactiveWebServerFactoryCustomizer implements
 WebServerFactoryCustomizer<ConfigurableReactiveWebServerFactory>, Ordered {

 private final ServerProperties serverProperties;

 public ReactiveWebServerFactoryCustomizer(ServerProperties serverProperties) {
 this.serverProperties = serverProperties;
 }

 @Override
 public int getOrder() {
 return 0;
 }

 @Override
 public void customize(ConfigurableReactiveWebServerFactory factory) {
 PropertyMapper map = PropertyMapper.get().alwaysApplyingWhenNonNull();
 map.from(this.serverProperties::getPort).to(factory::setPort);
 map.from(this.serverProperties::getAddress).to(factory::setAddress);
 map.from(this.serverProperties::getSsl).to(factory::setSsl);
 map.from(this.serverProperties::getCompression).to(factory::setCompression);
 map.from(this.serverProperties::getHttp2).to(factory::setHttp2);
 }
}

数据相关介绍

关系型数据

JDBC:数据源、 JdbcTemplate、自动装配
JPA:实体映射关系、实体操作、自动装配
事务∶ Spring事务抽象、JDBC事务处理、自动装配

JDBC

依赖

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>

依赖变化，在SpringBoot2.0 后依赖新的连接池 HikariCP

数据源

javax.sql.DataSource
JdbcTemplate
自动装配
DataSourceAutoConfiguration

DataSource

连接池多数实现是对DataSource的二次封装，提高执行效率

package javax.sql;

/**
 * @since 1.4
 */

public interface DataSource extends CommonDataSource, Wrapper {
 Connection getConnection() throws SQLException;
 
 Connection getConnection(String username, String password)
 throws SQLException;
}

自动装配

JdbcTemplate自动装配，其依赖数据源配置项，故有@AutoConfigureAfter注解标识，该类执行于DataSourceAutoConfiguration之后，额外还有多个条件装配的标识

package org.springframework.boot.autoconfigure.jdbc;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for {@link JdbcTemplate} and
 * {@link NamedParameterJdbcTemplate}.
 * @since 1.4.0
 */
@Configuration
@ConditionalOnClass({ DataSource.class, JdbcTemplate.class })
@ConditionalOnSingleCandidate(DataSource.class)
@AutoConfigureAfter(DataSourceAutoConfiguration.class)
@EnableConfigurationProperties(JdbcProperties.class)
public class JdbcTemplateAutoConfiguration {...}

DataSourceAutoConfiguration

package org.springframework.boot.autoconfigure.jdbc;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for {@link DataSource}.
 */
@Configuration
@ConditionalOnClass({ DataSource.class, EmbeddedDatabaseType.class })
@EnableConfigurationProperties(DataSourceProperties.class)
@Import({ DataSourcePoolMetadataProvidersConfiguration.class,
 DataSourceInitializationConfiguration.class })
public class DataSourceAutoConfiguration {...}

该类属于自动装配，故必然在spring.factories中，而数据源下即JdbcTemplateAutoConfiguration

此时直接启动会得到缺少数据源的错误报告

***************************
APPLICATION FAILED TO START
***************************

Description:

Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.

Reason: Failed to determine a suitable driver class


Action:

Consider the following:
 If you want an embedded database (H2, HSQL or Derby), please put it on the classpath.
 If you have database settings to be loaded from a particular profile you may need to activate it (no profiles are currently active).

JPA

依赖

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>

其属于SpringData项目，基于hibernate实现，同时也自动依赖JDBC

实体映射关系

@javax.persistence.OneToOne
@javax.persistence.OneToMany
@javax.persistence.ManyToOne
@javax.persistence.ManyToMany
…

实体操作

javax.persistence.EntityManager

属于Java类库的API

package javax.persistence;

public interface EntityManager {

 /**
 * 存
 */
 public void persist(Object entity);

 /**
 * 更新
 */
 public <T> T merge(T entity);

 /**
 * 删除
 */
 public void remove(Object entity);

 /**
 * 查询以及多个重载方法
 */
 public <T> T find(Class<T> entityClass, Object primaryKey);

 public <T> T find(Class<T> entityClass, Object primaryKey,
 Map<String, Object> properties);
 ...
}

自动装配

HibernateJpaAutoConfiguration

该类也同属于自动装配，且需要JDBC提供数据源，故其加载顺序标识为DataSourceAutoConfiguration之后

package org.springframework.boot.autoconfigure.orm.jpa;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for Hibernate JPA.
 */
@Configuration
@ConditionalOnClass({ LocalContainerEntityManagerFactoryBean.class, EntityManager.class })
@Conditional(HibernateEntityManagerCondition.class)
@EnableConfigurationProperties(JpaProperties.class)
@AutoConfigureAfter({ DataSourceAutoConfiguration.class })
@Import(HibernateJpaConfiguration.class)
public class HibernateJpaAutoConfiguration {...}

事务（Transaction）

依赖

<dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-tx</artifactId>
</dependency>

Spring 事务抽象

PlatformTransactionManager

该接口属于事务根接口，不仅限于数据库事务，默认实现包含分布式事务JtaTransactionManager和数据库事务DataSourceTransactionManager

package org.springframework.transaction;

import org.springframework.lang.Nullable;

/**
 * <p>The default implementations of this strategy interface are
 * {@link org.springframework.transaction.jta.JtaTransactionManager} and
 * {@link org.springframework.jdbc.datasource.DataSourceTransactionManager}

 * @see org.springframework.transaction.support.TransactionTemplate
 * @see org.springframework.transaction.interceptor.TransactionInterceptor
 * @see org.springframework.transaction.interceptor.TransactionProxyFactoryBean
 */
public interface PlatformTransactionManager {...}

JDBC 事务处理

DataSourceTransactionManager

内部通过IOC自动注入数据源接口

package org.springframework.jdbc.datasource;

@SuppressWarnings("serial")
public class DataSourceTransactionManager extends AbstractPlatformTransactionManager
 implements ResourceTransactionManager, InitializingBean {

 @Nullable
 private DataSource dataSource;
 ...
}

自动装配

TransactionAutoConfiguration

自动装配的条件是Spring事务依赖存在PlatformTransactionManager

package org.springframework.boot.autoconfigure.transaction;

/**
 * {@link org.springframework.boot.autoconfigure.EnableAutoConfiguration
 * Auto-configuration} for Spring transaction.
 * @since 1.3.0
 */
@Configuration
@ConditionalOnClass(PlatformTransactionManager.class)
@AutoConfigureAfter({ JtaAutoConfiguration.class, HibernateJpaAutoConfiguration.class,
 DataSourceTransactionManagerAutoConfiguration.class,
 Neo4jDataAutoConfiguration.class })
@EnableConfigurationProperties(TransactionProperties.class)
public class TransactionAutoConfiguration {...}

功能扩展介绍

Spring Boot 应用

Spring Application:失败分析、应用特性、事件监听等
Spring Boot配置:外部化配置、 Profile、配置属性
Spring Boot starter: Starter开发、最佳实践

SpringApplication

失败分析

FailureAnalysisReporter

该类在Spring Boot 1.4后开始支持

package org.springframework.boot.diagnostics;

/**
 * Reports a {@code FailureAnalysis} to the user.
 * @since 1.4.0
 */
@FunctionalInterface
public interface FailureAnalysisReporter {...}

应用特性

SpringApplication Fluent API

原始默认启动方法为SpringApplication.run(SpringbootApplication.class, args)其实际返回可配置上下文对象ConfigurableApplicationContext，可通过Builder模式重写

package top.fjy8018.springboot;

@SpringBootApplication
public class SpringbootApplication {

 public static void main(String[] args) {
 // Fluent API
 // 等价建造者设计模式调用方法
 new SpringApplicationBuilder(SpringbootApplication.class)
 // 不使用web模式
 .web(WebApplicationType.NONE)
 // 自定义参数
 .properties("name=fjy")
 .run(args);
 }
}

2.0 版本下提供3种WEB容器类型

package org.springframework.boot;

/**
 * @since 2.0.0
 */
public enum WebApplicationType {
 NONE,
 SERVLET,
 REACTIVE
}

Spring Boot 配置

外部化配置

ConfigurationProperty
@Profile

从SpringBoot 2.0 后引入，Origin类型支持其他配资源引入，非传统仅支持键值对，由此支持多数据源

package org.springframework.boot.context.properties.source;

/**
 * @since 2.0.0
 */
public final class ConfigurationProperty
 implements OriginProvider, Comparable<ConfigurationProperty> {
 
 private final ConfigurationPropertyName name;

 private final Object value;

 private final Origin origin;
 ...
}

官方文档

此处官方文档存在一定疏漏，部分场景配置无法实现

Spring Boot uses a very particular PropertySource order that is designed to allow sensible overriding of values. Properties are considered in the following order:

Devtools global settings properties on your home directory (~/.spring-boot-devtools.properties when devtools is active).

@TestPropertySource annotations on your tests.

@SpringBootTest#properties annotation attribute on your tests.

Command line arguments.

Properties from SPRING_APPLICATION_JSON (inline JSON embedded in an environment variable or system property).

ServletConfig init parameters.

ServletContext init parameters.

JNDI attributes from java:comp/env.

Java System properties (System.getProperties()).

OS environment variables.

A RandomValuePropertySource that has properties only in random.*.

Profile-specific application properties outside of your packaged jar (application-{profile}.properties and YAML variants).

Profile-specific application properties packaged inside your jar (application-{profile}.properties and YAML variants).

Application properties outside of your packaged jar (application.properties and YAML variants).

Application properties packaged inside your jar (application.properties and YAML variants).

@PropertySource annotations on your @Configuration classes.

Default properties (specified by setting SpringApplication.setDefaultProperties).

Profile支持以注解方式条件开启配置，用于区分开发环境和生产环境等

package org.springframework.context.annotation;

/**
 * @since 3.1
 */
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(ProfileCondition.class)
public @interface Profile {
 
 String[] value();
}

配置属性

PropertySources

提供两种编程方式：接口、注解，支持保存多个配置，按照17种顺序排列，该接口在SpringBoot中重新设计变为ConfigurationProperty

package org.springframework.core.env;

/**
 * @since 3.1
 * @see PropertySource
 */
public interface PropertySources extends Iterable<PropertySource<?>> {...}

运维管理介绍

官方文档

Spring Boot Actuator

端点（Endpoints）

Web Endpoints
JMX Endpoints

依赖

<!--运维监控-->
<dependency>
 <groupId>org.springframework.boot</groupId> 
 <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

启动效果

访问健康检查接口/actuator/health返回应用状态

{
 "status": "UP"
}

访问根路径

{
 "_links": {
 "self": {
 "href": "http://localhost:8080/actuator",
 "templated": false
 },
 "health": {
 "href": "http://localhost:8080/actuator/health",
 "templated": false
 },
 "info": {
 "href": "http://localhost:8080/actuator/info",
 "templated": false
 }
 }
}

依据SpringBoot打印的进程号或者依据应用名称使用Jconsole监控，该操作使用的为JMX端点

健康检查（Health Checks）

Health
HealthIndicator

使用JMX在默认情况下能获取比Web端点更多的信息，原因是JMX支持使用安全连接，具有更好的安全性

官方文档指出默认状态只暴露health和info，可通过配置文件调整

28.4 Actuator Security
For security purposes, all actuators other than /health and /info are disabled by default. The management.endpoints.web.exposure.include property can be used to enable the actuators.

在application.properties文件中加入配置重启应用

# 暴露所有健康监控点
management.endpoints.web.exposure.include=*

从日志输出可知增加了更多的URL映射

但尽管配置后，JMX显示的信息仍然比Web端更丰富，例如在JMX中不仅看到状态，还能看到磁盘、内存等信息，而Web端仅看到状态

指标（Metrics）

内建 Metrics
Web Endpoint : /actuator/metrics
自定义 Metrics

Web端点列举支持指标

{
 "names": [
 "jvm.memory.max",
 "jvm.gc.memory.promoted",
 "jvm.memory.used",
 "jvm.gc.max.data.size",
 "jvm.gc.pause",
 "jvm.memory.committed",
 "system.cpu.count",
 "logback.events",
 "jvm.buffer.memory.used",
 "jvm.threads.daemon",
 "system.cpu.usage",
 "jvm.gc.memory.allocated",
 "jvm.threads.live",
 "jvm.threads.peak",
 "process.uptime",
 "http.server.requests",
 "process.cpu.usage",
 "jvm.classes.loaded",
 "jvm.classes.unloaded",
 "jvm.gc.live.data.size",
 "jvm.buffer.count",
 "jvm.buffer.total.capacity",
 "process.start.time"
 ]
}

访问/actuator/metrics/jvm.memory.max可查看相关指标，转换后可知最大内存为5.2G

{
 "name": "jvm.memory.max",
 "description": "The maximum amount of memory in bytes that can be used for memory management",
 "baseUnit": "bytes",
 "measurements": [
 {
 "statistic": "VALUE",
 "value": 5595725823
 }
 ],
 "availableTags": [
 {
 "tag": "area",
 "values": [
 "heap",
 "nonheap"
 ]
 },
 {
 "tag": "id",
 "values": [
 "Compressed Class Space",
 "PS Survivor Space",
 "PS Old Gen",
 "Metaspace",
 "PS Eden Space",
 "Code Cache"
 ]
 }
 ]
}

设计模式学习——行为型

Wed, 22 Jul 2020 16:50:17 +0800

模板方法模式

介绍

定义:定义了一个算法的骨架,并允许子类为一个或多个步骤提供实现

模板方法使得子类可以在不改变算法结构的情况下,重新定义算法的某些步骤

类型：行为型

适用场景

一次性实现一个算法的不变的部分,并将可变的行为留给子类来实现
各子类中公共的行为被提取出来并集中到一个公共父类中, 从而避免代码重复

优点

提高复用性
提高扩展性
符合开闭原则

缺点

类数目增加
增加了系统实现的复杂度
继承关系自身缺点,如果父类添加新的抽象方法,所有子类都要改一遍

分类和扩展

钩子方法
- 提供了默认的行为，子类可以在必要时进行扩展

相关设计模式

模板方法模式和工厂方法模式
- 工厂方法是模板方法的一种特殊实现
模板方法模式和策略模式
- 策略模式在于实现算法可以互相替换而不影响应用层，不改变算法流程
- 模板方法定义了一个算法的骨架，不明确的交给子类实现，可以改变算法流程

代码样例

类图

抽象课程

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;
import org.apache.ibatis.mapping.MappedStatement;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.Collection;

/**
 * 抽象课程
 * <p>
 * {@link java.util.AbstractList} 提供抽象List实现
 * 例如 {@link java.util.AbstractList#addAll(int, Collection)} 定义通过模板
 * {@link java.util.AbstractMap} 同理
 * <p>
 * {@link javax.servlet.http.HttpServlet#doGet(HttpServletRequest, HttpServletResponse)} 也是模板定义
 * {@link org.apache.ibatis.executor.BaseExecutor#doUpdate(MappedStatement, Object)} 等方法交由子类实现
 * 例如 {@link org.apache.ibatis.executor.SimpleExecutor#doUpdate(MappedStatement, Object)}
 *
 * @author F嘉阳
 * @date 2020/3/4 10:59
 */
@Slf4j
public abstract class ACourse {

 /**
 * 固定不变的行为
 * 定义模板，不允许子类覆盖，防止影响流程执行顺序
 */
 protected final void makeCourse() {
 this.makePPT();
 this.makeVideo();
 // 调用可选方法
 if (needWriteArticle()) {
 this.writeArticle();
 }
 this.packageCourse();
 }

 /**
 * 固定不变的行为
 * 满足所有子类需求
 */
 final void makePPT() {
 log.info("制作PPT");
 }

 final void makeVideo() {
 log.info("制作视频");
 }

 /**
 * 可选行为
 * <p>
 * 手记可写可不写，但写的手记的流程还是一致的，故也声明为final
 * 通过定义钩子方法，允许子类覆盖
 */
 final void writeArticle() {
 log.info("编写手记");
 }

 /**
 * 钩子方法，提供默认实现
 *
 * @return
 */
 protected boolean needWriteArticle() {
 return false;
 }

 /**
 * 必须实现的行为
 * 课程内容打包
 */
 abstract void packageCourse();
}

具体课程实现

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;

/**
 * 具体课程实现
 *
 * @author F嘉阳
 * @date 2020/3/4 11:05
 */
@Slf4j
public class JavaCourse extends ACourse {
 /**
 * 必须实现的行为
 * 课程内容打包
 */
 @Override
 void packageCourse() {
 log.info("提供Java学习课程源码");
 }
}

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;

/**
 * 具体课程实现
 * @author F嘉阳
 * @date 2020/3/4 11:06
 */
@Slf4j
public class VueCourse extends ACourse {

 private boolean needWriteArticleFlag = false;

 /**
 * 通过构造器开放给实例决定
 * @param needWriteArticleFlag
 */
 public VueCourse(boolean needWriteArticleFlag) {
 this.needWriteArticleFlag = needWriteArticleFlag;
 }

 /**
 * 必须实现的行为
 * 课程内容打包
 */
 @Override
 void packageCourse() {
 log.info("提供Vue学习课程源码");
 log.info("提供Vue学习课程多媒体素材");
 }

 /**
 * 钩子方法，提供默认实现
 *
 * @return
 */
 @Override
 protected boolean needWriteArticle() {
 return this.needWriteArticleFlag;
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/4 11:08
 */
@Slf4j
class ACourseTest {

 @Test
 void makeJavaCourse() {
 ACourse javaCourse = new JavaCourse();
 javaCourse.makeCourse();
 }

 @Test
 void makeVueCourse() {
 ACourse vueCourse = new VueCourse(true);
 vueCourse.makeCourse();
 }
}

输出

11:27:56.942 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作PPT
11:27:56.957 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作视频
11:27:56.957 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.JavaCourse - 提供Java学习课程源码


11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作PPT
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作视频
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 编写手记
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.VueCourse - 提供Vue学习课程源码
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.VueCourse - 提供Vue学习课程多媒体素材

迭代器模式

介绍

定义提供一种方法,顺序访问一个集合对象中的各个元素,而又不暴露该对象的内部表示

类型：行为型

适用场景

访问一个集合对象的内容而无需暴露它的内部表示
为遍历不同的集合结构提供一个统一的接口

优点

分离了集合对象的遍历行为

缺点

类的个数成对增加

相关设计模式

迭代器模式和访问者模式
- 访问者模式扩展开放的部分，再作用于对象的操作上
- 迭代器模式扩展开放的部分，再作用于对象的种类上

代码样例

迭代器一般使用现成的，而不会自己实现

类图

课程接口

package top.fjy8018.designpattern.pattern.behavior.iterator;

import lombok.AllArgsConstructor;
import lombok.Getter;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:49
 */
@AllArgsConstructor
@Getter
public class Course {

 private String name;

}

迭代器

package top.fjy8018.designpattern.pattern.behavior.iterator;

import java.util.ArrayList;

/**
 * JDK迭代器
 * {@link java.util.Iterator}
 * 集合内部类实现 {@link ArrayList.Itr#hasNext()}
 * 同时做了进一步扩展 {@link ArrayList.ListItr}
 *
 * mybatis中 {@link org.apache.ibatis.cursor.defaults.DefaultCursor}
 * 其持有游标迭代器对象 {@link org.apache.ibatis.cursor.defaults.DefaultCursor.CursorIterator}
 *
 * @author F嘉阳
 * @date 2020/3/4 14:49
 */
public interface CourseIterator {

 /**
 * 获取下一个课程
 *
 * @return
 */
 Course nextCourse();

 /**
 * 是否是最后一个课程
 * @return
 */
 boolean isLastCourse();
}

迭代器实现

package top.fjy8018.designpattern.pattern.behavior.iterator;

import lombok.extern.slf4j.Slf4j;

import java.util.List;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:50
 */
@Slf4j
public class CourseIteratorImpl implements CourseIterator {

 private List courseList;

 private int position;

 private Course course;

 public CourseIteratorImpl(List courseList){
 this.courseList=courseList;
 }

 @Override
 public Course nextCourse() {
 log.info("返回课程,位置是: {}",position);
 course=(Course)courseList.get(position);
 position++;
 return course;
 }

 @Override
 public boolean isLastCourse(){
 if(position< courseList.size()){
 return false;
 }
 return true;
 }
}

对课程进行处理

package top.fjy8018.designpattern.pattern.behavior.iterator;

/**
 * 对课程进行处理
 *
 * @author F嘉阳
 * @date 2020/3/4 14:48
 */
public interface CourseAggregate {

 void addCourse(Course course);

 void removeCourse(Course course);

 /**
 * 获取课程迭代器
 * @return
 */
 CourseIterator getCourseIterator();
}

实现

package top.fjy8018.designpattern.pattern.behavior.iterator;

import java.util.ArrayList;
import java.util.List;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:50
 */
public class CourseAggregateImpl implements CourseAggregate {

 private List courseList;

 public CourseAggregateImpl() {
 this.courseList = new ArrayList();
 }

 @Override
 public void addCourse(Course course) {
 courseList.add(course);
 }

 @Override
 public void removeCourse(Course course) {
 courseList.remove(course);
 }

 @Override
 public CourseIterator getCourseIterator() {
 return new CourseIteratorImpl(courseList);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.iterator;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:52
 */
@Slf4j
class CourseIteratorTest {

 @Test
 void nextCourse() {
 Course course1 = new Course("Java电商一期");
 Course course2 = new Course("Java电商二期");
 Course course3 = new Course("Java设计模式精讲");
 Course course4 = new Course("Python课程");
 Course course5 = new Course("算法课程");
 Course course6 = new Course("前端课程");


 CourseAggregate courseAggregate = new CourseAggregateImpl();

 courseAggregate.addCourse(course1);
 courseAggregate.addCourse(course2);
 courseAggregate.addCourse(course3);
 courseAggregate.addCourse(course4);
 courseAggregate.addCourse(course5);
 courseAggregate.addCourse(course6);

 log.info("-----课程列表-----");
 printCourses(courseAggregate);

 courseAggregate.removeCourse(course4);
 courseAggregate.removeCourse(course5);

 log.info("-----删除操作之后的课程列表-----");
 printCourses(courseAggregate);
 }

 public static void printCourses(CourseAggregate courseAggregate){
 CourseIterator courseIterator= courseAggregate.getCourseIterator();
 while(!courseIterator.isLastCourse()){
 Course course=courseIterator.nextCourse();
 log.info(course.getName());
 }
 }
}

输出

15:18:23.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - -----课程列表-----
15:18:23.124 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 0
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java分布式
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 1
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java系统设计
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 2
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java设计模式精讲
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 3
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Python课程
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 4
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 算法课程
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 5
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 前端课程
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - -----删除操作之后的课程列表-----
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 0
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java分布式
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 1
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java系统设计
15:18:23.127 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 2
15:18:23.127 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java设计模式精讲
15:18:23.132 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 3
15:18:23.132 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 前端课程

策略模式

介绍

定义：定义了算法家族,分别封装起来,让它们之间可以互相替换, 此模式让算法的变化不会影响到使用算法的用户。

例如有大量if…else的情况

类型：行为型

适用场景

系统有很多类，而他们的区别仅仅在于他们的行为不同
一个系统需要动态地在几种算法中选择一种

优点

开闭原则
避免使用多重条件转移语句
提高算法的保密性和安全性

缺点

客户端必须知道所有的策略类,并自行决定使用哪一个策略类。
产生很多策略类

相关设计模式

策略模式和工厂模式
策略模式和状态模式

代码样例

改造前——初级策略模式

无法避免if…else

类图

促销接口

package top.fjy8018.designpattern.pattern.behavior.strategy;

/**
 * 促销接口
 *
 * @author F嘉阳
 * @date 2020/3/4 15:37
 */
public interface PromotionStrategy {

 void doPromotion();
}

满减促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 满减促销策略
 *
 * @author F嘉阳
 * @date 2020/3/4 15:38
 */
@Slf4j
public class ManJianPromotionStrategy implements PromotionStrategy {
 @Override
 public void doPromotion() {
 log.info("满减促销，满200-20");
 }
}

立减促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 立减促销策略
 *
 * @author F嘉阳
 * @date 2020/3/4 15:39
 */
@Slf4j
public class LiJianPromotionStrategy implements PromotionStrategy{
 @Override
 public void doPromotion() {
 log.info("立减促销，课程价格直接减去配置的价格");
 }
}

返现促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 返现促销策略
 *
 * @author F嘉阳
 * @date 2020/3/4 15:38
 */
@Slf4j
public class FanXianPromotionStrategy implements PromotionStrategy{
 @Override
 public void doPromotion() {
 log.info("返现促销，返回金额存放到用户余额中");
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/4 15:46
 */
@Slf4j
class PromotionStrategyTest {

 @Test
 void existIfElse() {
 String promotionKey = "LIJIAN";
 PromotionActivity activity = null;
 if (StringUtils.equals(promotionKey,"LIJIAN")){
 // 方法和类本身线程安全
 activity= new PromotionActivity(new LiJianPromotionStrategy());
 }else if(StringUtils.equals(promotionKey,"MANJIAN")){
 activity = new PromotionActivity(new FanXianPromotionStrategy());
 }

 activity.executePromotionStrategy();
 }
}

输出

16:08:28.582 [main] INFO top.fjy8018.designpattern.pattern.behavior.strategy.LiJianPromotionStrategy - 立减促销，课程价格直接减去配置的价格

改造后——结合工厂和策略模式

类图

空促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 空策略
 *
 * @author F嘉阳
 * @date 2020/3/4 16:16
 */
@Slf4j
public class EmptyPromotionStrategy implements PromotionStrategy {
 @Override
 public void doPromotion() {
 log.info("空促销策略");
 }
}

促销策略工厂

package top.fjy8018.designpattern.pattern.behavior.strategy;

import org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory;
import org.springframework.beans.factory.support.CglibSubclassingInstantiationStrategy;
import org.springframework.beans.factory.support.SimpleInstantiationStrategy;
import org.springframework.core.io.FileSystemResource;
import org.springframework.core.io.PathResource;

import java.util.Arrays;
import java.util.Comparator;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

/**
 * 促销策略工厂
 * <p>
 * {@link java.util.Comparator} 比较器，类似于抽象策略，有大量实现类，所有具体策略都要实现定义好的接口方法
 * 例如 {@link com.google.common.collect.ExplicitOrdering}
 * 应用层可以逐级传递比较器 {@link Arrays#sort(Object[], Comparator)}
 * {@link java.util.TreeMap#compare(Object, Object)} 利用内部的比较器处理
 * <p>
 * Spring资源策略抽象 {@link org.springframework.core.io.Resource}
 * 提供大量策略实现 {@link PathResource} {@link FileSystemResource}
 * <p>
 * {@link org.springframework.beans.factory.support.InstantiationStrategy} 初始化策略接口
 * 初始化具体策略实现 {@link SimpleInstantiationStrategy} {@link CglibSubclassingInstantiationStrategy}
 * 初始化策略在 {@link AbstractAutowireCapableBeanFactory} 中被使用
 *
 * @author F嘉阳
 * @date 2020/3/4 16:09
 */
public class PromotionStrategyFactory {

 private static Map<String, PromotionStrategy> STRATEGY_MAP = new ConcurrentHashMap<>();

 /**
 * 无促销
 */
 private static final PromotionStrategy NON_PROMOTION = new EmptyPromotionStrategy();

 static {
 STRATEGY_MAP.put(PromotionKey.FANXIAN, new FanXianPromotionStrategy());
 STRATEGY_MAP.put(PromotionKey.MANJIAN, new ManJianPromotionStrategy());
 STRATEGY_MAP.put(PromotionKey.LIJIAN, new LiJianPromotionStrategy());
 }

 /**
 * 禁止实例化
 */
 private PromotionStrategyFactory() {
 }

 /**
 * 开放获取策略的方法
 *
 * @param promotionKey
 * @return
 */
 public static PromotionStrategy getPromotionStrategy(String promotionKey) {
 PromotionStrategy strategy = STRATEGY_MAP.get(promotionKey);
 return strategy == null ? NON_PROMOTION : strategy;
 }

 /**
 * 常量定义
 */
 private interface PromotionKey {
 String LIJIAN = "LIJIAN";
 String FANXIAN = "FANXIAN";
 String MANJIAN = "MANJIAN";
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/3/4 15:46
 */
@Slf4j
class PromotionStrategyTest {

 @Test
 void nonIfElse() {
 String promotionKey = "LIJIAN";
 PromotionActivity activity = new PromotionActivity(PromotionStrategyFactory.getPromotionStrategy(promotionKey));
 activity.executePromotionStrategy();
 }
}

输出

16:08:28.582 [main] INFO top.fjy8018.designpattern.pattern.behavior.strategy.LiJianPromotionStrategy - 立减促销，课程价格直接减去配置的价格

解释器模式

介绍

定义：给定一个语言，定义它的文法的一种表示，并定义一个解释器，这个解释器使用该表示来解释语言中的句子。
为了解释一种语言，而为语言创建的解释器
类型：行为型

适用场景（较少）

某个特定类型问题发生频率足够高

优点

语法由很多类表示,容易改变及扩展此语言

缺点

当语法规则数目太多时,增加了系统复杂度

相关设计模式

解释器模式和适配器模式

代码样例

类图

解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * 解释器
 * <p>
 * JDK 源码
 * {@link java.util.regex.Pattern} 正则解释器
 * {@link org.springframework.expression.spel.standard.SpelExpressionParser} Spring解释器
 *
 * @author F嘉阳
 * @date 2020/3/5 14:45
 */
public interface Interpreter {
 int interpret();
}

加法解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * 加法解释器
 *
 * @author F嘉阳
 * @date 2020/3/5 14:46
 */
public class AddInterpreter implements Interpreter {

 /**
 * 定义两个表达式
 */
 private Interpreter firstExpression, secondExpression;

 public AddInterpreter(Interpreter firstExpression, Interpreter secondExpression) {
 this.firstExpression = firstExpression;
 this.secondExpression = secondExpression;
 }

 @Override
 public int interpret() {
 // 此时传入为NumberInterpreter类型，调interpret方法转int类型
 return this.firstExpression.interpret() + this.secondExpression.interpret();
 }

 @Override
 public String toString() {
 return "+";
 }
}

乘法解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * 乘法解释器
 *
 * @author F嘉阳
 * @date 2020/3/5 14:47
 */
public class MultiInterpreter implements Interpreter {

 private Interpreter firstExpression, secondExpression;

 public MultiInterpreter(Interpreter firstExpression, Interpreter secondExpression) {
 this.firstExpression = firstExpression;
 this.secondExpression = secondExpression;
 }

 @Override
 public int interpret() {
 return this.firstExpression.interpret() * this.secondExpression.interpret();
 }

 @Override
 public String toString() {
 return "*";
 }
}

类型解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:47
 */
public class NumberInterpreter implements Interpreter {
 private int number;

 public NumberInterpreter(int number) {
 this.number = number;
 }

 public NumberInterpreter(String number) {
 this.number = Integer.parseInt(number);
 }

 @Override
 public int interpret() {
 return this.number;
 }
}

操作工具类

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:47
 */
public class OperatorUtil {
 public static boolean isOperator(String symbol) {
 return (symbol.equals("+") || symbol.equals("*"));

 }

 public static Interpreter getExpressionObject(Interpreter firstExpression, Interpreter secondExpression, String symbol) {
 if (symbol.equals("+")) {
 return new AddInterpreter(firstExpression, secondExpression);
 } else if (symbol.equals("*")) {
 return new MultiInterpreter(firstExpression, secondExpression);
 }
 return null;
 }
}

自定义解释器实现

package top.fjy8018.designpattern.pattern.behavior.interpreter;

import lombok.extern.slf4j.Slf4j;

import java.util.Stack;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:48
 */
@Slf4j
public class CustomExpressionParser {
 private Stack<Interpreter> stack = new Stack<Interpreter>();

 public int parse(String str) {
 String[] strItemArray = str.split(" ");
 for (String symbol : strItemArray) {
 if (!OperatorUtil.isOperator(symbol)) {
 Interpreter numberExpression = new NumberInterpreter(symbol);
 stack.push(numberExpression);
 log.info(String.format("入栈: %d", numberExpression.interpret()));
 } else {
 //是运算符号，可以计算
 Interpreter firstExpression = stack.pop();
 Interpreter secondExpression = stack.pop();
 log.info(String.format("出栈: %d 和 %d",
 firstExpression.interpret(), secondExpression.interpret()));
 Interpreter operator = OperatorUtil.getExpressionObject(firstExpression, secondExpression, symbol);
 log.info(String.format("应用运算符: %s", operator));
 int result = operator.interpret();
 NumberInterpreter resultExpression = new NumberInterpreter(result);
 stack.push(resultExpression);
 log.info(String.format("阶段结果入栈: %d", resultExpression.interpret()));
 }
 }
 int result = stack.pop().interpret();
 return result;

 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.interpreter;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import org.springframework.expression.Expression;
import org.springframework.expression.spel.standard.SpelExpressionParser;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:48
 */
@Slf4j
class InterpreterTest {

 @Test
 void interpret() {
 String myInterpreter = "6 100 11 + *";
 CustomExpressionParser expressionParser = new CustomExpressionParser();
 int result = expressionParser.parse(myInterpreter);
 log.info("解释器计算结果: " + result);
 }

 @Test
 void springTest() {
 org.springframework.expression.ExpressionParser parser = new SpelExpressionParser();
 Expression expression = parser.parseExpression("100 * 2 + 400 * 1 + 66");
 int result = (Integer) expression.getValue();
 log.info(String.valueOf(result));
 }
}

输出

14:52:28.086 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 6
14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 100
14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 11
14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 出栈: 11 和 100
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 应用运算符: +
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 阶段结果入栈: 111
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 出栈: 111 和 6
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 应用运算符: *
14:52:28.091 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 阶段结果入栈: 666
14:52:28.091 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.InterpreterTest - 解释器计算结果: 666


14:52:28.550 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.InterpreterTest - 666

观察者模式

介绍

定义：定义了对象之间的一对多依赖，让多个观察者对象同时监听某一个主题对象，当主题对象发生变化时，它的所有依赖者（观察者）都会收到通知并更新

类型：行为型

适用场景

关联行为场景，建立一套触发机制

优点

观察者模式支持广播通信
观察者和被观察者之间建立一个抽象的耦合

缺点

使用要得当,要避免循环调用
观察者之间有过多的细节依赖、提高时间消耗及程序复杂度

代码样例——JDK观察者

类图

课程问题

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.AllArgsConstructor;
import lombok.Getter;

/**
 * 课程问题
 *
 * @author F嘉阳
 * @date 2020/3/5 15:23
 */
@Getter
@AllArgsConstructor
public class Question {

 private String username;

 private String content;

}

课程（被观察者）

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;

import java.util.EventListener;
import java.util.Observable;
import java.util.Vector;

/**
 * 课程
 * 被观察者类
 *
 * {@link Observable} 使用 {@link Vector} 保证线程安全
 *
 * JDK源码
 * {@link java.awt.Event} 事件监听器
 * {@link javax.servlet.ServletRequestListener} HTTP请求事件监听器
 * 继承 {@link EventListener}，有大量实现，例如{@link org.springframework.beans.factory.parsing.ReaderEventListener}
 *
 * @author F嘉阳
 * @date 2020/3/5 15:23
 */
@Slf4j
@Getter
@AllArgsConstructor
public class Course extends Observable {

 private String courseName;

 /**
 * 产生问题
 * @param course
 * @param question
 */
 public void produceQuestion(Course course,Question question){
 log.info("{}在{}提交了一个问题",question.getUsername(),course.getCourseName());
 // 状态变化
 setChanged();
 // 通知观察者
 notifyObservers(question);
 }
}

老师实体（观察者）

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;

import java.util.Observable;
import java.util.Observer;

/**
 * 老师实体
 *
 * 老师观察课程是否产生了新的问题
 * @author F嘉阳
 * @date 2020/3/5 15:24
 */
@Slf4j
@Getter
@AllArgsConstructor
public class Teacher implements Observer {

 private String teacherName;

 /**
 * This method is called whenever the observed object is changed. An
 * application calls an <tt>Observable</tt> object's
 * <code>notifyObservers</code> method to have all the object's
 * observers notified of the change.
 *
 * @param o the observable object.
 * @param arg an argument passed to the <code>notifyObservers</code>
 */
 @Override
 public void update(Observable o, Object arg) {
 Course course = (Course) o;
 Question question = (Question) arg;
 log.info("{}老师的{}课程接收到一个{}提交的问题，内容为：{}",
 teacherName,course.getCourseName(),question.getUsername(),question.getContent());
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/5 15:36
 */
@Slf4j
class TeacherTest {

 @Test
 void update() {
 Course course = new Course("Java设计模式");
 Teacher teacher = new Teacher("fjy8018");

 course.addObserver(teacher);

 // 业务逻辑
 Question question = new Question("fjy","问题内容XXX");

 course.produceQuestion(course,question);
 }
}

输出

15:38:34.437 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.Course - fjy在Java设计模式提交了一个问题
15:38:34.444 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.Teacher - fjy8018老师的Java设计模式课程接收到一个fjy提交的问题，内容为：问题内容XXX

代码样例——Guava监听器

package top.fjy8018.designpattern.pattern.behavior.observer.guava;

import com.google.common.eventbus.Subscribe;
import lombok.extern.slf4j.Slf4j;

import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CopyOnWriteArraySet;

/**
 * Guava事件监听器
 * {@link EventBus} 使用内部对象 {@link SubscriberRegistry} 内部使用 {@link ConcurrentMap} 存储订阅者
 * 同时使用 {@link CopyOnWriteArraySet} 保证线程安全和订阅者唯一
 * 故{@link com.google.common.eventbus.Subscriber}
 * 重写了 {@link com.google.common.eventbus.Subscriber#equals(Object)}
 * 和 {@link com.google.common.eventbus.Subscriber#hashCode()} 方法
 *
 * @author F嘉阳
 * @date 2020/3/5 15:56
 */
@Slf4j
public class GuavaEvent {

 @Subscribe
 public void subscribe(String str) {
 // 业务逻辑
 log.info("执行订阅方法，传入参数是：{}", str);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.observer;

import com.google.common.eventbus.EventBus;
import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import top.fjy8018.designpattern.pattern.behavior.observer.guava.Course;
import top.fjy8018.designpattern.pattern.behavior.observer.guava.GuavaEvent;

/**
 * @author F嘉阳
 * @date 2020/3/5 15:58
 */
@Slf4j
class GuavaEventTest {

 @Test
 void subscribe() {
 EventBus eventBus = new EventBus();
 GuavaEvent event = new GuavaEvent();
 // 注册
 eventBus.register(event);
 eventBus.post("提交事件的内容");
 }
}

输出

16:24:27.716 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.GuavaEvent - 执行订阅方法，传入参数是：提交事件的内容

备忘录模式

介绍

定义：保存一个对象的某个状态，以便在适当的时候恢复对象。（类似历史记录存储）

“后悔药”

类型：行为型

适用场景

保存及恢复数据相关业务场景
后悔的时候，即想恢复到之前的状态

优点

为用户提供一种可恢复机制
存档信息的封装

缺点

资源占用

相关设计模式

备忘录模式和状态模式

代码样例

类图

文章

package top.fjy8018.designpattern.pattern.behavior.memento;

import lombok.AllArgsConstructor;
import lombok.Data;

/**
 * 文章，字段不宜过多，否则导致过大内存占用
 *
 * @author F嘉阳
 * @date 2020/3/6 17:00
 */
@Data
@AllArgsConstructor
public class Article {

 private String title;

 private String content;

 private String imgs;

 /**
 * 保存
 * @return
 */
 public ArticleMemento saveToMemento() {
 ArticleMemento articleMemento = new ArticleMemento(this.title,this.content,this.imgs);
 return articleMemento;
 }

 /**
 * 撤销
 * @param articleMemento
 */
 public void undoFromMemento(ArticleMemento articleMemento) {
 this.title = articleMemento.getTitle();
 this.content = articleMemento.getContent();
 this.imgs = articleMemento.getImgs();
 }
}

文章备忘录（快照）

package top.fjy8018.designpattern.pattern.behavior.memento;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.ToString;
import org.springframework.binding.message.StateManageableMessageContext;

/**
 * 文章备忘录（快照）
 * 快照不需要set方法，只能通过构造器注入
 *
 * Spring工作流中源码 {@link StateManageableMessageContext#createMessagesMemento()}
 *
 * @author F嘉阳
 * @date 2020/3/6 17:03
 */
@Getter
@ToString
@AllArgsConstructor
public class ArticleMemento {

 private String title;

 private String content;

 private String imgs;
}

备忘录核心，栈实现管理者

package top.fjy8018.designpattern.pattern.behavior.memento;

import java.util.Stack;

/**
 * 保存一个对象的某个状态，以便在适当的时候恢复对象
 *
 * @author F嘉阳
 * @date 2020/3/6 17:04
 */
public class ArticleMementoManager {

 /**
 * 使用栈特性实现动作撤销
 */
 private final Stack<ArticleMemento> ARTICLE_MEMENTO_STACK = new Stack<ArticleMemento>();

 public ArticleMemento getMemento() {
 ArticleMemento articleMemento= ARTICLE_MEMENTO_STACK.pop();
 return articleMemento;
 }

 public void addMemento(ArticleMemento articleMemento)
 {
 ARTICLE_MEMENTO_STACK.push(articleMemento);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.memento;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/3/6 17:07
 */
@Slf4j
class ArticleMementoTest {

 @Test
 void getTitle() {
 ArticleMementoManager articleMementoManager = new ArticleMementoManager();

 Article article= new Article("如影随行的设计模式A","手记内容A","手记图片A");
 // 存档
 ArticleMemento articleMemento = article.saveToMemento();

 articleMementoManager.addMemento(articleMemento);
 log.info("标题:"+article.getTitle()+" 内容:"+article.getContent()+" 图片:"+article.getImgs()+" 暂存成功");

 log.info("手记完整信息:"+article);
 
 log.info("修改手记start");

 article.setTitle("如影随行的设计模式B");
 article.setContent("手记内容B");
 article.setImgs("手记图片B");

 log.info("修改手记end");

 log.info("手记完整信息:"+article);

 articleMemento = article.saveToMemento();
 articleMementoManager.addMemento(articleMemento);
 
 article.setTitle("如影随行的设计模式C");
 article.setContent("手记内容C");
 article.setImgs("手记图片C");

 log.info("暂存回退start");
 // 取档
 log.info("回退出栈1次");
 articleMemento = articleMementoManager.getMemento();
 article.undoFromMemento(articleMemento);

 log.info("回退出栈2次");
 articleMemento = articleMementoManager.getMemento();
 article.undoFromMemento(articleMemento);

 log.info("暂存回退end");
 log.info("手记完整信息:"+article);
 }
}

输出

17:33:51.154 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 标题:如影随行的设计模式A 内容:手记内容A 图片:手记图片A 暂存成功
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式A, content=手记内容A, imgs=手记图片A)
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 修改手记start
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 修改手记end
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式B, content=手记内容B, imgs=手记图片B)
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 暂存回退start
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 回退出栈1次
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 回退出栈2次
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 暂存回退end
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式A, content=手记内容A, imgs=手记图片A)

命令模式

介绍

定义：将请求封装成对象,以便使用不同的请求

特点：命令模式解决了应用程序中对象的职责以及它们之间的通信方式

类型：行为型

适用场景

请求调用者和请求接收者需要解耦,使得调用者和接收者不直接交互
需要抽象出等待执行的行为

优点

降低耦合
容易扩展新命令或者一组命令

缺点

命令的无限扩展会增加类的数量，提高系统实现复杂度

相关设计模式

命令模式和备忘录模式

代码样例

类图

命令抽象

package top.fjy8018.designpattern.pattern.behavior.command;

/**
 * 命令抽象
 * <p>
 * Java相关源码 {@link Runnable} 可理解为命令抽象
 * 其实现类为具体的命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
public interface Command {
 void execute();
}

操作视频

package top.fjy8018.designpattern.pattern.behavior.command;

import lombok.AllArgsConstructor;

/**
 * 关闭视频命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
@AllArgsConstructor
public class CloseCourseVideoCommand implements Command {
 private final CourseVideo courseVideo;

 @Override
 public void execute() {
 courseVideo.close();
 }
}

打开视频命令

package top.fjy8018.designpattern.pattern.behavior.command;

import lombok.AllArgsConstructor;

/**
 * 打开视频命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
@AllArgsConstructor
public class OpenCourseVideoCommand implements Command {
 private final CourseVideo courseVideo;

 @Override
 public void execute() {
 courseVideo.open();
 }
}

关闭视频命令

package top.fjy8018.designpattern.pattern.behavior.command;

import lombok.AllArgsConstructor;

/**
 * 关闭视频命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
@AllArgsConstructor
public class CloseCourseVideoCommand implements Command {
 private final CourseVideo courseVideo;

 @Override
 public void execute() {
 courseVideo.close();
 }
}

命令执行人

package top.fjy8018.designpattern.pattern.behavior.command;

import java.util.ArrayList;
import java.util.List;

/**
 * 命令执行人
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
public class Staff {
 /**
 * 保证执行顺序
 */
 private final List<Command> commandList = new ArrayList<Command>();

 public void addCommand(Command command) {
 commandList.add(command);
 }

 public void executeCommands() {
 for (Command command : commandList) {
 command.execute();
 }
 commandList.clear();
 }

}

测试

package top.fjy8018.designpattern.pattern.behavior.command;

import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/7/21 16:02
 */
class CommandTest {

 @Test
 void doTest() {
 CourseVideo courseVideo = new CourseVideo("Java设计模式精讲");
 OpenCourseVideoCommand openCourseVideoCommand = new OpenCourseVideoCommand(courseVideo);
 CloseCourseVideoCommand closeCourseVideoCommand = new CloseCourseVideoCommand(courseVideo);

 Staff staff = new Staff();

 staff.addCommand(openCourseVideoCommand);
 staff.addCommand(closeCourseVideoCommand);

 staff.executeCommands();
 }
}

输出

Java设计模式精讲课程视频开放
Java设计模式精讲课程视频关闭

中介者模式

介绍

定义：定义一个封装一组对象如何交互的对象

特点：通过使对象明确地相互引用来促进松散耦合,并允许独立地改变它们的交互

适用场景

系统中对象之间存在复杂的引用关系，产生的相互依赖关系结构混乱且难以理解
交互的公共行为，如果需要改变行为则可以增加新的中介者类

优点

将一对多转化成了一对一、降低程序复杂度
类之间解耦

缺点

中介者过多,导致系统复杂

相关设计模式

中介者模式和观察者模式

代码样例

类图

学习群组（中介者）

package top.fjy8018.designpattern.pattern.behavior.mediator;

import java.util.Date;
import java.util.Timer;

/**
 * 学习群组（中介者）
 * <p>
 * jdk源码 {@link Timer#schedule(java.util.TimerTask, Date)} 多个方法重载
 * 都是调用{@link Timer#sched(java.util.TimerTask, long, long)} 实现的，由底层方法进行整体协调
 *
 * @author F嘉阳
 * @date 2020/7/21 16:20
 */
public class StudyGroup {

 public static void showMessage(User user, String message) {
 System.out.println(new Date().toString() + " [" + user.getName() + "] : " + message);
 }
}

用户

package top.fjy8018.designpattern.pattern.behavior.mediator;

import lombok.AllArgsConstructor;
import lombok.Data;

/**
 * @author F嘉阳
 * @date 2020/7/21 16:20
 */
@Data
@AllArgsConstructor
public class User {
 private String name;

 public void sendMessage(String message) {
 // 只和中介者交互
 StudyGroup.showMessage(this, message);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.mediator;

import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/7/21 16:25
 */
class MediatorTest {

 @Test
 void doTest() {
 User fjy = new User("F嘉阳");
 User tom = new User("Tom");

 fjy.sendMessage(" Hey! Tom! Let's learn Design Pattern");
 tom.sendMessage("OK! F嘉阳");
 }
}

输出

Tue Jul 21 16:47:03 CST 2020 [F嘉阳] : Hey! Tom! Let's learn Design Pattern
Tue Jul 21 16:47:03 CST 2020 [Tom] : OK! F嘉阳

责任链模式

介绍

定义：为请求创建一个接收此次请求对象的链

类型：行为型

适用场景

一个请求的处理需要多个对象当中的一个或几个协作处理

优点

请求的发送者和接收者(请求的处理)解耦
责任链可以动态组合

缺点

责任链太长或者处理时间过长,影响性能
责任链有可能过多

相关设计模式

责任链模式和状态模式

代码样例

类图

模拟二级审批场景

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import javax.servlet.FilterChain;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;

/**
 * 模拟二级审批场景
 * <p>
 * 审批抽象
 * <p>
 * java 过滤器实现 {@link javax.servlet.Filter#doFilter(ServletRequest, ServletResponse, FilterChain)}
 * 实现 {@link FilterChain#doFilter(ServletRequest, ServletResponse)} 接口的组成一条过滤器链
 * 所有子类通过调用 chain.doFilter(request, response) 实现链式传递
 * <p>
 * SpringSecurity过滤器链实现也类似
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
public abstract class Approver {
 /**
 * 责任链核心：包含与自身相同的类型对象
 */
 protected Approver approver;

 /**
 * 设置下一个节点，组成链条
 *
 * @param approver
 */
 public void setNextApprover(Approver approver) {
 this.approver = approver;
 }

 /**
 * 交由子类实现
 *
 * @param course
 */
 public abstract void deploy(Course course);
}

手记审批

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import org.apache.commons.lang3.StringUtils;

/**
 * 手记审批
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
public class ArticleApprover extends Approver {
 @Override
 public void deploy(Course course) {
 if (StringUtils.isNotEmpty(course.getArticle())) {
 System.out.println(course.getName() + "含有手记,批准");
 // 是否为末尾
 if (approver != null) {
 // 执行下一个处理链
 approver.deploy(course);
 }
 } else {
 System.out.println(course.getName() + "不含有手记,不批准,流程结束");
 return;
 }
 }
}

视频审批

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import org.apache.commons.lang3.StringUtils;

/**
 * 视频审批
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
public class VideoApprover extends Approver {
 @Override
 public void deploy(Course course) {
 if (StringUtils.isNotEmpty(course.getVideo())) {
 System.out.println(course.getName() + "含有视频,批准");
 if (approver != null) {
 approver.deploy(course);
 }
 } else {
 System.out.println(course.getName() + "不含有视频,不批准,流程结束");
 return;
 }
 }
}

课程

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import lombok.Data;

/**
 * 课程
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
@Data
public class Course {
 private String name;

 private String article;

 private String video;
}

测试

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/7/21 15:06
 */
class ChainofResponsibilityTest {

 @Test
 void doTest() {
 Approver articleApprover = new ArticleApprover();
 Approver videoApprover = new VideoApprover();

 Course course = new Course();
 course.setName("Java设计模式");
 course.setArticle("Java设计模式的手记");
 course.setVideo("Java设计模式的视频");

 articleApprover.setNextApprover(videoApprover);

 articleApprover.deploy(course);
 }
}

输出

Java设计模式含有手记,批准
Java设计模式含有视频,批准

访问者模式

介绍

定义：封装作用于某数据结构(如List/set/Map等)中的各元素的操作

特性：可以在不改变各元素的类的前提下定义作用于这些元素的操作

类型：行为型

适用场景

一个数据结构如(List/set/Map等)包含很多类型对象
数据结构与数据操作分离

优点

增加新的操作很容易即增加一个新的访问者

缺点

增加新的数据结构困难
具体元素变更比较麻烦

相关设计模式

访问者模式和迭代器模式

代码样例

类图

课程抽象

package top.fjy8018.designpattern.pattern.behavior.visitor;

import lombok.Data;

/**
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
@Data
public abstract class Course {
 private String name;

 public abstract void accept(IVisitor visitor);

}

实战课

package top.fjy8018.designpattern.pattern.behavior.visitor;

import lombok.Data;
import lombok.EqualsAndHashCode;

/**
 * 实战课，包含价格
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
@Data
@EqualsAndHashCode(callSuper = true)
public class CodingCourse extends Course {
 private int price;

 @Override
 public void accept(IVisitor visitor) {
 visitor.visit(this);
 }
}

免费课

package top.fjy8018.designpattern.pattern.behavior.visitor;

/**
 * 免费课
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
public class FreeCourse extends Course {

 /**
 * 接受策略
 *
 * @param visitor
 */
 @Override
 public void accept(IVisitor visitor) {
 visitor.visit(this);
 }
}

访问者

package top.fjy8018.designpattern.pattern.behavior.visitor;

/**
 * 封装操作数据结构的方法
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
public interface IVisitor {

 void visit(FreeCourse freeCourse);

 void visit(CodingCourse codingCourse);
}

访问者核心实现

package top.fjy8018.designpattern.pattern.behavior.visitor;

/**
 * 访问者
 * 核心：对不同类型产生不同的操作行为
 * <p>
 * jdk源码 {@link java.nio.file.FileVisitor} 实现对文件的访问
 * spring源码 {@link org.springframework.beans.factory.config.BeanDefinitionVisitor} 遍历Bean具体属性并填充
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
public class Visitor implements IVisitor {

 /**
 * 访问免费课程，打印所有免费课程名称
 *
 * @param freeCourse
 */
 @Override
 public void visit(FreeCourse freeCourse) {
 System.out.println("免费课程:" + freeCourse.getName());
 }

 /**
 * 访问实战课程，打印所有实战课程名称及价格
 *
 * @param codingCourse
 */
 @Override
 public void visit(CodingCourse codingCourse) {
 System.out.println("实战课程:" + codingCourse.getName() + " 价格:" + codingCourse.getPrice() + "元");
 }

}

测试

package top.fjy8018.designpattern.pattern.behavior.visitor;

import org.junit.jupiter.api.Test;

import java.util.ArrayList;
import java.util.List;

/**
 * @author F嘉阳
 * @date 2020/7/22 16:19
 */
class VisitorTest {

 @Test
 void visit() {
 List<Course> courseList = new ArrayList<>();

 FreeCourse freeCourse = new FreeCourse();
 freeCourse.setName("SpringMVC数据绑定");

 CodingCourse codingCourse = new CodingCourse();
 codingCourse.setName("Java设计模式精讲");
 codingCourse.setPrice(299);
 // 数组组装
 courseList.add(freeCourse);
 courseList.add(codingCourse);

 for (Course course : courseList) {
 course.accept(new Visitor());
 }
 }
}

输出

免费课程:SpringMVC数据绑定
实战课程:Java设计模式精讲 价格:299元

状态模式

介绍

定义：允许一个对象在其内部状态改变时改变它的行为

类型：行为型

适用场景

一个对象存在多个状态(不同状态下行为不同),且状态可相互转换

优点

将不同的状态隔离
把各种状态的转换逻辑,分布到 State的子类中,减少相互间依赖
增加新的状态非常简单

缺点

状态多的业务场景导致类数目增加,系统变复杂

相关设计模式

状态模式和享元模式

代码样例

类图

视频播放状态转换场景

package top.fjy8018.designpattern.pattern.behavior.state;

import javax.faces.lifecycle.Lifecycle;

/**
 * 视频播放状态转换场景
 * <p>
 * {@link Lifecycle#execute(javax.faces.context.FacesContext)} 该方法通过外部状态控制改变行为
 * {@link javax.faces.webapp.FacesServlet} 状态控制
 *
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public abstract class CourseVideoState {
 /**
 * 视频上下文
 */
 protected CourseVideoContext courseVideoContext;

 public void setCourseVideoContext(CourseVideoContext courseVideoContext) {
 this.courseVideoContext = courseVideoContext;
 }

 public abstract void play();

 public abstract void speed();

 public abstract void pause();

 public abstract void stop();

}

视频上下文

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * 视频上下文
 *
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class CourseVideoContext {

 /**
 * 存储当前状态
 */
 private CourseVideoState courseVideoState;

 /**
 * 结合享元模式设置为内存共享
 */
 public final static PlayState PLAY_STATE = new PlayState();
 public final static StopState STOP_STATE = new StopState();
 public final static PauseState PAUSE_STATE = new PauseState();
 public final static SpeedState SPEED_STATE = new SpeedState();

 public CourseVideoState getCourseVideoState() {
 return courseVideoState;
 }

 public void setCourseVideoState(CourseVideoState courseVideoState) {
 this.courseVideoState = courseVideoState;
 // 把自己本身设置到实例上下文中
 this.courseVideoState.setCourseVideoContext(this);
 }
 public void play(){
 this.courseVideoState.play();
 }

 public void speed(){
 this.courseVideoState.speed();
 }

 public void stop(){
 this.courseVideoState.stop();
 }

 public void pause(){
 this.courseVideoState.pause();
 }
}

各状态实现

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class PlayState extends CourseVideoState {

 @Override
 public void play() {
 System.out.println("正常播放课程视频状态");
 }

 @Override
 public void speed() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.SPEED_STATE);
 }

 @Override
 public void pause() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PAUSE_STATE);
 }

 @Override
 public void stop() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE);
 }
}

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class PauseState extends CourseVideoState {

 @Override
 public void play() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE);
 }

 @Override
 public void speed() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.SPEED_STATE);
 }

 @Override
 public void pause() {
 System.out.println("暂停播放课程视频状态");
 }

 @Override
 public void stop() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE);
 }
}

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class SpeedState extends CourseVideoState {
 @Override
 public void play() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE);
 }

 @Override
 public void speed() {
 System.out.println("快进播放课程视频状态");
 }

 @Override
 public void pause() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PAUSE_STATE);
 }

 @Override
 public void stop() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE);
 }
}

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class StopState extends CourseVideoState {
 @Override
 public void play() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE);
 }

 @Override
 public void speed() {
 System.out.println("ERROR 停止状态不能快进!!");
 }

 @Override
 public void pause() {
 System.out.println("ERROR 停止状态不能暂停!!");
 }

 @Override
 public void stop() {
 System.out.println("停止播放课程视频状态");
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.state;

import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/7/22 16:19
 */
class PlayStateTest {

 @Test
 void play() {
 CourseVideoContext courseVideoContext = new CourseVideoContext();
 courseVideoContext.setCourseVideoState(new PlayState());

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());
 courseVideoContext.pause();

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());

 courseVideoContext.speed();

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());

 courseVideoContext.stop();

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());

 courseVideoContext.speed();
 }
}

输出

当前状态:PlayState
当前状态:PauseState
当前状态:SpeedState
当前状态:StopState
ERROR 停止状态不能快进!!

微服务平台开发总结（3）——第二代密码管理平台

Thu, 07 May 2020 21:38:00 +0800

背景

吸取前两代系统的失败经验，对第二代密码管理微服务进行重新规划，并且在探索过程中见识到了cloud-platform、SpringBlade、pigx等优秀的微服务项目，考虑基于开源项目二次开发

规划

为了尽快落地上线，第一期以实现最小功能为主，其他非必须功能尽可能往后放

第一期

SpringCloud微服务集成
Nacos、Sentinel、graylog、zipkin、 Turbine集成
前后端密码管理功能对接
操作日志记录
邮件通知
Excel导入导出
v1迁移到v2工具包

第二期

dble读写分离
支持忘记密码、密码锁定等
集成微信推送、微信注册功能
支持上传前端日志
页面排序、检索优化

第三期

增加用户反馈功能
接入Seata分布式事务
支持配置导入导出权限
支持录入标签
…

技术体系

前端

前后端分离-采用前后端分离模式，前端基于Vue
采用Vuex进行状态管理，采用ES6作为语法支持
前后端基于开源项目SpringBlade二次开发
前后端代码生成-定制针对两套前端与后端的代码生成模板，轻松生成整个模块的前后端代码，减少重复工作量
多租户顶部菜单配置-提供给每个租户独立的顶部菜单配置模块，可以自定义顶部菜单切换

后端

基于SpringCloud的分布式架构
采用Mybatis与MybatisPlus作为ORM持久层框架
多租户系统-完整的SaaS多租户架构
借鉴OAuth2，通过Jwt令牌实现了多终端认证系统，可控制子系统的token权限互相隔离
权限管理-精心设计的权限管理方案，角色权限精确到按钮
集成邮件服务，通过事件传播实现不依赖消息队列通知发送服务
集成微信推送服务，实现微信注册、敏感操作消息推送
动态聚合文档-实现基于Nacos的Swagger SpringCloud聚合文档
实现动态网关，不重启就能设置网关配置项
集成Turbine聚合熔断监控

数据架构

采用MySQL进行数据存储，底层配置为一主两从的高可用集群
采用ElasticSearch三节点集群进行日志数据存储

分布式支持

Nacos-集成阿里巴巴的Nacos完成统一的服务注册与配置
Sentinel-集成Sentinel从流量控制、熔断降级、系统负载等多个维度保护服务的稳定性
采用Docker打包，部署于Kubernetes集群上
Seata分布式事务-定制集成Seata，支持分布式事务，无代码侵入，不失灵活与简洁
使用graylog进行分布式日志采集，集中监控
通过zipkin进行分布式链路追踪，耗时统计和服务调用情况

DevOps

配置管理采用单仓库模式，集成docker打包插件，自动推送到harbor仓库
采用GitLab Runner CICD流水线自动部署到测试环境和生产环境
结合Maven私服和Harbor私服为内网自动化部署提供支持

官方架构图

截图

管理员

操作日志记录

错误日志系统内采集

Turbine集成熔断监控

通知发布

普通用户

主题切换

微信扫码注册

补全信息

主页资源权限控制

菜单资源权限控制

Excel导入导出

密码管理

运维

Nacos注册中心

Sentinel限流管控

SpringBoot Admin应用内指标监控

Harbor内网私服

Nexus内网私服，提供maven和npm包

graylog日志采集

三节点ES集群支持日志存储

耗时统计

新UI风格

动态服务依赖关系图

新UI依赖图

Prometheus指标采集

Grafana资源指标监控

Kuberneties服务部署监控

开发过程工具

Gitlab CICD流水线

看板

功能开发周期分析

gitlab集成k8s部署环境

Sonarqube代码质量检查

分支图

工程模块图

结果——成功

经过两个月的开发和调试，第一期工程符合预定计划，按期投产上线运行，并经过2周试运行和测试后，进入生产环境，迁移老版本数据，正式替换了运行一年半的第一代密码管理系统

时间线

2020年3月开始基于SpringBlade 2.6.2编码
2020年4月20日第一个测试版本进入k8s测试环境
2020年4月28日第一个稳定版进行试运行阶段
2020年5月4日试运行阶段结束，第一个稳定版1.1.2发布，正式进行数据迁移并投入使用

微服务平台开发总结（1）——第一代密码管理系统

Thu, 07 May 2020 21:34:00 +0800

背景

随着网站、工作使用需要，对密码存储的需求越来越大，初期采用本地数据库进行存储，但不适合多地访问，一旦离开了本地电脑，就无法查询密码信息，故计划搭建个人使用的密码信息管理系统

概要

时间线

密码管理信息系统于2018年12月立项，2019年1月9日开始第一次提交
2019年2月7日第一个版本上线试运行
2019年4月7日进入维护阶段，不再支持新功能
2019年11月16日停止维护

架构

后端

JDK8
SpringBoot 2.1.1.RELEASE
Jetty
SpringData Jpa
MySQL 5.7
Docker
Kuberneties 1.14
Redis
Graylog
GitLab CICD

前端

Vue 2.5
element 2.4.11
vuex 3
vue-axios 2.1.4
eslint 5.8.0
nodejs
nginx
https
Docker
GitLab CICD

原理

后端存储密码数据字段和网站相关信息
采用SQL全字段模糊查询匹配结果
密码字段存储采用AES128结合SHA随机盐值加密存储，即使数据库被破解也无法查看对应的密码信息
密码加密秘钥不由服务端存储，只在用户登陆时录入并临时通过AES128加密后存储在Redis中，用户登出或者登陆超时后销毁

优势

单体项目，部署简单
容器化部署，迁移便利
Kuberneties云平台部署，支持灰度发布
无状态后端服务，支持集群化部署

安全审计

支持简单的安全审计

登陆日志记录
登陆归属地记录

安全问题

通过实现的Spring拦截器进行登陆拦截，安全性不高
没有采用session会话方式，故多地登陆账号且加密秘钥未过期则不需要再次录入
支持所有跨域行为，没有对客户端进行识别

其他问题

独立用户控制，扩展性不高
没有基于RBAC模型的权限控制
没有管理后台
单模块、单体项目，不利于扩展

界面演示

登陆页

若用户不存在则自动新增

录入加密秘钥

主页

信息录入

支持全字段模糊检索

支持Excel模板信息导入

登出

效果

系统稳定运行一年，符合系统初期设计目的
同时兼容移动端渲染

Git工作使用笔记

Fri, 13 Mar 2020 14:08:00 +0800

分离头指针

当前分支图

切换到之前的某一次提交

执行命令

$ git checkout de11fa87ea

提示，当前位于“分离头指针”状态

分支图

注意

在分离头指针状态下没有绑定分支
产生的commit不会被保存
分支切换后在分支图中看不到该commit
一段时间后会被git自动清除

使用场景

尝试性做一些变更，可以随时丢弃

此处演示分离头指针丢失commit的情况和补救措施

丢失commit

使用IDE修改文件并提交，忽略所有警告，修改后分支图如下

此时接到其他需求，需要切换分支进行紧急修复

执行命令

$ git checkout master

分支切换成功，并弹出提示和告警

Warning: you are leaving 1 commit behind, not connected to any of your branches:

199ac20 游离状态修改文件

If you want to keep it by creating a new branch, this may be a good time to do so with:

git branch 199ac20

Switched to branch ‘master’ Your branch is up to date with ‘origin/master’.

此时查看分支图，刚才的commit已经不可见

补救

此时发现刚才的commit十分重要，可根据git的提示进行补救

执行命令

$ git branch hot-fix 199ac20

再次查看分支图，可见commit已经恢复

本地分支操作

以下操作仅适用于本地分支，无远程分支协同工作的情况

修改当前commit的message——`amend`

当前分支图

执行命令

$ git commit --amend

自动弹出编辑器

修改后保存并关闭编辑器即可，输出如下

再次查看分支图

修改前面某次commit的message——`rebase`

现计划修改如下message

使用IDE拷贝其父提交的SHA值

执行命令，-i表示交互式执行

$ git rebase -i 199ac203c90f881024c6870d56517df9e2080841

自动弹出编辑器

同时包含提示操作

Rebase 199ac20..bfe2b7f onto 199ac20 (3 commands)

Commands: p, pick = use commit r, reword = use commit, but edit the commit message e, edit = use commit, but stop for amending s, squash = use commit, but meld into previous commit f, fixup = like “squash”, but discard this commit’s log message x, exec = run command (the rest of the line) using shell b, break = stop here (continue rebase later with ‘git rebase –continue’) d, drop = remove commit l, label = label current HEAD with a name t, reset = reset HEAD to a label m, merge [-C | -c ] [# ] . create a merge commit using the original merge commit’s . message (or the oneline, if no original merge commit was . specified). Use -c to reword the commit message.

These lines can be re-ordered; they are executed from top to bottom.

If you remove a line here THAT COMMIT WILL BE LOST.

However, if you remove everything, the rebase will be aborted.

Note that empty commits are commented out

由此可见“变基”支持很多操作，此处需要使用的是reword，修改目标提交的命令为r

保存并关闭编辑器，会自动弹出新编辑器界面

修改后保存并关闭即可，输出如下

查看分支图

整理连续多个commit——`rebase`

现计划合并3个commit

使用IDE拷贝其父SHA值

执行命令

$ git rebase -i 199ac203c90f881024c6870d56517df9e2080841

此处使用squash，官方介绍如下

s, squash = use commit, but meld into previous commit

修改内容如下，说明将中间两个合并进第一个commit中

保存后关闭，会弹出新的编辑器界面，提示可以输出合并commit的message

编写message

保存后关闭，提示修改成功

查看分支图

分支合并策略

演示不同分支合并策略效果，当前的分支图如下

现在将hot-fix合并进master

GitLab默认合并策略——`merge commit`

gitlab默认合并请求使用的策略

由于没有冲突，可以线上直接合并

其对应执行的命令如下

Step 1. Fetch and check out the branch for this merge request
$ git fetch origin
$ git checkout -b hot-fix origin/hot-fix
Step 2. Review the changes locally

Step 3. Merge the branch and fix any conflicts that come up
$ git fetch origin
$ git checkout origin/master
$ git merge --no-ff hot-fix
Step 4. Push the result of the merge to GitLab
$ git push origin master

合并完成

查看分支图，故当前分支会影响特性分支并产生新的提交

GitLab合并提交策略

提交合并请求的时候勾选策略Squash commits when merge request is accepted

其对应执行的命令如下（与默认策略一致）

Step 1. Fetch and check out the branch for this merge request
$ git fetch origin
$ git checkout -b hot-fix origin/hot-fix
Step 2. Review the changes locally

Step 3. Merge the branch and fix any conflicts that come up
$ git fetch origin
$ git checkout origin/master
$ git merge --no-ff hot-fix
Step 4. Push the result of the merge to GitLab
$ git push origin master

合并成功

查看分支图，可见其影响特性分支，会产生新的提交，同时覆盖特性分支最后一次提交

分支合并策略——`squash merge`

GitHub支持squash merge合并策略，此处通过命令行执行测试

$ git merge --squash origin/hot-fix
$ git commit -m "squash commit"

查看分支图，特性分支不变，目标分支合并提交后产生一条提交，效果与GitLab合并提交类似，但命令使用方式不同

注意：squash merge会变更提交者作者信息，这是一个很大的问题，后期问题追溯不好处理

变基合并策略——`rebase merge`

该策略也是GitHub支持的合并策略，可能会产生较多冲突

选择合并策略，最后一种即变基合并策略

合并成功

查看分支图，其产生的三次提交即变基合并的结果，不会影响特性分支，也不会变更提交人，类似cherry pick

分支回滚策略

回滚策略分为两种，一种是合并后回滚，一种是普通提交回滚，不同回滚策略操作方法不同

普通提交回滚

当前分支图，计划回滚最新提交

直接在IDE中执行revert操作即可

回滚会产生新的提交，回滚结束

合并后回滚

此时分支图，使用merge commit方式进行过一次合并，此时不想要合并的结果，想恢复到未合并状态

错误操作——直接执行`revert`

此时IDE已经不支持通过revert进行回滚合并操作，通过命令行强行执行

查找SHA值

回滚merge需要加上-m参数git就知道到底要revert哪一个merge

$ git revert -n 25d7b405a4d50fe1b36019d90276973a8ed9160d -m 1

回滚后做一次commit即可，使用默认的message后如下

验证回滚结果，使用两个提交进行差异比对

无差异，回滚看似成功了

此时在原分支做了修复后再进行合并

报无法合并，因为分支已经进行过合并

正确操作——`reset`

使用reset操作到上一次提交

使用--hard参数，丢弃后续的commit

此时继续进行操作，并支持重新合并

禁止类操作

说明：禁止类说明是对开发人员禁止，而该命令在必要时候依然需要由配置管理员使用

`push -f`

官方使用说明

-f, –force force updates

原因：

默认情况下远端分支若能顺利合并（fast forward），则会顺利合并
若远端分支产生冲突，则会停止push，同时提示错误，要pull到本地处理冲突后重新push
加-f之后，本地分支会强行推送到远端，若本地分支落后于远端分支也不会弹出提示，由此导致远程commit丢失

适用场景：

远程分支已经混乱，本地分支符合需要，需要强制整理远程分支

公共分支禁止`rebase`

此处对场景进行模拟，正常情况下，多位开发者对公共分支common-branch进行修改，分支图如下

另一开发者在本地修改后执行变基操作，此处以修改历史提交message为例

变基

录入变基信息

变基成功

此时查看本地分支图，可见产生了新的分支路径，同时远程message并没有变化，故rebase对存在远程分支的情况无效

变基后推送到远端分支，提示需要合并，选择“合并”

合并后分支图如下

此时其他开发者修改文件后并fetch远程分支查看变更情况如下

此时无法对远程分支进行push操作

处理方法

pull远程分支在本地合并后重新推送，此时在本地再产生一次合并

此时可以成功推送

由此一来一回操作导致大量不必要的合并操作，并可能导致大量冲突，并且没有解决问题

警告类操作

`reset --hard`

官方对reset操作用法说明

usage: git reset [–mixed | –soft | –hard | –merge | –keep] [-q] [] or: git reset [-q] [] [–] … or: EXPERIMENTAL: git reset [-q] [–stdin [-z]] [] or: git reset –patch [] [–] […]

-q, --quiet be quiet, only report errors
--mixed reset HEAD and index
--soft reset only HEAD
--hard reset HEAD, index and working tree
--merge reset HEAD, index and working tree
--keep reset HEAD but keep local changes
--recurse-submodules[=<reset>]
 control recursive updating of submodules
-p, --patch select hunks interactively
-N, --intent-to-add record only the fact that removed paths will be added later
-z EXPERIMENTAL: paths are separated with NUL character
--stdin EXPERIMENTAL: read paths from <stdin>

原因：

在分支上执行该操作会丢失目标commit以后的所有commit

适用场景

用于分支回滚且不打算保留其后的所有变更

设计模式学习——结构型

Wed, 04 Mar 2020 10:44:04 +0800

外观模式

介绍

又叫门面模式,提供了一个统一的接口,用来访问子系统中的一群接口

外观模式定义了一个高层接口,让子系统更容易使用

类型：结构型

适用场景：

子系统越来越复杂,增加外观模式提供简单调用接口
构建多层系统结构,利用外观对象作为每层的入口,简化层间调用

优点：

简化了调用过程，无需了解深入子系统，防止带来风险。
减少系统依赖、松散耦合
更好的划分访问层次
符合迪米特法则，即最少知道原则

缺点：

增加子系统、扩展子系统行为容易引入风险
不符合开闭原则

代码样例

改造前

类图

package top.fjy8018.designpattern.pattern.structural.facade.before;

import lombok.AllArgsConstructor;
import lombok.Getter;

/**
 * 外观模式又叫门面模式,提供了一个统一的接口,用来访问子系统中的一群接口
 * 积分礼物
 *
 * @author F嘉阳
 * @date 2020/2/28 10:33
 */
@Getter
@AllArgsConstructor
public class PointGift {
 private String name;
}

子系统一：积分校验子系统

package top.fjy8018.designpattern.pattern.structural.facade.before;

import lombok.extern.slf4j.Slf4j;

/**
 * 积分校验子系统
 *
 * @author F嘉阳
 * @date 2020/2/28 10:34
 */
@Slf4j
public class QualifyService {

 /**
 * 模拟积分校验逻辑
 * @param pointGift
 * @return
 */
 public boolean isAvalible(PointGift pointGift){
 log.info("校验礼物 {} 通过，积分校验通过，库存通过",pointGift.getName());
 return true;
 }
}

子系统二：积分支付校验子系统

package top.fjy8018.designpattern.pattern.structural.facade.before;

import lombok.extern.slf4j.Slf4j;

/**
 * 积分支付校验子系统
 *
 * @author F嘉阳
 * @date 2020/2/28 10:37
 */
@Slf4j
public class PointPaymentService {

 public boolean pay(PointGift pointGift){
 // 扣减积分逻辑
 log.info("支付礼物 {} 积分成功",pointGift.getName());
 return true;
 }
}

子系统三：物流子系统

package top.fjy8018.designpattern.pattern.structural.facade.before;

import lombok.extern.slf4j.Slf4j;

/**
 * 物流子系统
 *
 * @author F嘉阳
 * @date 2020/2/28 10:39
 */
@Slf4j
public class ShippingService {

 /**
 * 返回物流运单号
 *
 * @param pointGift
 * @return
 */
 public String shipGift(PointGift pointGift){
 // 物流系统对接逻辑
 log.info("礼物 {} 进入物流系统",pointGift.getName());
 return "123456";
 }
}

外观类——礼物兑换逻辑

package top.fjy8018.designpattern.pattern.structural.facade.before;

import lombok.extern.slf4j.Slf4j;

/**
 * 外观类——礼物兑换逻辑
 *
 * 该服务定义了与各个子系统交互的逻辑，业务系统无需知道与内部子系统的交互逻辑，符合迪米特法则
 * @author F嘉阳
 * @date 2020/2/28 10:44
 */
@Slf4j
public class GiftExchangeService {

 private PointPaymentService pointPaymentService;

 private QualifyService qualifyService;

 private ShippingService shippingService;

 public void setPointPaymentService(PointPaymentService pointPaymentService) {
 this.pointPaymentService = pointPaymentService;
 }

 public void setQualifyService(QualifyService qualifyService) {
 this.qualifyService = qualifyService;
 }

 public void setShippingService(ShippingService shippingService) {
 this.shippingService = shippingService;
 }

 /**
 * 系统交互逻辑
 *
 * @param pointGift
 */
 public void giftExchange(PointGift pointGift){
 if (qualifyService.isAvalible(pointGift)){
 // 资格校验通过
 if (pointPaymentService.pay(pointGift)){
 // 如果积分支付成功
 String shippingOrderNo = shippingService.shipGift(pointGift);
 log.info("物流系统下单成功，订单号为{}",shippingOrderNo);
 }
 }
 }
}

测试

package top.fjy8018.designpattern.pattern.structural.facade.before;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/2/28 10:53
 */
@Slf4j
class GiftExchangeServiceTest {

 @Test
 void giftExchange() {
 PointGift pointGift = new PointGift("手环");
 // 注入服务
 GiftExchangeService exchangeService = new GiftExchangeService();
 exchangeService.setPointPaymentService(new PointPaymentService());
 exchangeService.setQualifyService(new QualifyService());
 exchangeService.setShippingService(new ShippingService());
 // 兑换
 exchangeService.giftExchange(pointGift);
 }
}

执行结果

10:56:55.639 [main] INFO top.fjy8018.designpattern.pattern.structural.facade.before.QualifyService - 校验礼物 手环 通过，积分校验通过，库存通过
10:56:55.652 [main] INFO top.fjy8018.designpattern.pattern.structural.facade.before.PointPaymentService - 支付礼物 手环 积分成功
10:56:55.652 [main] INFO top.fjy8018.designpattern.pattern.structural.facade.before.ShippingService - 礼物 手环 进入物流系统
10:56:55.652 [main] INFO top.fjy8018.designpattern.pattern.structural.facade.before.GiftExchangeService - 物流系统下单成功，订单号为123456

改造后

改造点很简单，只要将应用层注入的类在逻辑层注入即可

package top.fjy8018.designpattern.pattern.structural.facade;

import lombok.extern.slf4j.Slf4j;

/**
 * 外观类——礼物兑换逻辑
 * <p>
 * 该服务定义了与各个子系统交互的逻辑，业务系统无需知道与内部子系统的交互逻辑，符合迪米特法则
 *
 * @author F嘉阳
 * @date 2020/2/28 10:44
 */
@Slf4j
public class GiftExchangeService {

 private PointPaymentService pointPaymentService = new PointPaymentService();

 private QualifyService qualifyService = new QualifyService();

 private ShippingService shippingService = new ShippingService();

 /**
 * 系统交互逻辑
 *
 * @param pointGift
 */
 public void giftExchange(PointGift pointGift) {
 if (qualifyService.isAvalible(pointGift)) {
 // 资格校验通过
 if (pointPaymentService.pay(pointGift)) {
 // 如果积分支付成功
 String shippingOrderNo = shippingService.shipGift(pointGift);
 log.info("物流系统下单成功，订单号为{}", shippingOrderNo);
 }
 }
 }
}

业务层

package top.fjy8018.designpattern.pattern.structural.facade;

import lombok.extern.slf4j.Slf4j;
import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.RequestFacade;

/**
 * 外观类——礼物兑换逻辑
 * <p>
 * 该服务定义了与各个子系统交互的逻辑，业务系统无需知道与内部子系统的交互逻辑，符合迪米特法则
 *
 * 源码：
 * {@link org.springframework.jdbc.support.JdbcUtils} 对数据库相关的操作进行封装
 * {@link org.apache.ibatis.session.Configuration#newMetaObject(Object)} 对创建对象的方法进行封装
 * {@link RequestFacade} 其定义了操作HTTPRequest的公用方法 {@link Request} 在Request中都用该外观类操作
 * 其他类似逻辑：
 * {@link org.apache.catalina.connector.ResponseFacade}
 * {@link org.apache.catalina.session.StandardSessionFacade}
 *
 *
 * @author F嘉阳
 * @date 2020/2/28 10:44
 */
@Slf4j
public class GiftExchangeService {

 private PointPaymentService pointPaymentService = new PointPaymentService();

 private QualifyService qualifyService = new QualifyService();

 private ShippingService shippingService = new ShippingService();

 /**
 * 系统交互逻辑
 *
 * @param pointGift
 */
 public void giftExchange(PointGift pointGift) {
 if (qualifyService.isAvalible(pointGift)) {
 // 资格校验通过
 if (pointPaymentService.pay(pointGift)) {
 // 如果积分支付成功
 String shippingOrderNo = shippingService.shipGift(pointGift);
 log.info("物流系统下单成功，订单号为{}", shippingOrderNo);
 }
 }
 }
}

类图

此时业务层无需关心子系统，只和外观类通信

装饰器模式

介绍

定义：在不改变原有对象的基础之上，将功能附加到对象上

提供了比继承更有弹性的替代方案（扩展原有对象功能）

类型：结构型

适用场景

扩展一个类的功能或给一个类添加附加职责
动态的给一个对象添加功能，这些功能可以再动态的撤销

优点

继承的有力补充,比继承灵活,不改变原有对象的情况下给一个对象扩展功能
通过使用不同装饰类以及这些装饰类的排列组合,可以实现不同效果
符合开闭原则

缺点

会出现更多的代码,更多的类,增加程序复杂性
动态装饰时,多层装饰时会更复杂

相关设计模式

装饰者模式和代理模式
装饰者模式和适配器模式

代码样例

改造前

类图

Battercake

package top.fjy8018.designpattern.pattern.structural.decorator.before;

/**
 * @author F嘉阳
 * @date 2018-09-27 16:56
 */
public class Battercake {
 protected String getDesc() {
 return "煎饼";
 }

 protected int cost() {
 return 8;
 }
}

BattercakeWithEgg

package top.fjy8018.designpattern.pattern.structural.decorator.before;

/**
 * 不使用装饰器则只能通过类继承扩展功能，可能引起类爆炸
 *
 * @author F嘉阳
 * @date 2018-09-27 16:59
 */
public class BattercakeWithEgg extends Battercake {
 @Override
 protected String getDesc() {
 return super.getDesc() + " 加一个鸡蛋";
 }

 @Override
 protected int cost() {
 return super.cost() + 1;
 }
}

BattercakeWithEggSausage

package top.fjy8018.designpattern.pattern.structural.decorator.before;

/**
 * @author F嘉阳
 * @date 2018-09-27 17:00
 */
public class BattercakeWithEggSausage extends BattercakeWithEgg {
 @Override
 protected String getDesc() {
 return super.getDesc() + " 加一根香肠";
 }

 @Override
 protected int cost() {
 return super.cost() + 2;
 }
}

测试

package top.fjy8018.designpattern.pattern.structural.decorator;

@Slf4j
class BattercakeTest {

 /**
 * 通过继承扩展灵活度低，无法自由组合
 */
 @Test
 void before() {
 Battercake battercake = new Battercake();
 log.info(battercake.getDesc() + " 销售价格:" + battercake.cost());

 Battercake battercakeWithEgg = new BattercakeWithEgg();
 log.info(battercakeWithEgg.getDesc() + " 销售价格:" + battercakeWithEgg.cost());


 Battercake battercakeWithEggSausage = new BattercakeWithEggSausage();
 log.info(battercakeWithEggSausage.getDesc() + " 销售价格:" + battercakeWithEggSausage.cost());
 }
}

输出

14:42:18.640 [main] INFO top.fjy8018.designpattern.pattern.structural.decorator.BattercakeTest - 煎饼 销售价格:8
14:42:18.652 [main] INFO top.fjy8018.designpattern.pattern.structural.decorator.BattercakeTest - 煎饼 加一个鸡蛋 销售价格:9
14:42:18.652 [main] INFO top.fjy8018.designpattern.pattern.structural.decorator.BattercakeTest - 煎饼 加一个鸡蛋 加一根香肠 销售价格:11

改造后

类图

所有类的抽象

package top.fjy8018.designpattern.pattern.structural.decorator.after;

/**
 * 装饰器模式按照业务场景一般有两层抽象，
 * 一层是所有类的抽象，一层是所有装饰器的抽象，
 * 所有装饰器的抽象可以为实现类提供实现额外功能的能力
 *
 * @author F嘉阳
 * @date 2018-09-27 17:02
 */
public abstract class AbstractBattercake {
 public abstract String getDesc();

 public abstract int cost();
}

所有装饰器的抽象

package top.fjy8018.designpattern.pattern.structural.decorator.after;

/**
 * @author F嘉阳
 * @date 2018-09-27 17:05
 */
public abstract class AbstractBattercakeDecorator extends AbstractBattercake {

 private AbstractBattercake abstractBattercake;

 /**
 * 通过构造器注入待装饰对象
 *
 * @param abstractBattercake
 */
 public AbstractBattercakeDecorator(AbstractBattercake abstractBattercake) {
 this.abstractBattercake = abstractBattercake;
 }

 /**
 * 实现类可以提供额外能力
 */
 protected abstract void doSomething();

 @Override
 public String getDesc() {
 return this.abstractBattercake.getDesc();
 }

 @Override
 public int cost() {
 return this.abstractBattercake.cost();
 }
}

package top.fjy8018.designpattern.pattern.structural.decorator.after;

/**
 * Servlet源码：{@link javax.servlet.http.HttpServletRequestWrapper}
 * JDK源码：IO类：{@link java.io.BufferedReader} {@link java.io.BufferedInputStream}
 * Spring源码:{@link org.springframework.cache.transaction.TransactionAwareCacheDecorator}
 * mybatis源码：{@link org.apache.ibatis.cache.decorators.LruCache}
 *
 * @author F嘉阳
 * @date 2018-09-27 17:04
 */
public class ABattercake extends AbstractBattercake {
 @Override
 public String getDesc() {
 return "煎饼";
 }

 @Override
 public int cost() {
 return 8;
 }
}

package top.fjy8018.designpattern.pattern.structural.decorator.after;

import lombok.extern.slf4j.Slf4j;

/**
 * @author F嘉阳
 * @date 2018-09-27 17:08
 */
@Slf4j
public class EggBattercake extends AbstractBattercakeDecorator {

 public EggBattercake(AbstractBattercake abstractBattercake) {
 super(abstractBattercake);
 }

 @Override
 protected void doSomething() {
 log.info("EggBattercake doSomething...");
 }

 @Override
 public String getDesc() {
 return super.getDesc() + " 加一个鸡蛋";
 }

 @Override
 public int cost() {
 return super.cost() + 1;
 }
}

package top.fjy8018.designpattern.pattern.structural.decorator.after;

import lombok.extern.slf4j.Slf4j;

/**
 * @author F嘉阳
 * @date 2018-09-27 17:10
 */
@Slf4j
public class SausageBattercake extends AbstractBattercakeDecorator {
 public SausageBattercake(AbstractBattercake abstractBattercake) {
 super(abstractBattercake);
 }

 @Override
 protected void doSomething() {
 log.info("SausageBattercake doSomething...");
 }

 @Override
 public String getDesc() {
 return super.getDesc() + " 加一根香肠";
 }

 @Override
 public int cost() {
 return super.cost() + 2;
 }
}

测试

package top.fjy8018.designpattern.pattern.structural.decorator;

@Slf4j
class BattercakeTest {

 @Test
 void after() {
 // 自由组合装饰
 AbstractBattercake battercake = new EggBattercake(new SausageBattercake(new ABattercake()));
 log.info(battercake.getDesc() + " 销售价格:" + battercake.cost());

 AbstractBattercake battercake2 = new EggBattercake((new ABattercake()));
 log.info(battercake2.getDesc() + " 销售价格:" + battercake2.cost());
 }
}

输出

14:42:18.660 [main] INFO top.fjy8018.designpattern.pattern.structural.decorator.BattercakeTest - 煎饼 加一根香肠 加一个鸡蛋 销售价格:11
14:42:18.660 [main] INFO top.fjy8018.designpattern.pattern.structural.decorator.BattercakeTest - 煎饼 加一个鸡蛋 销售价格:9

适配器模式

介绍

定义将一个类的接口转换成客户期望的另一个接口

使原本接口不兼容的类可以一起工作

类型：结构型

适用场景

不是软件设计阶段考虑的设计模式,是随着软件维护,由于不同产品、不同厂家造成功能类似而接口不相同情况下的解决方案
已经存在的类,它的方法和需求不匹配时(方法结果相同或相似)

优点

能提高类的透明性和复用,现有的类复用但不需要改变
目标类和适配器类解耦,提高程序扩展性
符合开闭原则

缺点

适配器编写过程需要全面考虑,可能会增加系统的复杂性
增加系统代码可读的难度

分类与扩展

对象适配器
类适配器

相关设计模式

适配器模式和外观模式

代码样例——电源适配器

类图

220V

package top.fjy8018.designpattern.pattern.structural.adapter.poweradapter;

import lombok.extern.slf4j.Slf4j;

/**
 * @author F嘉阳
 * @date 2018-10-09 16:24
 */
@Slf4j
public class AC220 {
 public int output() {
 int output = 220;
 log.info(this.getClass().getSimpleName() + "output:" + output + "V");
 return output;
 }
}

package top.fjy8018.designpattern.pattern.structural.adapter.poweradapter;

/**
 * @author F嘉阳
 * @date 2018-10-09 16:25
 */
public interface DC5 {
 int output();
}

适配器

package top.fjy8018.designpattern.pattern.structural.adapter.poweradapter;

import lombok.extern.slf4j.Slf4j;

/**
 * @author F嘉阳
 * @date 2018-10-09 16:25
 */
@Slf4j
public class Adapter implements DC5 {

 private AC220 ac220 = new AC220();

 @Override
 public int output() {
 int output220 = ac220.output();
 // 变压器模拟
 int output5 = output220 / 44;
 log.info("{} 输入{}V,输出{}V", this.getClass().getSimpleName(), output220, output5);
 return output5;
 }
}

测试

package top.fjy8018.designpattern.pattern.structural.adapter.poweradapter;

import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

class AdapterTest {

 @Test
 void output() {
 AC220 ac220 = new AC220();
 ac220.output();

 DC5 dc5 = new Adapter();
 dc5.output();
 }
}

输出

14:58:14.719 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.poweradapter.AC220 - AC220output:220V
14:58:14.729 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.poweradapter.AC220 - AC220output:220V
14:58:14.729 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.poweradapter.Adapter - Adapter 输入220V,输出5V

代码样例——类适配器

类图

目标类实现接口

package top.fjy8018.designpattern.pattern.structural.adapter.classadapter;

/**
 * 目标类实现接口
 *
 * @author F嘉阳
 * @date 2018-10-09 16:05
 */
public interface Target {
 void request();
}

被适配对象

package top.fjy8018.designpattern.pattern.structural.adapter.classadapter;

import lombok.extern.slf4j.Slf4j;

/**
 * 适配器模式：
 * 类适配器模式
 * 被适配对象
 *
 * @author F嘉阳
 * @date 2018-10-09 16:04
 */
@Slf4j
public class Adaptee {

 public void adapteeRequest() {
 log.info("adapteeRequest...");
 }
}

package top.fjy8018.designpattern.pattern.structural.adapter.classadapter;

/**
 * 适配器模式：将一个类的接口转换成客户希望的另一个接口。适配器模式让那些接口不兼容的类可以一起工作
 *
 * JDK源码：{@link javax.xml.bind.annotation.adapters.XMLAdapter}
 * spring源码：{@link jdk.internal.org.objectweb.asm.commons.AdviceAdapter}
 * {@link org.springframework.web.servlet.HandlerAdapter}
 * JPA源码：{@link org.springframework.orm.jpa.JpaVendorAdapter}
 *
 * @author F嘉阳
 * @date 2018-10-09 16:07
 */
public class Adapter extends Adaptee implements Target {
 @Override
 public void request() {
 // ... 新增业务逻辑
 super.adapteeRequest();
 }
}

目标类具体实现

package top.fjy8018.designpattern.pattern.structural.adapter.classadapter;

import lombok.extern.slf4j.Slf4j;

/**
 * 目标类具体实现
 *
 * @author F嘉阳
 * @date 2018-10-09 16:06
 */
@Slf4j
public class ConcreteTarget implements Target {

 @Override
 public void request() {
 log.info("ConcreteTarget...");
 }
}

测试

package top.fjy8018.designpattern.pattern.structural.adapter.classadapter;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

@Slf4j
class AdapterTest {

 /**
 * 通过适配器实现调用新接口将具体实现转发到旧类中实现
 */
 @Test
 void request() {
 Target newTarget = new ConcreteTarget();
 newTarget.request();

 Target adapter = new Adapter();
 adapter.request();
 }
}

输出

15:05:50.583 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.classadapter.ConcreteTarget - ConcreteTarget...
15:05:50.601 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.classadapter.Adaptee - adapteeRequest...

代码样例——对象适配器

类图

Target、Adaptee、ConcreteTarget都与类适配器一致、Adapter修改如下

package top.fjy8018.designpattern.pattern.structural.adapter.objectadapter;

/**
 * 适配器模式：将一个类的接口转换成客户希望的另一个接口。适配器模式让那些接口不兼容的类可以一起工作
 *
 * @author F嘉阳
 * @date 2018-10-09 16:07
 */
public class Adapter implements Target {

 /**
 * 将继承改成组合方式，降低耦合度
 */
 private Adaptee adaptee = new Adaptee();

 @Override
 public void request() {
 // ... 新增业务逻辑
 adaptee.adapteeRequest();
 }
}

测试

package top.fjy8018.designpattern.pattern.structural.adapter.objectadapter;

import org.junit.jupiter.api.Test;

class AdapterTest {

 @Test
 void request() {
 Target newTarget = new ConcreteTarget();
 newTarget.request();

 Target adapter = new Adapter();
 adapter.request();
 }
}

输出

15:10:00.866 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.objectadapter.ConcreteTarget - ConcreteTarget...
15:10:00.876 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.objectadapter.Adaptee - adapteeRequest...

享元模式

介绍

定义：提供了减少对象数量从而改善应用所需的对象结构的方式

运用共享技术有效地支持大量细粒度的对象

类型：结构型

适用场景

常常应用于系统底层的开发,以便解决系统的性能问题
系统有大量相似对象、需要缓冲池的场景。

优点

减少对象的创建，降低内存中对象的数量，降低系统的内存，提高效率
减少内存之外的其他资源占用

缺点

关注内/外部状态、关注线程安全问题
使系统、程序的逻辑复杂化

分类与扩展

内部状态
外部状态

相关设计模式

享元模式和代理模式
享元模式和单例模式

代码样例

类图

package top.fjy8018.designpattern.pattern.structural.flyweight;

/**
 * 享元模式：利用共享的方式来支持大量细粒度的对象，这些对象一部分内部状态是相同的。
 * JDK源码：{@link Integer#valueOf(int)} {@link Integer.IntegerCache}
 * 其他类型同理 {@link Long#valueOf(long)} {@link Boolean#valueOf(boolean)} {@link Byte#valueOf(byte)} {@link Character#valueOf(char)}
 * Apache源码：连接池{@link org.apache.commons.pool2.impl.GenericObjectPool#returnObject(Object)},连接池配置 {@link org.apache.commons.pool2.impl.GenericObjectPoolConfig}
 *
 * @author F嘉阳
 * @date 2018-10-14 16:13
 */
public interface Employee {
 void report();
}

员工工厂类

package top.fjy8018.designpattern.pattern.structural.flyweight;

import lombok.extern.slf4j.Slf4j;

import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

/**
 * 员工工厂类
 *
 * @author F嘉阳
 * @date 2018-10-14 16:15
 */
@Slf4j
public class EmployeeFactory {
 private static final Map<String, Employee> EMPLOYEE_MAP = new ConcurrentHashMap<>();

 public static Employee getEmployee(String department) {
 Manager manager = (Manager) EMPLOYEE_MAP.get(department);
 // 如果在对象池中不存在，则新建对象
 if (manager == null) {
 manager = new Manager(department);
 log.warn("创建{}部门经理", department);
 String content = "报告内容……";
 log.warn("创建报告，内容为:{}", content);
 manager.setContent(content);
 EMPLOYEE_MAP.put(department, manager);
 }
 return manager;
 }
}

package top.fjy8018.designpattern.pattern.structural.flyweight;

import lombok.extern.slf4j.Slf4j;

/**
 * @author F嘉阳
 * @date 2018-10-14 16:16
 */
@Slf4j
public class Manager implements Employee {

 private String content;

 private String department;

 /**
 * 内部状态
 */
 private String title = "部门经理";

 public Manager(String department) {
 this.department = department;
 }

 public void setContent(String content) {
 this.content = content;
 }

 @Override
 public void report() {
 log.info("{}{}，汇报内容：{}", department, title, content);
 }
}

测试

package top.fjy8018.designpattern.pattern.structural.flyweight;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

@Slf4j
class EmployeeTest {

 private static final String departments[] = {"RD", "QA", "PM", "BD"};

 @Test
 void report() {
 for (int i = 0; i < 10; i++) {
 String department = departments[(int) (Math.random() * departments.length)];
 Employee employee = EmployeeFactory.getEmployee(department);
 employee.report();
 }
 }
}

输出

15:47:51.177 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建PM部门经理
15:47:51.187 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建报告，内容为:报告内容……
15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - PM部门经理，汇报内容：报告内容……
15:47:51.188 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建BD部门经理
15:47:51.188 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建报告，内容为:报告内容……
15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - BD部门经理，汇报内容：报告内容……
15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - BD部门经理，汇报内容：报告内容……
15:47:51.188 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建RD部门经理
15:47:51.188 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建报告，内容为:报告内容……
15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - RD部门经理，汇报内容：报告内容……
15:47:51.188 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建QA部门经理
15:47:51.188 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建报告，内容为:报告内容……
15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - QA部门经理，汇报内容：报告内容……
15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - RD部门经理，汇报内容：报告内容……
15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - QA部门经理，汇报内容：报告内容……
15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - BD部门经理，汇报内容：报告内容……
15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - RD部门经理，汇报内容：报告内容……
15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - BD部门经理，汇报内容：报告内容……

由此可见，在初次创建部门经理后都无需重新创建，后续的汇报可以直接从对象池里直接获取s

组合模式

介绍

定义：将对象组合成树形结构以表示“部分-整体”的层次结构

组合模式使客户端对单个对象和组合对象保持一致的方式处理

类型：结构型

适用场景

希望客户端可以忽略组合对象与单个对象的差异时
处理一个树形结构时

优点

清楚地定义分层次的复杂对象,表示对象的全部或部分层次
让客户端忽略了层次的差异,方便对整个层次结构进行控制
简化客户端代码

缺点

使设计变得更加抽象
限制类型时会较为复杂

相关设计模式

组合模式和访问者模式

代码样例

类图

课程抽象类

package top.fjy8018.designpattern.pattern.structural.composite;

import java.awt.*;
import java.util.Collection;
import java.util.Map;

/**
 * 课程抽象类
 * 抽象组件应当多加考虑，哪些参数需要，哪些不需要
 *
 * JDK组合模式源码
 * {@link java.awt.Container#add(Component)} 增加一个父类操作
 * {@link java.util.HashMap#putAll(Map)}
 * {@link java.util.ArrayList#addAll(Collection)}
 *
 * Mybatis源码
 * {@link org.apache.ibatis.scripting.xmltags.SqlNode} 公共接口
 * 实现：{@link org.apache.ibatis.scripting.xmltags.MixedSqlNode}
 *
 * @author F嘉阳
 * @date 2020/2/28 16:19
 */
public abstract class CatalogComponent {

 public void add(CatalogComponent catalogComponent) {
 throw new UnsupportedOperationException("不支持添加操作");
 }

 public void remove(CatalogComponent catalogComponent) {
 throw new UnsupportedOperationException("不支持删除操作");
 }

 public String getName(CatalogComponent catalogComponent) {
 throw new UnsupportedOperationException("不支持获取名称操作");
 }

 public Double getPrice(CatalogComponent catalogComponent) {
 throw new UnsupportedOperationException("不支持获取价格操作");
 }

 public void print() {
 throw new UnsupportedOperationException("不支持获取打印操作");
 }
}

课程

package top.fjy8018.designpattern.pattern.structural.composite;

import lombok.AllArgsConstructor;
import lombok.extern.slf4j.Slf4j;

/**
 * 课程
 * 重写抽象类需要充分考虑业务逻辑
 *
 * @author F嘉阳
 * @date 2020/2/28 16:24
 */
@Slf4j
@AllArgsConstructor
public class Course extends CatalogComponent {

 private String name;

 private Double price;

 @Override
 public String getName(CatalogComponent catalogComponent) {
 return this.name;
 }

 @Override
 public Double getPrice(CatalogComponent catalogComponent) {
 return this.price;
 }

 @Override
 public void print() {
 log.info("课程名称：{}，价格：{}",name,price);
 }
}

课程目录

package top.fjy8018.designpattern.pattern.structural.composite;

import lombok.extern.slf4j.Slf4j;

import java.util.ArrayList;
import java.util.List;

/**
 * 课程目录
 * 可以添加目录节点
 *
 * @author F嘉阳
 * @date 2020/2/28 16:34
 */
@Slf4j
public class CourseCatalog extends CatalogComponent{

 private List<CatalogComponent> items = new ArrayList<>();

 private String name;

 public CourseCatalog(String name) {
 this.name = name;
 }

 @Override
 public void add(CatalogComponent catalogComponent) {
 items.add(catalogComponent);
 }

 @Override
 public void remove(CatalogComponent catalogComponent) {
 items.remove(catalogComponent);
 }

 @Override
 public void print() {
 log.info("课程目录名称：{}",name);
 items.forEach(CatalogComponent::print);
 }
}

测试

package top.fjy8018.designpattern.pattern.structural.composite;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/2/28 16:37
 */
@Slf4j
class CatalogComponentTest {

 @Test
 void add() {
 // 将课程与课程目录都认为是同一个类型，进行统一处理
 CatalogComponent linuxCourse = new Course("Linux课程",35.0);
 CatalogComponent windowsCourse = new Course("windows课程",61.0);
 CatalogComponent macOsCourse = new Course("macOs课程",20.0);

 CatalogComponent springCourse = new Course("spring课程",55.0);
 CatalogComponent servletCourse = new Course("servlet课程",11.0);

 // 课程目录
 CatalogComponent javaCourseCatalog = new CourseCatalog("java课程目录");
 CatalogComponent schoolCourseCatalog = new CourseCatalog("学校课程主目录");

 javaCourseCatalog.add(springCourse);
 javaCourseCatalog.add(servletCourse);

 // 既可以添加课程，也可以添加目录
 schoolCourseCatalog.add(linuxCourse);
 schoolCourseCatalog.add(windowsCourse);
 schoolCourseCatalog.add(macOsCourse);
 schoolCourseCatalog.add(javaCourseCatalog);

 schoolCourseCatalog.print();
 }
}

输出

09:24:29.687 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.CourseCatalog - 课程目录名称：学校课程主目录
09:24:29.691 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.Course - 课程名称：Linux课程，价格：35.0
09:24:29.693 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.Course - 课程名称：windows课程，价格：61.0
09:24:29.694 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.Course - 课程名称：macOs课程，价格：20.0
09:24:29.694 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.CourseCatalog - 课程目录名称：java课程目录
09:24:29.694 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.Course - 课程名称：spring课程，价格：55.0
09:24:29.694 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.Course - 课程名称：servlet课程，价格：11.0

桥接模式

介绍

定义：将抽象部分与它的具体实现部分分离,使它们都可以独立地变化

通过组合的方式建立两个类之间联系,而不是继承

类型：结构型

适用场景

抽象和具体实现之间增加更多的灵活性
一个类存在两个(或多个)独立变化的维度,且这两个(或多个) 维度都需要独立进行扩展
不希望使用继承,或因为多层继承导致系统类的个数剧增

优点

分离抽象部分及其具体实现部分
提高了系统的可扩展性
符合开闭原则
符合合成复用原则

缺点

增加了系统的理解与设计难度
需要正确地识别出系统中两个独立变化的维度

相关设计模式

桥接模式和组合模式
桥接模式和适配器模式

代码样例

类图

桥接模式核心在于将抽象与实现通过组合模式连接，使抽象和实现都可以自由扩展

银行账号——行为接口

package top.fjy8018.designpattern.pattern.structural.bridge;

import java.sql.Driver;
import java.sql.DriverManager;

/**
 * 银行账号——行为接口
 *
 * JDBC 为操作不同的数据库提供了同一个{@link java.sql.Connection} 连接
 * 由MySQL或Oracle连接驱动实现相关接口操作数据库
 *
 * {@link java.sql.Driver}
 * {@link DriverManager} 中的 {@link java.sql.DriverInfo}
 * 实现对 {@link java.sql.Driver} 的具体封装
 *
 * JDBC 实现的 {@link com.mysql.cj.jdbc.Driver} 初始化时调 {@link DriverManager#registerDriver(Driver)}
 *
 * @author F嘉阳
 * @date 2020/2/29 9:55
 */
public interface Account {

 /**
 * 打开一个账号
 *
 * @return
 */
 Account openAccount();

 /**
 * 查看账号类型
 */
 void showAccountType();
}

抽象银行

package top.fjy8018.designpattern.pattern.structural.bridge;

import lombok.AllArgsConstructor;

/**
 * 抽象银行
 * 组合账户接口，交由子类实现具体的行为
 *
 * @author F嘉阳
 * @date 2020/2/29 10:05
 */
@AllArgsConstructor
public abstract class Bank {

 protected Account account;

 /**
 * 打开一个账户
 * 该方法定义与接口相同，方法名称可以不同
 * 当前抽象类的openAccount方法需要委托给Account接口openAccount方法实现
 * 当前类为抽象类，委托给接口的实现类执行，体现抽象与实现分离
 * @return
 */
 abstract Account openAccount();
}

活期账户

package top.fjy8018.designpattern.pattern.structural.bridge;

import lombok.extern.slf4j.Slf4j;

/**
 * 活期账户
 *
 * @author F嘉阳
 * @date 2020/2/29 9:58
 */
@Slf4j
public class SavingAccount implements Account {
 /**
 * 打开一个账号
 *
 * @return
 */
 @Override
 public Account openAccount() {
 log.info("打开活期账户");
 return new SavingAccount();
 }

 /**
 * 查看账号类型
 */
 @Override
 public void showAccountType() {
 log.info("这是一个活期账户");
 }
}

定期账户

package top.fjy8018.designpattern.pattern.structural.bridge;

import lombok.extern.slf4j.Slf4j;

/**
 * 定期账户
 *
 * @author F嘉阳
 * @date 2020/2/29 9:57
 */
@Slf4j
public class DepositAccount implements Account{
 /**
 * 打开一个账号
 *
 * @return
 */
 @Override
 public Account openAccount() {
 log.info("打开定期账户");
 return new DepositAccount();
 }

 /**
 * 查看账号类型
 */
 @Override
 public void showAccountType() {
 log.info("这是一个定期账户");
 }
}

广州银行实现

package top.fjy8018.designpattern.pattern.structural.bridge;

import lombok.extern.slf4j.Slf4j;

/**
 * @author F嘉阳
 * @date 2020/2/29 10:09
 */
@Slf4j
public class GCBBank extends Bank {

 public GCBBank(Account account) {
 super(account);
 }

 /**
 * 打开一个账户
 * 该方法定义与接口相同，方法名称可以不同
 * 当前抽象类的openAccount方法需要委托给Account接口openAccount方法实现
 * 当前类为抽象类，委托给接口的实现类执行，体现抽象与实现分离
 *
 * @return
 */
 @Override
 Account openAccount() {
 log.info("打开广州银行账户");
 // 委托
 account.openAccount();
 return account;
 }
}

工商银行实现

package top.fjy8018.designpattern.pattern.structural.bridge;

import lombok.extern.slf4j.Slf4j;

/**
 * @author F嘉阳
 * @date 2020/2/29 10:11
 */
@Slf4j
public class ICBCBank extends Bank {

 public ICBCBank(Account account) {
 super(account);
 }

 /**
 * 打开一个账户
 * 该方法定义与接口相同，方法名称可以不同
 * 当前抽象类的openAccount方法需要委托给Account接口openAccount方法实现
 * 当前类为抽象类，委托给接口的实现类执行，体现抽象与实现分离
 *
 * @return
 */
 @Override
 Account openAccount() {
 log.info("打开工商银行账户");
 // 委托
 account.openAccount();
 return account;
 }
}

测试

package top.fjy8018.designpattern.pattern.structural.bridge;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/2/29 14:30
 */
@Slf4j
class AccountTest {

 @Test
 void openAccount() {
 Bank gcbBank = new GCBBank(new DepositAccount());
 Account gcbAccount = gcbBank.openAccount();
 gcbAccount.showAccountType();

 Bank icbcBank = new ICBCBank(new SavingAccount());
 Account icbcAccount = icbcBank.openAccount();
 icbcAccount.showAccountType();
 }
}

输出

14:36:13.478 [main] INFO top.fjy8018.designpattern.pattern.structural.bridge.GCBBank - 打开广州银行账户
14:36:13.488 [main] INFO top.fjy8018.designpattern.pattern.structural.bridge.DepositAccount - 打开定期账户
14:36:13.489 [main] INFO top.fjy8018.designpattern.pattern.structural.bridge.DepositAccount - 这是一个定期账户
14:36:13.489 [main] INFO top.fjy8018.designpattern.pattern.structural.bridge.ICBCBank - 打开工商银行账户
14:36:13.489 [main] INFO top.fjy8018.designpattern.pattern.structural.bridge.SavingAccount - 打开活期账户
14:36:13.489 [main] INFO top.fjy8018.designpattern.pattern.structural.bridge.SavingAccount - 这是一个活期账户

代理模式

介绍

定义：为其他对象提供一种代理,以控制对这个对象的访问

代理对象在客户端和目标对象之间起到中介的作用

类型：结构型

适用场景

保护目标对象
增强目标对象

优点

代理模式能将代理对象与真实被调用的目标对象分离
一定程度上降低了系统的耦合度，扩展性好
保护目标对象
增强目标对象

缺点

代理模式会造成系统设计中类的数目增加
在客户端和目标对象增加一个代理对象，会造成请求处理速度变慢
增加系统的复杂度

分类与扩展

静态代理
- 显式在业务实现类新增代理类实现代理功能，在代理类中对同名的业务方法进行增强和包装
CGLib代理
- 可以代理类
- 代理类对象时，CGLib会生成代理类的子类，覆盖对应的方法
- 由于利用了继承和重写的特性，若类或者方法被标识为final，则无法进行代理
动态代理
- 无法代理类，可以代理接口
- 若增强的对象新增了一个方法，而接口中不包含该新增方法，则无法进行增强
- 对类的代理是在使用时动态创建了一个代理类的class文件，该class文件被字节码引擎执行，被代理类进行方法调用

Spring代理选择

当Bean有实现接口时, Spring就会用JDK的动态代理
当Bean没有实现接口时, Spring使用CGLib
可以强制使用 Cglib
- 在Spring配置中加入<aop: aspectj- autoproxy proxy-target-dass="true"/>
参考资料： https://docs.spring.io/spring-framework/docs/current/spring-framework-reference/core.html

代理速度对比

CGLib：基于ASM字节码生成，比反射效率高
JDK动态代理
速度对比：万次执行的情况下，JDK7与JDK8的动态代理性能比CGLib速度快20%左右

相关设计模式

代理模式和装饰者模式
代理模式和适配器模式

代码样例——Spring 分库代理实现

自定义数据源上下文

package top.fjy8018.designpattern.pattern.structural.proxy.db;

/**
 * 数据源上下文
 *
 * @author F嘉阳
 * @date 2020/2/29 16:27
 */
public class DataSourceContextHolder {
 /**
 * 存放数据源Bean Name
 */
 private static final ThreadLocal<String> CONTEXT_HOLDER = new ThreadLocal<>();

 /**
 * 配置哪个数据库，则动态数据源线程就会获取到哪个数据库
 * @param dbType
 */
 public static void setDBType(String dbType){
 CONTEXT_HOLDER.set(dbType);
 }

 public static String getDBType(){
 return CONTEXT_HOLDER.get();
 }

 /**
 * 清除，防止内存泄漏
 */
 public static void clear(){
 CONTEXT_HOLDER.remove();
 }
}

定义动态数据源

package top.fjy8018.designpattern.pattern.structural.proxy.db;

import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;

/**
 * Spring分库实现
 *
 * @author F嘉阳
 * @date 2020/2/29 16:26
 */
public class DynamicDataSource extends AbstractRoutingDataSource {

 /**
 * 动态获取数据库
 * @return 当前线程所属数据源
 */
 @Override
 protected Object determineCurrentLookupKey() {
 return DataSourceContextHolder.getDBType();
 }
}

多数据源配置

<bean id="dataSource" class="top.fjy8018.designpattern.pattern.structural.proxy.db.DynamicDataSource">
	<property name="targetDataSources">
		<map key-type="java.lang.String">
			<entry value-ref="db0" key="db0"></entry>
			<entry value-ref="db1" key="db1"></entry>
		</map>
	</property>
	<property name="defaultTargetDataSource" ref="db0"></property>
</bean>


<bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
	<property name="dataSource" ref="dataSource" />
</bean>

代码样例——静态代理

类图

订单

package top.fjy8018.designpattern.pattern.structural.proxy;

import lombok.Getter;
import lombok.Setter;

/**
 * 订单
 *
 * @author F嘉阳
 * @date 2020/2/29 16:12
 */
@Getter
@Setter
public class Order {

 private Object orderInfo;

 private Integer userId;
}

DAO层

package top.fjy8018.designpattern.pattern.structural.proxy;

/**
 * DAO层——与业务无关
 * 仿mybatis
 *
 * @author F嘉阳
 * @date 2020/2/29 16:15
 */
public interface IOrderDao {

 int insert(Order order);
}

package top.fjy8018.designpattern.pattern.structural.proxy;

import lombok.extern.slf4j.Slf4j;

/**
 * @author F嘉阳
 * @date 2020/2/29 16:17
 */
@Slf4j
public class IOrderDaoImpl implements IOrderDao {
 @Override
 public int insert(Order order) {
 log.info("DAO层添加订单记录成功");
 return 1;
 }
}

订单服务层

package top.fjy8018.designpattern.pattern.structural.proxy;

/**
 * 订单服务层
 *
 * @author F嘉阳
 * @date 2020/2/29 16:14
 */
public interface IOrderService {

 /**
 * 保存订单
 * @param order
 * @return 保存的行数
 */
 int save(Order order);
}

package top.fjy8018.designpattern.pattern.structural.proxy;

import lombok.extern.slf4j.Slf4j;

/**
 * @author F嘉阳
 * @date 2020/2/29 16:16
 */
@Slf4j
public class IOrderServiceImpl implements IOrderService {

 /**
 * 模拟Spring注入
 */
 private IOrderDao orderDao = new IOrderDaoImpl();

 /**
 * 保存订单
 *
 * @param order
 * @return 保存的行数
 */
 @Override
 public int save(Order order) {
 log.info("Service层调用DAO层添加订单记录成功");
 return orderDao.insert(order);
 }
}

静态代理类

package top.fjy8018.designpattern.pattern.structural.proxy.staticproxy;

import lombok.extern.slf4j.Slf4j;
import top.fjy8018.designpattern.pattern.structural.proxy.IOrderService;
import top.fjy8018.designpattern.pattern.structural.proxy.IOrderServiceImpl;
import top.fjy8018.designpattern.pattern.structural.proxy.Order;
import top.fjy8018.designpattern.pattern.structural.proxy.db.DataSourceContextHolder;

/**
 * 静态代理实现分库
 *
 * @author F嘉阳
 * @date 2020/2/29 16:20
 */
@Slf4j
public class OrderServiceStaticProxy {

 /**
 * 模拟Spring注入
 */
 private IOrderService orderService = new IOrderServiceImpl();

 private static final String DB = "db";

 /**
 * 对源方法增强
 *
 * @param order
 * @return
 */
 public int save(Order order) {
 beforeMethod();

 // 通过用户ID对2取模分库
 Integer userId = order.getUserId();
 int dbRouter = userId % 2;
 log.info("静态代理分配到【db{}】处理数据", dbRouter);

 // 设置DataSource
 DataSourceContextHolder.setDBType(DB + String.valueOf(dbRouter));

 afterMethod();

 return orderService.save(order);
 }

 private void beforeMethod() {
 log.info("静态代理before code");
 }

 private void afterMethod() {
 log.info("静态代理after code");
 }
}

测试

package top.fjy8018.designpattern.pattern.structural.proxy.staticproxy;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import top.fjy8018.designpattern.pattern.structural.proxy.Order;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/2/29 17:17
 */
@Slf4j
class OrderServiceStaticProxyTest {

 @Test
 void save() {
 Order order1 = new Order();
 order1.setUserId(1);
 Order order2 = new Order();
 order2.setUserId(2);

 OrderServiceStaticProxy proxy = new OrderServiceStaticProxy();
 proxy.save(order1);
 proxy.save(order2);
 }
}

输出


17:20:36.012 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.staticproxy.OrderServiceStaticProxy - 静态代理before code
17:20:36.022 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.staticproxy.OrderServiceStaticProxy - 静态代理分配到【db1】处理数据
17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.staticproxy.OrderServiceStaticProxy - 静态代理after code
17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderServiceImpl - Service层调用DAO层添加订单记录成功
17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderDaoImpl - DAO层添加订单记录成功
17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.staticproxy.OrderServiceStaticProxy - 静态代理before code
17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.staticproxy.OrderServiceStaticProxy - 静态代理分配到【db0】处理数据
17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.staticproxy.OrderServiceStaticProxy - 静态代理after code
17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderServiceImpl - Service层调用DAO层添加订单记录成功
17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderDaoImpl - DAO层添加订单记录成功

代码样例——动态代理

类图

动态代理类

package top.fjy8018.designpattern.pattern.structural.proxy.dynamic;

import lombok.extern.slf4j.Slf4j;
import org.apache.ibatis.binding.MapperProxy;
import org.springframework.aop.framework.ProxyFactoryBean;
import top.fjy8018.designpattern.pattern.structural.proxy.Order;
import top.fjy8018.designpattern.pattern.structural.proxy.db.DataSourceContextHolder;

import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;

/**
 * 动态代理实现分库
 * 动态代理适用场景更加广泛，可以实现通用分库逻辑
 * <p>
 * {@link ProxyFactoryBean#getObject()} 代理工厂类
 * {@link org.springframework.aop.framework.JdkDynamicAopProxy} spring AOP实现，对动态代理封装
 * {@link org.springframework.aop.framework.CglibAopProxy} Cglib 对类进行代理增强
 * {@link org.apache.ibatis.binding.MapperProxyFactory#newInstance(MapperProxy)} 实现mapper代理
 * 实际封装在 {@link MapperProxy#invoke(Object, Method, Object[])}
 * 其获取的方法利用了享元模式 {@link MapperProxy#cachedMapperMethod(Method)}
 *
 * @author F嘉阳
 * @date 2020/3/3 11:00
 */
@Slf4j
public class OrderServiceDynamicProxy implements InvocationHandler {

 private Object target;

 private static final String DB = "db";

 public OrderServiceDynamicProxy(Object target) {
 this.target = target;
 }

 /**
 * 绑定目标对象
 *
 * @return
 */
 public Object bind() {
 Class clazz = target.getClass();
 return Proxy.newProxyInstance(clazz.getClassLoader(), clazz.getInterfaces(), this);
 }

 /**
 * 代理
 *
 * @param proxy 代理对象，一般不调用该参数
 * @param method 要被增强的方法对象
 * @param args 方法参数
 * @return
 * @throws Throwable
 */
 @Override
 public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
 // 参数传递
 Object argObject = args[0];
 beforeMethod(argObject);
 // object是要被增强的方法的返回值
 Object object = method.invoke(target, args);
 afterMethod();
 return object;
 }

 /**
 * 前置增强方法
 *
 * @param args 方法参数
 */
 private void beforeMethod(Object args) {
 int userId = 0;
 // 对参数取模
 log.info("动态代理，before code");
 if (args instanceof Order) {
 Order order = (Order) args;
 userId = order.getUserId();
 }
 int dbRouter = userId % 2;
 log.info("动态代理分配到【db{}】处理数据", dbRouter);

 // 设置DataSource
 DataSourceContextHolder.setDBType(DB + dbRouter);
 }

 private void afterMethod() {
 log.info("动态代理after code");
 }
}

测试类

package top.fjy8018.designpattern.pattern.structural.proxy.dynamic;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import top.fjy8018.designpattern.pattern.structural.proxy.IOrderService;
import top.fjy8018.designpattern.pattern.structural.proxy.IOrderServiceImpl;
import top.fjy8018.designpattern.pattern.structural.proxy.Order;

/**
 * @author F嘉阳
 * @date 2020/3/3 15:54
 */
@Slf4j
class OrderServiceDynamicProxyTest {

 @Test
 void invoke() {
 Order order1 = new Order();
 order1.setUserId(1);
 Order order2 = new Order();
 order2.setUserId(2);

 // 动态代理直接代理接口
 IOrderService proxy = (IOrderService) new OrderServiceDynamicProxy(new IOrderServiceImpl()).bind();
 proxy.save(order1);
 proxy.save(order2);
 }
}

输出

15:57:57.234 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.dynamic.OrderServiceDynamicProxy - 动态代理，before code
15:57:57.238 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.dynamic.OrderServiceDynamicProxy - 动态代理分配到【db1】处理数据
15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderServiceImpl - Service层调用DAO层添加订单记录成功
15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderDaoImpl - DAO层添加订单记录成功
15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.dynamic.OrderServiceDynamicProxy - 动态代理after code
15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.dynamic.OrderServiceDynamicProxy - 动态代理，before code
15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.dynamic.OrderServiceDynamicProxy - 动态代理分配到【db0】处理数据
15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderServiceImpl - Service层调用DAO层添加订单记录成功
15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderDaoImpl - DAO层添加订单记录成功
15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.dynamic.OrderServiceDynamicProxy - 动态代理after code

动态代理源码分析

Proxy类

package java.lang.reflect;

public class Proxy implements java.io.Serializable {
 // 反射调用开始
	@CallerSensitive
 public static Object newProxyInstance(ClassLoader loader,
 Class<?>[] interfaces,
 InvocationHandler h)
 throws IllegalArgumentException
 {
 Objects.requireNonNull(h);

 final Class<?>[] intfs = interfaces.clone();
 final SecurityManager sm = System.getSecurityManager();
 if (sm != null) {
 // 合法性检查
 checkProxyAccess(Reflection.getCallerClass(), loader, intfs);
 }

 /*
 * 动态生成代理类
 */
 Class<?> cl = getProxyClass0(loader, intfs);

 /*
 * 实例化并返回新创建的类实例对象
 */
 try {
 if (sm != null) {
 checkNewProxyPermission(Reflection.getCallerClass(), cl);
 }

 final Constructor<?> cons = cl.getConstructor(constructorParams);
 final InvocationHandler ih = h;
 if (!Modifier.isPublic(cl.getModifiers())) {
 AccessController.doPrivileged(new PrivilegedAction<Void>() {
 public Void run() {
 cons.setAccessible(true);
 return null;
 }
 });
 }
 return cons.newInstance(new Object[]{h});
 } catch (IllegalAccessException|InstantiationException e) {
 throw new InternalError(e.toString(), e);
 } catch (InvocationTargetException e) {
 Throwable t = e.getCause();
 if (t instanceof RuntimeException) {
 throw (RuntimeException) t;
 } else {
 throw new InternalError(t.toString(), t);
 }
 } catch (NoSuchMethodException e) {
 throw new InternalError(e.toString(), e);
 }
 }
 
 private static void checkProxyAccess(Class<?> caller,
 ClassLoader loader,
 Class<?>... interfaces)
 {
 // 安全校验
 SecurityManager sm = System.getSecurityManager();
 if (sm != null) {
 ClassLoader ccl = caller.getClassLoader();
 if (VM.isSystemDomainLoader(loader) && !VM.isSystemDomainLoader(ccl)) {
 sm.checkPermission(SecurityConstants.GET_CLASSLOADER_PERMISSION);
 }
 ReflectUtil.checkProxyPackageAccess(ccl, interfaces);
 }
 }
 
 // 生成代理类
 private static Class<?> getProxyClass0(ClassLoader loader,
 Class<?>... interfaces) {
 // 若一个类中的接口数量超过限制则报错
 if (interfaces.length > 65535) {
 throw new IllegalArgumentException("interface limit exceeded");
 }

 // 从代理类缓存中读取
 return proxyClassCache.get(loader, interfaces);
 }
}

从代理类缓存中读取实现

package java.lang.reflect;

final class WeakCache<K, P, V> {
 	public V get(K key, P parameter) {
 Objects.requireNonNull(parameter);

 expungeStaleEntries();

 Object cacheKey = CacheKey.valueOf(key, refQueue);

 // 使用并发包
 ConcurrentMap<Object, Supplier<V>> valuesMap = map.get(cacheKey);
 if (valuesMap == null) {
 ConcurrentMap<Object, Supplier<V>> oldValuesMap
 = map.putIfAbsent(cacheKey,
 valuesMap = new ConcurrentHashMap<>());
 if (oldValuesMap != null) {
 valuesMap = oldValuesMap;
 }
 }

 Object subKey = Objects.requireNonNull(subKeyFactory.apply(key, parameter));
 Supplier<V> supplier = valuesMap.get(subKey);
 Factory factory = null;

 while (true) {
 if (supplier != null) {
 // 循环获取，直到获取成功才返回
 V value = supplier.get();
 if (value != null) {
 return value;
 }
 }
 // 若首次创建则通过工厂类创建对应的类
 if (factory == null) {
 factory = new Factory(key, parameter, subKey, valuesMap);
 }

 if (supplier == null) {
 supplier = valuesMap.putIfAbsent(subKey, factory);
 if (supplier == null) {
 supplier = factory;
 }
 } else {
 if (valuesMap.replace(subKey, supplier, factory)) {
 supplier = factory;
 } else {
 supplier = valuesMap.get(subKey);
 }
 }
 }
 }
}

断点查看执行情况