Java on F嘉阳

ClassLoader类隔离

Wed, 13 Nov 2024 21:56:49 +0800

基本概念

参考oracle jdk 21中对ClassLoader定义

A class loader is an object that is responsible for loading classes. The class ClassLoader is an abstract class. Given the binary name of a class, a class loader should attempt to locate or generate data that constitutes a definition for the class. A typical strategy is to transform the name into a file name and then read a “class file” of that name from a file system.

在Java中，ClassLoader负责加载类，当给定一个类的二进制名称，ClassLoader会尝试定位或生成构成该类定义的数据。

层次结构

ClassLoader遵循双亲委派模型，当请求查找类或资源时， ClassLoader 实例通常会将类或资源的搜索委托给其父类加载器，然后再尝试查找类或资源本身

由于JDK9中模块系统（Jigsaw）的引入，jdk8和jdk9类加载器层次结构有区别，ExtClassLoader变成PlatformClassLoader负责加载--module-path（或MODPATH环境变量）指定的模块

graph TD
 Bootstrap[Bootstrap ClassLoader]
 Extension[Extension ClassLoader]
 Application[Application ClassLoader]

 Bootstrap --> Extension
 Extension --> Application

 subgraph "JDK 8"
 Bootstrap:::primary
 Extension:::secondary
 Application:::tertiary
 end

graph TD
 Bootstrap[Bootstrap ClassLoader]
 Platform[Platform ClassLoader]
 App[Application ClassLoader]

 Bootstrap --> Platform
 Platform --> App

 subgraph "JDK 9"
 Bootstrap:::primary
 Platform:::secondary
 App:::tertiary
 end

生命周期

类加载的过程可以分为几个阶段及常见异常

加载（Loading）
ClassNotFoundException（找不到目标类）
验证（Verification）
ClassFormatError（类文件格式不正确）
NoClassDefFoundError（已加载的类无法初始化）
准备（Preparation）
解析（Resolution）
LinkageError（类加载器无法找到或者无法链接到其他类或接口）
IncompatibleClassChangeError
AbstractMethodError
NoSuchMethodError
…
初始化（Initialization）

核心方法

ClassLoader类提供了几个关键的方法来支持类的加载过程，其中loadClass和findClass最常使用

loadClass
protected Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException
Loads the class with the specified binary name. The default implementation of this method searches for classes in the following order:

Invoke findLoadedClass(String) to check if the class has already been loaded.

Invoke the loadClass method on the parent class loader. If the parent is null the class loader built into the virtual machine is used, instead.

Invoke the findClass(String) method to find the class.

If the class was found using the above steps, and the resolve flag is true, this method will then invoke the resolveClass(Class) method on the resulting Class object.

Subclasses of ClassLoader are encouraged to override findClass(String), rather than this method.

Unless overridden, this method synchronizes on the result of getClassLoadingLock method during the entire class loading process.

Parameters:

name - The binary name of the class

resolve - If true then resolve the class

Returns:

The resulting Class object

Throws:

ClassNotFoundException - If the class could not be found

官方文档给出了方法设计介绍，loadClass用于实现双亲委派职责，当父类无法加载目标class时，调用子类的findClass加载。

当需要打破双亲委派模型时，则直接覆盖loadClass

findClass
protected Class<?> findClass(String name) throws ClassNotFoundException
Finds the class with the specified binary name. This method should be overridden by class loader implementations that follow the delegation model for loading classes, and will be invoked by the loadClass method after checking the parent class loader for the requested class.

Implementation Requirements:

The default implementation throws ClassNotFoundException.

Parameters:

name - The binary name of the class

Returns:

The resulting Class object

Throws:

ClassNotFoundException - If the class could not be found

Since:

1.2

在findClass用于实现自定义类加载逻辑

以常用的URLClassLoader为例，findClass将传入的类名转换成路径格式匹配在URLClassPath类路径下实际存在的文件结构，如果找到了对应的资源，则继续处理；否则返回null。

public class URLClassLoader extends SecureClassLoader implements Closeable {
 protected Class<?> findClass(final String name)
 throws ClassNotFoundException
 {
 final Class<?> result;
 try {
 result = AccessController.doPrivileged(
 new PrivilegedExceptionAction<Class<?>>() {
 public Class<?> run() throws ClassNotFoundException {
 String path = name.replace('.', '/').concat(".class");
 Resource res = ucp.getResource(path, false);
 if (res != null) {
 try {
 return defineClass(name, res);
 } catch (IOException e) {
 throw new ClassNotFoundException(name, e);
 }
 } else {
 return null;
 }
 }
 }, acc);
 } catch (java.security.PrivilegedActionException pae) {
 throw (ClassNotFoundException) pae.getException();
 }
 if (result == null) {
 throw new ClassNotFoundException(name);
 }
 return result;
 }
}

应用

tomcat

为了支持应用隔离和模块化部署，tomcat扩展了已有的双亲委派模型

以tomcat9为例，其层次结构如下

graph TD
 Bootstrap[Bootstrap ClassLoader]
 System[System ClassLoader]
 Common[Common ClassLoader]
 Webapp1[Webapp1 ClassLoader]
 Webapp2[Webapp2 ClassLoader]
 Webapp3[Webapp3 ClassLoader]

 Bootstrap --> System
 System --> Common
 Common --> Webapp1
 Common --> Webapp2
 Common --> Webapp3

Bootstrap（所有应用可见，JVM核心依赖）
$JAVA_HOME/jre/lib/ext
System（所有应用可见，tomcat核心依赖）
$CATALINA_HOME/bin
Common（需要对所有应用可见的扩展类）
Webapp（每个应用隔离，打破双亲委派，本地优先）

As mentioned above, the web application class loader diverges from the default Java delegation model (in accordance with the recommendations in the Servlet Specification, version 2.4, section 9.7.2 Web Application Classloader). When a request to load a class from the web application’s WebappX class loader is processed, this class loader will look in the local repositories first, instead of delegating before looking.

重点在于webapp classloader，其打破双亲委派机制，当从 Web 应用程序的 WebappX 类加载器加载类的请求被处理时，这个类加载器会首先查看本地存储库，而不是先进行委托。

在应用程序视角，类查找顺序如下

Bootstrap ClassLoader
WebApp ClassLoader
/WEB-INF/classes
/WEB-INF/lib/*.jar
System ClassLoader
Common ClassLoader

当应用配置<Loader delegate="true"/>时，可切换为标准双亲委派模型

OSGi

OSGi是一个 Java 模块化系统和服务平台，它提供了一种动态管理模块（Bundle）的方法，支持不停止应用程序的情况下动态更新、添加或删除模块。

以Apache Felix实现的OSGi为例，其查找逻辑如下

循环检测：防止在查找过程中出现无限递归。
委托给父类加载器：如果目标类或资源属于需要委托给父类加载器处理的包，则先尝试使用父类加载器进行查找。
从导入列表中查找：在当前修订版本的导入列表中搜索类或资源。
从本地类路径查找：如果前面步骤未能找到，则尝试从当前修订版本自己的类路径中查找。
从动态导入中查找：最后，如果还没有找到，则会尝试从动态导入中查找。

private Object findClassOrResourceByDelegation(String name, boolean isClass)
 throws ClassNotFoundException, ResourceNotFoundException
 {
 // 初始化结果为null，表示尚未找到类或资源
 Object result = null;

 // 用于循环检测的集合，避免在查找过程中出现无限递归的情况
 Set requestSet = (Set) m_cycleCheck.get();
 if (requestSet == null)
 {
 // 如果当前线程中没有设置循环检测集合，则创建一个新的并设置到当前线程
 requestSet = new HashSet();
 m_cycleCheck.set(requestSet);
 }
 // 将要查找的名字添加到循环检测集合中，如果已经存在说明出现了循环依赖
 if (requestSet.add(name))
 {
 try
 {
 // 根据是查找类还是资源来获取目标的包名
 String pkgName = (isClass)
 ? Util.getClassPackage(name)
 : Util.getResourcePackage(name);

 // 检查是否应该将该包委托给父类加载器
 if (shouldBootDelegate(pkgName))
 {
 try
 {
 // 获取适当的类加载器进行委托
 ClassLoader bdcl = getBootDelegationClassLoader();
 // 根据是要查找类还是资源，调用相应的方法
 result = (isClass)
 ? (Object) bdcl.loadClass(name)
 : (Object) bdcl.getResource(name);
 // 如果是java.*包下的内容或者已经找到了结果，则直接返回
 if (pkgName.startsWith("java.") || (result != null))
 {
 return result;
 }
 }
 catch (ClassNotFoundException ex)
 {
 // 如果是java.*包下的类且未找到，则抛出异常
 if (pkgName.startsWith("java."))
 {
 throw ex;
 }
 }
 }

 // 在导入列表中查找
 result = searchImports(pkgName, name, isClass);

 // 如果在导入列表中未找到，则尝试从修订版本自己的类路径中查找
 if (result == null)
 {
 if (isClass)
 {
 // 获取内部类加载器，并尝试加载类
 ClassLoader cl = getClassLoaderInternal();
 if (cl == null)
 {
 // 如果类加载器无效，则抛出异常
 throw new ClassNotFoundException(
 "Unable to load class '"
 + name
 + "' because the bundle wiring for "
 + m_revision.getSymbolicName()
 + " is no longer valid.");
 }
 result = ((BundleClassLoader) cl).findClass(name);
 }
 else
 {
 // 如果不是查找类而是资源，则使用修订版本的方法来查找资源
 result = m_revision.getResourceLocal(name);
 }

 // 如果还是没找到，则尝试从动态导入中查找
 if (result == null)
 {
 result = searchDynamicImports(pkgName, name, isClass);
 }
 }
 }
 finally
 {
 // 清理：从循环检测集合中移除当前查找的名字
 requestSet.remove(name);
 }
 }
 else
 {
 // 如果检测到循环依赖，则返回null以打破循环
 return null;
 }

 // 如果最终结果仍为null，根据查找类型抛出相应的异常
 if (result == null)
 {
 if (isClass)
 {
 throw new ClassNotFoundException(
 name + " not found by " + this.getBundle());
 }
 else
 {
 throw new ResourceNotFoundException(
 name + " not found by " + this.getBundle());
 }
 }

 // 返回最终找到的结果
 return result;
 }

SofaArk

SofaArk是蚂蚁集团开源轻量类隔离框架

其定义3个概念：Ark Container，Ark Plugin，Ark Biz。

sofaark通过maven插件在打包时将插件启动器、导出类、导入类、导出资源、导入资源、优先级等元信息保存在 Ark Plugin 中的 META-INF/MANIFEST.MF 中，每个ark plugin都有独立的类加载器读取插件元信息和类资源导入导出关系。插件之间可以互相委托加载类。
ark biz只能加载自己模块的类信息，也可以委托plugin加载所需的类。

graph TD
 Bootstrap[BootstrapClassLoader]
 Extension[ExtensionClassLoader]
 Application[ApplicationClassLoader]
 URLClassLoader[URLClassLoader]
 Container[ContainerClassLoader]
 Agent[AgentClassLoader]
		Biz[BizClassLoader]
		Plugin[PluginClassLoader]

 Bootstrap --> Extension
 Extension --> Application
 Application --> URLClassLoader
	 URLClassLoader --> Container
		URLClassLoader --> Biz
		URLClassLoader --> Plugin
		URLClassLoader --> Agent

源码

AbstractClasspathClassLoader提供了类加载查找模式的抽象实现，loadClass会从以下顺序查找

本地类加载缓存
子类实现的loadClassInternal方法

BizClassLoader实现如下

package com.alipay.sofa.ark.container.service.classloader;

public class BizClassLoader extends AbstractClasspathClassLoader {

 
 @Override
 protected Class<?> loadClassInternal(String name, boolean resolve) throws ArkLoaderException {
 Class<?> clazz = null;

 // 0. sun reflect related class throw exception directly
 if (classloaderService.isSunReflectClass(name)) {
 throw new ArkLoaderException(
 String
 .format(
 "[ArkBiz Loader] %s : can not load class: %s, this class can only be loaded by sun.reflect.DelegatingClassLoader",
 bizIdentity, name));
 }

 // 1. findLoadedClass
 if (clazz == null) {
 clazz = findLoadedClass(name);
 }

 // 2. JDK related class
 if (clazz == null) {
 clazz = resolveJDKClass(name);
 }

 // 3. Ark Spi class
 if (clazz == null) {
 clazz = resolveArkClass(name);
 }

 // 4. pre find class
 if (clazz == null) {
 clazz = preLoadClass(name);
 }

 // 5. Plugin Export class
 if (clazz == null) {
 clazz = resolveExportClass(name);
 }

 // 6. Biz classpath class
 if (clazz == null) {
 clazz = resolveLocalClass(name);
 }

 // 7. Java Agent ClassLoader for agent problem
 if (clazz == null) {
 clazz = resolveJavaAgentClass(name);
 }

 // 8. post find class
 if (clazz == null) {
 clazz = postLoadClass(name);
 }

 if (clazz != null) {
 if (resolve) {
 super.resolveClass(clazz);
 }
 return clazz;
 }

 throw new ArkLoaderException(String.format("[ArkBiz Loader] %s : can not load class: %s",
 bizIdentity, name));
 }
}

PluginClassLoader实现如下

package com.alipay.sofa.ark.container.service.classloader;

public class PluginClassLoader extends AbstractClasspathClassLoader {

	@Override
 protected Class<?> loadClassInternal(String name, boolean resolve) throws ArkLoaderException {
 Class<?> clazz = null;

 // 0. sun reflect related class throw exception directly
 if (classloaderService.isSunReflectClass(name)) {
 throw new ArkLoaderException(
 String
 .format(
 "[ArkPlugin Loader] %s : can not load class: %s, this class can only be loaded by sun.reflect.DelegatingClassLoader",
 pluginName, name));
 }

 // 1. findLoadedClass
 if (clazz == null) {
 clazz = findLoadedClass(name);
 }

 // 2. JDK related class
 if (clazz == null) {
 clazz = resolveJDKClass(name);
 }

 // 3. Ark Spi class
 if (clazz == null) {
 clazz = resolveArkClass(name);
 }

 // 4. pre find class
 if (clazz == null) {
 clazz = preLoadClass(name);
 }

 // 5. Import class export by other plugins
 if (clazz == null) {
 clazz = resolveExportClass(name);
 }

 // 6. Plugin classpath class
 if (clazz == null) {
 clazz = resolveLocalClass(name);
 }

 // 7. Java Agent ClassLoader for agent problem
 if (clazz == null) {
 clazz = resolveJavaAgentClass(name);
 }

 // 8. Post find class
 if (clazz == null) {
 clazz = postLoadClass(name);
 }

 if (clazz != null) {
 if (resolve) {
 super.resolveClass(clazz);
 }
 return clazz;
 }

 throw new ArkLoaderException(String.format(
 "[ArkPlugin Loader] %s : can not load class: %s", pluginName, name));
 }
}

流程基本一致，主要在于plugin和biz读取到的import类元信息不同

参考资料

GraphQL实践10——Netflix Dgs Graphql异步订阅

Fri, 28 Oct 2022 21:59:23 +0800

介绍

GraphQL订阅模式支持服务端主动向客户端推送数据通知，避免客户端轮训

DGS Subscriptions

新增依赖

此处使用的是SpringBoot Web组件，故引入websockets依赖

如果使用的是WebFlux则要引入对应的dgs-webflux

		 <dependency>
 <groupId>com.netflix.graphql.dgs</groupId>
 <artifactId>graphql-dgs-subscriptions-websockets-autoconfigure</artifactId>
 </dependency>

异步通知解析

此处定义为当新增一个Actor就给所有订阅者发异步通知

@Slf4j
@DgsComponent
@RequiredArgsConstructor
public class ActorDataFetcher {

	private final ActorAssembler actorAssembler;

	private FluxSink<Actor> actorStream;

	private ConnectableFlux<Actor> actorPublisher;

	@PostConstruct
	private void createActorPublisher() {
		Flux<Actor> publisher = Flux.create(emitter -> {
			actorStream = emitter;
		});

		actorPublisher = publisher.publish();
		actorPublisher.connect();
	}

	@DgsMutation
	public Actor addActor(@InputArgument SubmitActor actor) {
		Actor actorEntity = actorAssembler.convert(actor);
		actorEntity.setActorId(10);
		actorEntity.setLastUpdate(DateUtil.format(new Date(), DatePattern.NORM_DATETIME_PATTERN));
		actorStream.next(actorEntity);
		log.info("服务端创建演员：{}", actor);
		return actorEntity;
	}

	/**
	 * 异步通知
	 *
	 * @return
	 */
	@DgsSubscription
	public Publisher<Actor> actorAdded() {
		return actorPublisher;
	}
}

单元测试

由于异步通知无法用浏览器自带的控制台测试，只能通过GraphQL客户端订阅后接收异步通知，此处参考官方最佳实践，使用单元测试模拟

@Slf4j
@SpringBootTest
class ActorDataFetcherTest {

	@Autowired
	private DgsQueryExecutor dgsQueryExecutor;

	@Test
	void addActor() {

		GraphQLQueryRequest graphQLQueryRequest = new GraphQLQueryRequest(
				AddActorGraphQLQuery.newRequest()
						.actor(
								SubmitActor.newBuilder()
										.firstName("fu")
										.lastName("jiayang")
										.build())
						.build(),
				new AddActorProjectionRoot()
						.actorId()
						.firstName()
						.lastUpdate());
		dgsQueryExecutor.execute(graphQLQueryRequest.serialize());
	}

	@Test
	void actorAdded() {
		ExecutionResult executionResult = dgsQueryExecutor.execute("""
				subscription {
				 actorAdded {
				 actorId
				 firstName
				 lastName
				 lastUpdate
				 }
				}
				""");
		Publisher<ExecutionResult> reviewPublisher = executionResult.getData();
		List<Actor> actors = new CopyOnWriteArrayList<>();

		reviewPublisher.subscribe(new Subscriber<>() {
			@Override
			public void onSubscribe(Subscription s) {
				s.request(2);
			}

			@Override
			public void onNext(ExecutionResult executionResult) {
				if (executionResult.getErrors().size() > 0) {
					executionResult.getErrors().forEach(error -> log.error(error.toString()));
				}
				Map<String, Object> actorResult = executionResult.getData();
				Actor actor = new ObjectMapper().convertValue(actorResult.get("actorAdded"), Actor.class);
 log.info("客户端监听到演员新增：{}",actor);
				actors.add(actor);
			}

			@Override
			public void onError(Throwable t) {
				t.printStackTrace();
			}

			@Override
			public void onComplete() {
			}
		});

		addActor();
		addActor();

		assertThat(actors.size()).isEqualTo(2);
	}
}

从日志输出可以看到客户端成功接收异步通知

总结

订阅模式属于GraphQL特色功能，类似消息队列实现方式，减少客户端对普通查询接口轮训性能开销

GraphQL实践9——Netflix Dgs Graphql数据更新

Fri, 28 Oct 2022 21:58:54 +0800

功能介绍

graphql-java提供了Mutations类型支持数据更新

NGS Mutations

类型定义

type Mutation {
 addActor(actor: SubmitActor!): Actor
}

type Subscription {
 actorAdded: Actor
}

input SubmitActor {
 firstName: String!
 lastName: String!
}

SubmitActor类型可以交给代码生成器自动生成

类型转换器

@Mapper(componentModel = SPRING)
public interface ActorAssembler {

	Actor convert(SubmitActor submit);
}

定义解析器

	@DgsMutation
	public Actor addActor(@InputArgument SubmitActor actor) {
		Actor actorEntity = actorAssembler.convert(actor);
		actorEntity.setActorId(10);
		actorEntity.setLastUpdate(new Date());
		return actorEntity;
	}

测试

访问http://localhost:8080/graphiql即可看到在线查询页面

GraphQL实践8——Netflix Dgs Graphql代码生成

Fri, 28 Oct 2022 21:57:00 +0800

功能介绍

DGS官方开发了识别schema配置生成对应类型、测试类的代码生成器，而Maven插件则由社区维护

Maven代码生成插件

新增依赖

<dependency> 
 <groupId>com.netflix.graphql.dgs.codegen</groupId> 
 <artifactId>graphql-dgs-codegen-client-core</artifactId> 
 <version>5.1.17</version> 
</dependency>

还需要新增构建插件

	 <plugin>
 <groupId>io.github.deweyjose</groupId>
 <artifactId>graphqlcodegen-maven-plugin</artifactId>
 <version>1.24</version>
 <executions>
 <execution>
 <goals>
 <goal>generate</goal>
 </goals>
 </execution>
 </executions>
 <configuration>
 <addGeneratedAnnotation>true</addGeneratedAnnotation>
 <packageName>top.fjy8018.graphsqldemo.generated</packageName>
 <schemaPaths>
 <param>src/main/resources/graphql/actor.graphqls</param>
 </schemaPaths>
 <!-- 生成测试客户端 -->
 <generateClient>true</generateClient>
 </configuration>
 </plugin>
 <plugin>
 <groupId>org.codehaus.mojo</groupId>
 <artifactId>build-helper-maven-plugin</artifactId>
 <executions>
 <execution>
 <phase>generate-sources</phase>
 <goals>
 <goal>add-source</goal>
 </goals>
 <configuration>
 <sources>
 <source>${project.build.directory}/generated-sources</source>
 </sources>
 </configuration>
 </execution>
 </executions>
 </plugin>

插件配置指引：deweyjose/graphqlcodegen: Maven port of the Netflix Gradle code generation plugin for graphql.

编译

执行

mvn clean package

即可生成对应的类

存在问题

IDEA默认不识别target目录下的资源文件，生成后会导致无法引用

解决方法

需要手动将对应的生成目录配置为Source Root即可

总结

DGS官方推荐开发者优先使用代码生成器生成标准类库后再开发，可以减少开发量

GraphQL实践7——Netflix Dgs Graphql分页查询

Fri, 28 Oct 2022 16:06:31 +0800

问题背景

上一篇文章提到，使用List查询每次都返回全量数据，而实际场景更多使用分页查询，graphql-java提供Connection实现游标分页，在Dgs也有对应功能扩展Relay Pagination

集成Relay Pagination

新增依赖

 <dependency>
 <groupId>com.netflix.graphql.dgs</groupId>
 <artifactId>graphql-dgs-pagination</artifactId>
 </dependency>

Mybatis Plus 配置分页插件

@Configuration
@MapperScan("top.fjy8018.graphsqldemo.mapper")
public class MybatisPlusConfig {

 /**
 * 分页插件配置
 *
 * @return
 */
 @Bean
 public MybatisPlusInterceptor mybatisPlusInterceptor() {
 MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor();
 PaginationInnerInterceptor pageConfig = new PaginationInnerInterceptor(DbType.MYSQL);
 interceptor.addInnerInterceptor(pageConfig);
 return interceptor;
 }
}

Relay Pagination配置

type Query {
 actorList: [Actor]
 findOneActor(id : Int!): Actor
 filmList(current: Int!,size: Int!): FilmConnection
 findOneFilm(id : Int!): Film
}

type Actor {
 actorId: Int!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

type Film @connection {
 filmId: Int!
 title: String!
 description: String
 releaseYear: Int
 languageId: Int!
 originalLanguageId: Int
 rentalDuration: Int
 rentalRate: Float
 length: Int
 replacementCost: Float
 rating: String
 specialFeatures: String
 lastUpdate: String
 actors: [Actor]
}

@connection和XXXConection已经由graphql-java底层框架自动生成，无需手动定义

类型转换器

public interface ConnectionAssembler {

 /**
 * mybatis plus分页对象转为Connection
 *
 * @param page 分页对象
 * @param <T> 泛型
 * @return Connection
 */
 static <T> Connection<T> convert(IPage<T> page) {
 AtomicInteger index = new AtomicInteger();
 List<Edge<T>> defaultEdges = Stream.ofNullable(page.getRecords())
 .flatMap(Collection::stream)
 .map(e -> (Edge<T>) new DefaultEdge<>(e, new DefaultConnectionCursor(String.valueOf(index.incrementAndGet()))))
 .toList();
 return new DefaultConnection<>(defaultEdges,
 new DefaultPageInfo(new DefaultConnectionCursor(String.valueOf(page.getCurrent())),
 new DefaultConnectionCursor(String.valueOf(page.getPages())),
 page.getCurrent() > 1,
 page.getCurrent() < page.getPages()));
 }
}

Graphql解析配置

	@DgsQuery
	public Connection<Film> filmList(@InputArgument("current") Integer current,
									 @InputArgument("size") Integer size,
									 DgsDataFetchingEnvironment dfe) {
		IPage<Film> page = new PageDTO<>(current, size);
		IPage<Film> pageResult = filmRepository.page(page);
		return ConnectionAssembler.convert(pageResult);
	}

将List查询改为返回Connection即可

测试

访问http://localhost:8080/graphiql即可看到在线查询页面

查看查询数量，确认是按照分页查询

总结

到此，DSG核心功能尝试基本完成，基本实现常用的Rest功能，同时集成方便，而且官方文档清晰，但目前开源社区还没有太多的基于DSG的系统集成样例，还需要在实践中评估摸索。

thoughtworks也从2016年开始关注graphql，目前给出的建议也是“评估”，可以进行小规模尝试并在合适的时候推广

We’ve seen many successful GraphQL implementations on our projects. We’ve seen some interesting patterns of use too, including GraphQL for server-side resource aggregation. That said, we’ve concerns about misuse of this framework and some of the problems that can occur. Examples include performance gotchas around N+1 queries and lots of boilerplate code needed when adding new models, leading to complexity. There are workarounds to these gotchas such as query caching. Even though it’s not a silver bullet, we still think it’s worth assessing as part of your architecture.

GraphQL实践6——Netflix Dgs Graphql N+1问题

Fri, 28 Oct 2022 16:05:51 +0800

N+1问题介绍

对于上一篇文章样例，如果要获取每个电影的演员名单，要执行如下动作

查询所有电影清单
遍历N个电影，查询对应电影的演员名单

总查询开销为N+1次查询，代价非常大，效率低

优化方案 DataLoader

对于一对一表关联的情况且每个关联对象只有一个值，可以直接使用BatchLoader

但实际上，对于大部分表关联情况，通常为一对多或者多对多，不保证每个关联都有值，此时需要使用MappedBatchLoader维护关联关系

维护多对多实体关系Map

public interface ActorService {
	Map<Integer, List<Actor>> listByFilmId(Collection<Integer> filmIds);
}

@Slf4j
@Service
@RequiredArgsConstructor
public class ActorServiceImpl implements ActorService {

	private final ActorRepository actorRepository;

	private final FilmActorRepository filmActorRepository;

	@Override
	public Map<Integer, List<Actor>> listByFilmId(Collection<Integer> filmIds) {
		List<FilmActor> filmActors = filmActorRepository.listByFilmId(filmIds);
		if (CollectionUtils.isEmpty(filmActors)) {
			return Collections.emptyMap();
		}
		Map<Integer, List<FilmActor>> filmActorMap = filmActors.stream().collect(Collectors.groupingBy(FilmActor::getFilmId));
		List<Integer> actorIds = filmActors.stream().map(FilmActor::getActorId).distinct().toList();
		List<Actor> actors = actorRepository.listByIds(actorIds);
		Map<Integer, Actor> actorsMap = Stream.ofNullable(actors).flatMap(Collection::stream).collect(Collectors.toMap(Actor::getActorId, Function.identity(), (e1, e2) -> e1));
		Map<Integer, List<Actor>> result = Maps.newHashMapWithExpectedSize(filmIds.size());
		filmActorMap.forEach((k, v) -> {
			List<Actor> actorList = Stream.ofNullable(filmActorMap.get(k)).flatMap(Collection::stream).map(e -> actorsMap.get(e.getActorId())).toList();
			result.put(k, actorList);
		});
		return result;
	}
}

Dgs MappedBatchLoader 配置

@DgsDataLoader(name = "actors")
@RequiredArgsConstructor
public class FilmActorsDataLoader implements MappedBatchLoader<Integer, List<Actor>> {

	private final ActorService actorService;

	@Override
	public CompletionStage<Map<Integer, List<Actor>>> load(Set<Integer> keys) {
		return CompletableFuture.supplyAsync(() -> actorService.listByFilmId(keys));
	}
}

测试

访问http://localhost:8080/graphiql即可看到在线查询页面

此时执行嵌套查询只会查询2次，一次查电影列表，一次查所有电影的Actor列表，非常快速

总结

使用BatchLoader实现对N+1问题优化，但还有一个潜在问题，即大数据分页，该样例中，有1000个电影，对应5000+演员，查询一次获取全量数据，对数据库压力较大，也不符合实际场景，还需要进行分页优化

GraphQL实践5——Netflix Dgs Graphql懒加载

Fri, 28 Oct 2022 16:04:43 +0800

问题背景

在graphql中，需要查询的字段由调用方传递，如果此时存在关联表才能获取的字段，但前端不需要该字段的时候，默认查询所有字段会带来较大开销

例如

type Actor {
 actorId: Int!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

type Film {
 filmId: Int!
 title: String!
 description: String
 actors: [Actor]
}

type Query {
 filmList: [Film]
}

解决方案——懒加载

DGS框架中支持配置调用方传递了指定字段才读取对应的数据

配置懒加载

增加实体定义，模拟嵌套实体

Mybatis Plus配置

扫描配置

@Configuration
@MapperScan("top.fjy8018.graphsqldemo.mapper")
public class MybatisPlusConfig {
}

实体

表结构还是使用sakila样例数据库的表

@Data
public class Actor implements Serializable {

 @Serial
 private static final long serialVersionUID = 1L;

 /**
 * 联系人ID
 */
 @TableId(type = IdType.AUTO)
 private Integer actorId;

 private String firstName;

 private String lastName;

 private Date lastUpdate;

}

@Data
public class Film implements Serializable {

	@Serial
	private static final long serialVersionUID = 1L;

	/**
	 * 联系人ID
	 */
	@TableId(type = IdType.AUTO)
	private Integer filmId;

	private String title;

	private String description;

	private Integer releaseYear;

	private Integer languageId;

	private Integer originalLanguageId;

	private Integer rentalDuration;

	private BigDecimal rentalRate;

	private Integer length;

	private BigDecimal replacementCost;

	private String rating;

	private String specialFeatures;

	private Date lastUpdate;
}

@Data
public class FilmActor implements Serializable {

	@Serial
	private static final long serialVersionUID = 1L;

	/**
	 * 联系人ID
	 */
	@TableId(type = IdType.AUTO)
	private Integer actorId;

	private Integer filmId;

	private Date lastUpdate;
}

Mapper

public interface ActorMapper extends BaseMapper<Actor> {}
public interface FilmActorMapper extends BaseMapper<FilmActor> {}
public interface FilmMapper extends BaseMapper<Film> {}

DAO

public interface ActorRepository extends IService<Actor> {
}
public interface FilmActorRepository extends IService<FilmActor> {
	List<FilmActor> listByFilmId(@Size.List({@Size(min = 1, message = "至少要有一个元素")}) List<Integer> filmIds);
}
public interface FilmRepository extends IService<Film> {
}

@Validated
@Repository
@RequiredArgsConstructor
public class ActorRepositoryImpl extends ServiceImpl<ActorMapper, Actor> implements ActorRepository {
}
@Validated
@Repository
@RequiredArgsConstructor
public class FilmActorRepositoryImpl extends ServiceImpl<FilmActorMapper, FilmActor> implements FilmActorRepository {
	@Override
	public List<FilmActor> listByFilmId(List<Integer> filmIds) {
		return lambdaQuery().in(FilmActor::getFilmId, filmIds).list();
	}
}
@Validated
@Repository
@RequiredArgsConstructor
public class FilmRepositoryImpl extends ServiceImpl<FilmMapper, Film> implements FilmRepository {
}

Service

多表查询聚合逻辑

public interface ActorService {
	List<Actor> listByFilmId(Integer filmId);
}

@Slf4j
@Service
@RequiredArgsConstructor
public class ActorServiceImpl implements ActorService {

	private final ActorRepository actorRepository;

	private final FilmActorRepository filmActorRepository;

	@Override
	public List<Actor> listByFilmId(Integer filmId) {
		List<FilmActor> filmActors = filmActorRepository.listByFilmId(List.of(filmId));
		if (CollectionUtils.isEmpty(filmActors)) {
			return Collections.emptyList();
		}
		List<Integer> actorIds = filmActors.stream().map(FilmActor::getActorId).toList();
		return actorRepository.listByIds(actorIds);
	}
}

Graphql 查询解析器

@DgsComponent
@RequiredArgsConstructor
public class FilmDataFetcher {

	private final FilmRepository filmRepository;

	private final ActorService actorService;

	@DgsQuery
	public Collection<Film> filmList() {
		return filmRepository.list();
	}

	@DgsData(parentType = "Film")
	public List<Actor> actors(DgsDataFetchingEnvironment dfe) {
		Film film = dfe.getSource();
		return actorService.listByFilmId(film.getFilmId());
	}

	@DgsQuery
	public Film findOneFilm(@InputArgument Integer id) {
		return filmRepository.getById(id);
	}
}

增加DgsDataFetchingEnvironment入参，DGS会自动获取请求上下文参数

此处定义即为传入了actors参数才执行该方法获取对应的Actor集合，实现懒加载

测试

访问http://localhost:8080/graphiql即可看到在线查询页面

不访问Actor

此时可以看到日志输出，没有查询Actor表

访问Actor

此时DSG会执行N+1查询Actor表，性能损耗非常大

总结

对于表关联，开销较大字段，都可以使用DSG懒加载实现性能优化，但此时还是存在N+1查询性能问题，将在后面使用DataLoader优化

GraphQL实践4——Netflix Dgs Graphql集成Mybatis Plus

Fri, 28 Oct 2022 16:03:00 +0800

介绍

通过JPA或者Mybatis Plus集成Graphql框架已经很简便，但其实使用graphql-java集成过于底层，很多扩展内容实现并不方便，例如：

分页查询
资源懒加载
N+1问题

Netflix 基于长期的Graphql实践，2020年开源了DGS Framework，让解决上述问题更加简单，大量开源系统集成Graphql也是通过集成DGS组件

The DGS framework project started at Netflix in 2019 as internal teams began developing multiple GraphQL services. As 2020 wrapped up, Netflix decided to open source the framework and build a community around it.

集成Netflix DGS

官网：Home - DGS Framework (netflix.github.io)

引入依赖

技术栈：

SpringBoot 2.7.1
Mybatis Plus
JDK 17
Dgs SpringBoot

本次和上一篇文章一样，选择原因不在赘述

	<dependencyManagement>
 <dependencies>
 <dependency>
 <groupId>com.netflix.graphql.dgs</groupId>
 <artifactId>graphql-dgs-platform-dependencies</artifactId>
 <!-- The DGS BOM/platform dependency. This is the only place you set version of DGS -->
 <version>4.9.16</version>
 <type>pom</type>
 <scope>import</scope>
 </dependency>
 </dependencies>
 </dependencyManagement>

 <dependencies>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-validation</artifactId>
 </dependency>

 <dependency>
 <groupId>com.netflix.graphql.dgs</groupId>
 <artifactId>graphql-dgs-spring-boot-starter</artifactId>
 </dependency>

 <dependency>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 <optional>true</optional>
 </dependency>

 <dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <scope>runtime</scope>
 </dependency>
 <!-- mybatis-plus-boot-starter -->
 <dependency>
 <groupId>com.baomidou</groupId>
 <artifactId>mybatis-plus-boot-starter</artifactId>
 <version>3.5.2</version>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-test</artifactId>
 <scope>test</scope>
 </dependency>
 </dependencies>

配置文件

spring:
 datasource:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: root
 password: ****
 url: jdbc:mysql://localhost:3306/sakila?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
server:
 port: 8080
logging:
 level:
 top.fjy8018.graphsqldemo.mapper: DEBUG

dgs:
 graphql:
 schema-locations:
 - classpath*:graphql/*.graphqls

由于DGS默认扫描路径为classpath*:schema/**/*.graphql*，本次基于上一篇文章的工程配置，需要手动指定一下扫描文件目录

Mybatis Plus配置

扫描配置

@Configuration
@MapperScan("top.fjy8018.graphsqldemo.mapper")
public class MybatisPlusConfig {
}

实体

表结构还是使用sakila样例数据库的表

@Data
public class Actor implements Serializable {

 @Serial
 private static final long serialVersionUID = 1L;

 /**
 * 联系人ID
 */
 @TableId(type = IdType.AUTO)
 private Integer actorId;

 private String firstName;

 private String lastName;

 private Date lastUpdate;

}

Mapper

public interface ActorMapper extends BaseMapper<Actor> {
}

DAO

从分层模型定义上，MP自带的CRUD接口本质上还是一个DAO，故定义为Repository更加合适，不与业务逻辑Serice冲突

同时也建议QueryWrapper仅在对应的Repository使用，防止查询条件散落在各处

public interface ActorRepository extends IService<Actor> {
}

@Validated
@Repository
@RequiredArgsConstructor
public class ActorRepositoryImpl extends ServiceImpl<ActorMapper, Actor> implements ActorRepository {
}

Graphql 查询解析器

@DgsComponent
@RequiredArgsConstructor
public class ActorDataFetcher {

	private final ActorRepository actorRepository;

	@DgsQuery
	public Collection<Actor> actorList() {
		return actorRepository.list();
	}

	@DgsQuery
	public Actor findOneActor(@InputArgument Integer id) {
		return actorRepository.getById(id);
	}
}

type Query {
 actorList: [Actor]
 findOneActor(id : Int!): Actor
}

type Actor {
 actorId: Int!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

此处集成和graphql-java-kickstart类似，也是十分简便，不指定别名默认将方法名映射到文件方法名

测试

访问http://localhost:8080/graphiql即可看到在线查询页面

总结

此处还看不出DGS与其他graphql-java组件的区别，都是集成简便，可以和任何数据源、ORM框架集成，后续将会逐步实践DGS特性，包括懒加载、N+1问题、分页组件等

GraphQL实践3——Spring-Graphql集成Mybatis Plus

Thu, 27 Oct 2022 15:37:00 +0800

介绍

考虑到国内大部分场景对接的是Mybatis或者Mybatis Plus生态，而不是JPA，同时Graphql版本也进一步迭代，故计划使用Mybatis Plus对接Graphql

集成过程

引入依赖

由于JDK已经发展到19，目前最新的LTS为17，我所在的公司也将生产升级到了JDK 17，未来更多新应用会直接基于17构建，故此处直接基于JDK17对接
SpringBoot版本为2.7.1，目前已知3.0.0 RC1还不兼容第三方graphql
此处不使用spring官方的starter，改为使用graphql-java-kickstart的starter，编写更加简单


 <dependencies>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 </dependency>

 <dependency>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 <optional>true</optional>
 </dependency>

 <!-- graphql-spring-boot-starter -->
 <dependency>
 <groupId>com.graphql-java-kickstart</groupId>
 <artifactId>graphql-spring-boot-starter</artifactId>
 <version>12.0.0</version>
 </dependency>

 <dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <scope>runtime</scope>
 </dependency>
 <!-- mybatis-plus-boot-starter -->
 <dependency>
 <groupId>com.baomidou</groupId>
 <artifactId>mybatis-plus-boot-starter</artifactId>
 <version>3.5.2</version>
 </dependency>
 <!-- querydsl-core -->
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-core</artifactId>
 </dependency>
 
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-test</artifactId>
 <scope>test</scope>
 </dependency>
 </dependencies>

配置文件

spring:
 datasource:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: root
 password: ****
 url: jdbc:mysql://localhost:3306/sakila?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
server:
 port: 8080
graphql:
 tools:
 schema-location-pattern: graphql/*.graphqls
 graphiql:
 # 开启可交互页面，用于调试
 enabled: true
logging:
 level:
 top.fjy8018.graphsqldemo.mapper: DEBUG

Mybatis Plus配置

扫描配置

@Configuration
@MapperScan("top.fjy8018.graphsqldemo.mapper")
public class MybatisPlusConfig {
}

实体

@Data
public class Actor implements Serializable {

 @Serial
 private static final long serialVersionUID = 1L;

 /**
 * 联系人ID
 */
 @TableId(type = IdType.AUTO)
 private Integer actorId;

 private String firstName;

 private String lastName;

 private Date lastUpdate;

}

Mapper

public interface ActorMapper extends BaseMapper<Actor> {
}

DAO

public interface ActorRepository extends IService<Actor> {
}

@Validated
@Repository
@RequiredArgsConstructor
public class ActorRepositoryImpl extends ServiceImpl<ActorMapper, Actor> implements ActorRepository {
}

Graphql 查询解析器

@Component
@AllArgsConstructor
public class ActorGraphQLQueryResolver implements GraphQLQueryResolver {

 private final ActorRepository actorRepository;

 public Collection<Actor> actorList() {
 return actorRepository.list();
 }

 public Actor findOneActor(Integer id) {
 return actorRepository.getById(id);
 }
}

查询文件定义在resources/graphql/actor.graphqls

type Query {
 actorList: [ActorEntity]
 findOneActor(id : ID!): ActorEntity
}

type ActorEntity {
 actorId: ID!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

测试

访问http://localhost:8080/graphiql即可看到在线查询页面

总结

依托Mybatis Plus强大的自带CRUD方法，可以实现和JPA类似的快速对接

Java 17 生产实践

Fri, 06 May 2022 11:04:00 +0800

背景

版本发布

2021年3月16日 Java 17 作为继Java 11的下一个LTS版本正式发布，JDK 17 作为下一代 LTS 将提供至少到 2026 年的支持。

同业动态

2022年ThoughtWorks 26期技术雷达也提到，建议各个组织将Java 17纳入评估。

评估 82. Java 17

我们通常不会专门介绍某一语言的新版本，但我们还是想关注一下 Java 新的长期支持（LTS）版本——Java 17。尽管出现了前景不错的新特性，比如模式匹配等预览特性，但其实向新 LTS 版本升级的方案更应该吸引各个组织的兴趣。我们建议各个组织在 Java 有新的发布时对其进行评估，确保能够恰当地适配这些新特性和版本。尽管定期更新有利于简化开发并且方便管理，但许多组织却意外地并不经常更新语言的版本。希望通过 LTS 版本的升级以及开发团队对语言的定期更新，能够使生产软件免于因为“更新成本太高”而一直困在 Java 的过时版本上。

框架发展情况

Spring Framework 6和SpringBoot 3.0 将在2022年底正式发布，将基于Java 17构建。

Spring Boot 3.0.0 M1 Release Notes · spring-projects/spring-boot Wiki (github.com)

Minimum Requirements Changes
Spring Boot 3.0 makes the following changes to its minimum supported versions:

Gradle 7.3

Jakarta EE 9

Java 17

Kotlin 1.6

Spring Framework 6

其他团队落地情况

在此之前，我所在的团队使用的Java版本一直停留在Java8，今年经过其他开发团队的尝试，我们团队在4月份也正式决定在生产环境使用Java 17。

实践情况

开发组件版本

本次计划在内部运营管理系统重构中使用Java 17作为基础版本，其他核心组件版本如下

SpringBoot 2.6.6
mapstruct 1.4.2.Final
mybatis-plus 3.5.1
jdk 17.0.1

开发时SpringBoot 3.0还处于M2里程碑阶段，暂时不适合带入生产环境，等未来3.0 GA后再考虑升级

基础镜像

原本使用的基于jdk8 的Docker image也需要替换成基于jdk 17的image

带来的问题：

原本基于jdk8构建并注入到镜像中的opentelemetry-javaagent暂时不可用，等待后续基于jdk17 构建后才能重新启用

JVM参数

由于从Java11开始G1替代CMS作为默认的垃圾收集器，Java14开始移除了CMS GC。

故团队内定义的系统框架配置的GC调优参数基本废弃，并且可以使用Java 14新引入的JVM参数，例如-XX:+ShowCodeDetailsInExceptionMessages

开启前

Exception in thread "main" java.lang.NullPointerException
 at Book.main(Book.java:5)

开启后

Exception in thread "main" java.lang.NullPointerException: 
 Cannot assign field "book" because "shoopingcart.buy" is null
 at Book.main(Book.java:5)

语法特性

从Java8到Java17期间新增的语法特性很多，此处提及几个比较常用的

快速创建不可变集合

不可变对象具有许多优点，包括：

可安全使用不受信任的库。
线程安全。
不需要支持变化判断，所有不可变的集合实现都比其可变的同级实现更节省内存。

Java9开始新增List、Map、Set等of方法快速创建不可变集合

List.of(prodStrategy, instStrategy)

实现源码

 static <E> List<E> of(E e1, E e2) {
 return new ImmutableCollections.List12<>(e1, e2);
 }

Java16开始，Stream类中也新增toList()方法，快速将流转成不可变集合

List<Long> ids = idList.stream().map(ContactId::id).toList();

实现源码

 default List<T> toList() {
 return (List<T>) Collections.unmodifiableList(new ArrayList<>(Arrays.asList(this.toArray())));
 }

注意：如果需要将流转成可变集合，还是需要使用collect(Collectors.toList())方法

record类型

Java14开始新增record类型，用于表示不可变class，实践中可以方便的表示值对象，无需使用@Value注解

public record ContactId(@NotNull(message = "ID不能为空") Long id) {
}

instanceof模式匹配

Java14开始支持模式匹配，无需做强制类型转换

	private void configMessageConverters(List<HttpMessageConverter<?>> converters) {
		converters.removeIf(x -> x instanceof StringHttpMessageConverter || x instanceof MappingJackson2HttpMessageConverter);
		converters.add(new StringHttpMessageConverter(Charsets.UTF_8));
		converters.add(new MappingJackson2HttpMessageConverter(objectMapper));
	}

创建类型安全的流

Java9 开始，新增ofNullable方法创建类型安全的流对象，避免在调用stream方法时报NPE

// query.getDescColumns()可能为null
List<OrderItem> descItems = Stream.ofNullable(query.getDescColumns())
				.flatMap(Collection::stream)
				.map(OrderItem::desc)
				.toList();

性能和安全提升

从同一段功能逻辑中测试发现，Java8和Java17在并发场景下性能差距达到2倍以上

并且同样的逻辑（使用了并发安全的工具）在并发量达到50000的时候在Java8中依然有线程安全问题，而Java17中不会有线程安全问题

外部测评结果

从规划调度引擎 OptaPlanner 项目（原文 (opens new window)）对 JDK 17和 JDK 11 的性能基准测试进行了对比来看：

对于 G1GC（默认），Java 17 比 Java 11 快 8.66%；

对于 ParallelGC，Java 17 比 Java 11 快 6.54%；

Parallel GC 整体比 G1 GC 快 16.39%

简而言之，JDK17 更快，高吞吐量垃圾回收器比低延迟垃圾回收器更快。

GraphQL实践2——Spring-GraphQL集成JPA与MySQL

Mon, 15 Nov 2021 15:38:00 +0800

介绍

上一篇介绍到用第三方库集成GraphQL，目前spring-graphql项目已经出到1.0.0-M3版本，属于内部预览版，此处尝鲜验证

引入依赖

由于SpringBoot 2.6.0还未发布，因此需要引入较多依赖

<dependencies>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 </dependency>
 <dependency>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 <optional>true</optional>
 </dependency>
 <dependency>
 <groupId>org.springframework.experimental</groupId>
 <artifactId>graphql-spring-boot-starter</artifactId>
 <version>1.0.0-M3</version>
 </dependency>
 <!--升级到Spring Boot 2.6后可以移除-->
 <dependency>
 <groupId>org.springframework.data</groupId>
 <artifactId>spring-data-commons</artifactId>
 <version>2.6.0-RC1</version>
 </dependency>
 <dependency>
 <groupId>org.springframework.data</groupId>
 <artifactId>spring-data-jpa</artifactId>
 <version>2.6.0-RC1</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/org.hibernate.javax.persistence/hibernate-jpa-2.1-api -->
 <dependency>
 <groupId>org.hibernate.javax.persistence</groupId>
 <artifactId>hibernate-jpa-2.1-api</artifactId>
 <version>1.0.2.Final</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/com.querydsl/querydsl-jpa -->
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-jpa</artifactId>
 <version>4.4.0</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/com.querydsl/querydsl-apt -->
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-apt</artifactId>
 <version>4.4.0</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/javax.annotation/javax.annotation-api -->
 <dependency>
 <groupId>javax.annotation</groupId>
 <artifactId>javax.annotation-api</artifactId>
 </dependency>
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-core</artifactId>
 </dependency>
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-jpa</artifactId>
 </dependency>
 <dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <scope>runtime</scope>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-data-jpa</artifactId>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-test</artifactId>
 <scope>test</scope>
 </dependency>
</dependencies>

引入里程碑仓库

由于还没有GA，所以要引入里程碑仓库

<repositories>
 <repository>
 <id>spring-milestones</id>
 <name>Spring Milestones</name>
 <url>https://repo.spring.io/milestone</url>
 </repository>
 <repository>
 <id>spring-snapshots</id>
 <name>Spring Snapshots</name>
 <url>https://repo.spring.io/snapshot</url>
 <snapshots>
 <enabled>true</enabled>
 </snapshots>
 </repository>
</repositories>

引入插件

由于使用了dsl动态生成，还要引入注解处理器

<build>
 <plugins>
 <plugin>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-maven-plugin</artifactId>
 <configuration>
 <excludes>
 <exclude>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 </exclude>
 </excludes>
 </configuration>
 </plugin>
 <!-- added for QueryDsl-->
 <plugin>
 <groupId>com.mysema.maven</groupId>
 <artifactId>apt-maven-plugin</artifactId>
 <version>1.1.3</version>
 <executions>
 <execution>
 <goals>
 <goal>process</goal>
 </goals>
 <configuration>
 <outputDirectory>target/generated-sources/java</outputDirectory>
 <processor>com.querydsl.apt.jpa.JPAAnnotationProcessor</processor>
 </configuration>
 </execution>
 </executions>
 </plugin>
 </plugins>
</build>

SpringBoot 配置

编写配置文件

spring:
 datasource:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: root
 password: "****"
 url: jdbc:mysql://localhost:3306/sakila?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
 jpa:
 hibernate:
 ddl-auto: none
 show-sql: true
 database-platform: org.hibernate.dialect.MySQL55Dialect
 properties:
 hibernate:
 format_sql: true
 use_sql_comments: true
 graphql:
 path: /graphql
 schema:
 locations: classpath:graphql/
 fileExtensions: .graphqls, .gqls
 printer:
 enabled: true
server:
 port: 8080

实体类

package top.fjy8018.graphsqldemo.entity;

import lombok.Data;

import javax.persistence.*;
import java.sql.Timestamp;
import java.util.Objects;

/**
 * 演员表实体类
 *
 * @author F嘉阳
 * @date 2021/11/5 10:34
 */
@Data
@Entity
@Table(name = "actor", schema = "sakila")
public class ActorEntity {
 @Id
 @Column(name = "actor_id", nullable = false)
 private Integer actorId;

 @Column(name = "first_name", nullable = false, length = 45)
 private String firstName;

 @Column(name = "last_name", nullable = false, length = 45)
 private String lastName;

 @Column(name = "last_update", nullable = false)
 private Date lastUpdate;

}

DAO

DAO直接继承dsl相关处理器，可自动配对增删改查方法，相比第三方库更加简便

package top.fjy8018.graphsqldemo.repository;

import org.springframework.data.querydsl.QuerydslPredicateExecutor;
import org.springframework.data.repository.CrudRepository;
import org.springframework.graphql.data.GraphQlRepository;
import top.fjy8018.graphsqldemo.entity.ActorEntity;

/**
 * 演员表DAO
 *
 * @author F嘉阳
 * @date 2021/11/5 10:35
 */
@GraphQlRepository
public interface ActorEntityRepository extends
 CrudRepository<ActorEntity, Integer>, QuerydslPredicateExecutor<ActorEntity> {
}

Graphql资源定义

在resources/graphql下定义接口和实体文件

接口声明和实体定义schema.graphqls

type Query {
 findOneActor(actorId : ID!): ActorEntity
 actorList: [ActorEntity]
}

type ActorEntity {
 actorId: ID!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

接口定义acotrEntity.graphql

query findOneActor($id: ID!) {
 findOneActor(actorId: $id) {
 actorId
 firstName
 lastName
 lastUpdate
 }
}

query actorList {
 actorList{
 actorId
 }
}

启动测试

Spring-Graphql默认不包含可视化界面，此处使用postman进行测试

唯一查询

列表查询

总结

使用Spring官方组件好处在于和Spring生态集成度很高，如果本身就采用Jpa方式进行业务开发，迁移更加方便，需要开发的代码也很少。

样例源码地址

FJiayang/graphql-demo

GraphQL实践1——集成JPA与MySQL

Fri, 05 Nov 2021 17:41:00 +0800

介绍

官方定义：

GraphQL 既是一种用于 API 的查询语言也是一个满足你数据查询的运行时。 GraphQL 对你的 API 中的数据提供了一套易于理解的完整描述，使得客户端能够准确地获得它需要的数据，而且没有任何冗余，也让 API 更容易地随着时间推移而演进，还能用于构建强大的开发者工具。

中文官网：

GraphQL — A query language for your API

集成方案选择

GraphQL经过近几年的发展，SpringBoot官方、第三方都提供了starter

SpringBoot官方提供的starter

Spring GraphQL

第三方提供的starter

graphql-spring-boot

两者活跃度都还不错，但考虑到SpringBoot官方的依赖库还未GA，而且使用起来与Spring家族耦合过大，最终决定采用第三方的starter

集成过程

数据库配置

此处采用MySQL数据库，数据内容采用MySQL官方样例sakila数据，表结构和数据地址如下

sakila database下载地址

导入后如下

工程配置

引入最新的SpringBoot框架，依赖如下

<dependencies>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 </dependency>

 <dependency>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 <optional>true</optional>
 </dependency>

 <!-- https://mvnrepository.com/artifact/com.graphql-java-kickstart/graphql-spring-boot-starter -->
 <dependency>
 <groupId>com.graphql-java-kickstart</groupId>
 <artifactId>graphql-spring-boot-starter</artifactId>
 <version>12.0.0</version>
 </dependency>

 <dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <scope>runtime</scope>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-data-jpa</artifactId>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-test</artifactId>
 <scope>test</scope>
 </dependency>
</dependencies>

增加数据库配置

spring:
 datasource:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: root
 password: "****"
 url: jdbc:mysql://localhost:3306/sakila?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
 jpa:
 hibernate:
 ddl-auto: update
 show-sql: true
 database-platform: org.hibernate.dialect.MySQL55Dialect
 properties:
 hibernate:
 format_sql: true
 use_sql_comments: true
server:
 port: 8080
graphql:
 tools:
 schema-location-pattern: graphql/*.graphqls
 graphiql:
 # 开启可交互页面，用于调试
 enabled: true

实体映射

新增实体映射，此处只映射Actor表作为样例

package top.fjy8018.graphsqldemo.entity;

import lombok.Data;

import javax.persistence.*;
import java.sql.Timestamp;
import java.util.Objects;

/**
 * 演员表实体映射
 *
 * @author F嘉阳
 * @date 2021/11/5 10:34
 */
@Data
@Entity
@Table(name = "actor", schema = "sakila")
public class ActorEntity {
 @Id
 @Column(name = "actor_id", nullable = false)
 private Integer actorId;

 @Column(name = "first_name", nullable = false, length = 45)
 private String firstName;

 @Column(name = "last_name", nullable = false, length = 45)
 private String lastName;

 @Column(name = "last_update", nullable = false)
 private Timestamp lastUpdate;
}

新增DAO

package top.fjy8018.graphsqldemo.repository;

import org.springframework.data.jpa.repository.JpaRepository;
import top.fjy8018.graphsqldemo.entity.ActorEntity;

/**
 * 演员表DAO
 *
 * @author F嘉阳
 * @date 2021/11/5 10:35
 */
public interface ActorEntityRepository extends JpaRepository<ActorEntity, Integer> {
}

新增查询接口配置

在resources/graphql下新增文件actor.graphqls

type Query {
 actorList: [ActorEntity]
 findOneActor(id : ID!): ActorEntity
}

type ActorEntity {
 actorId: ID!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

新增查询解析器

package top.fjy8018.graphsqldemo.resolver;

import graphql.kickstart.tools.GraphQLQueryResolver;
import lombok.AllArgsConstructor;
import org.springframework.stereotype.Component;
import top.fjy8018.graphsqldemo.entity.ActorEntity;
import top.fjy8018.graphsqldemo.repository.ActorEntityRepository;

import java.util.Collection;

/**
 * 演员表GraphQL查询解析器
 *
 * @author F嘉阳
 * @date 2021/11/5 16:27
 */
@Component
@AllArgsConstructor
public class ActorGraphQLQueryResolver implements GraphQLQueryResolver {

 private final ActorEntityRepository actorRepository;

 public Collection<ActorEntity> actorList() {
 return actorRepository.findAll();
 }

 public ActorEntity findOneActor(Integer id) {
 return actorRepository.findById(id).orElse(null);
 }
}

启动验证

启动工程后，浏览器打开http://localhost:8080/graphiql

列表查询

由于采用交互式页面，输入也会有语法提示，比较方便

唯一查询

优缺点分析

优点

接口定义清晰，配置即接口
前端按需获取字段，减少数据传输量
接口整合、演进更加便利

缺点

减少字段个数对后端实现没有变化，并不能提升数据库性能

样例源码地址

FJiayang/graphql-demo

SpringCloudStream函数式编程模型

Sun, 21 Feb 2021 12:22:00 +0800

背景

SpringCloud 3.1版本之后，@EnableBinding、@Output等StreamApi注解都标记为废弃

/**
 * Indicates that an output binding target will be created by the framework.
 *
 * @author Dave Syer
 * @author Marius Bogoevici
 * @author Artem Bilan
 *
 * @deprecated as of 3.1 in favor of functional programming model
 */

@Qualifier
@Target({ ElementType.FIELD, ElementType.METHOD, ElementType.ANNOTATION_TYPE,
		ElementType.PARAMETER })
@Retention(RetentionPolicy.RUNTIME)
@Inherited
@Documented
@Deprecated
public @interface Output {

	/**
	 * Specify the binding target name; used as a bean name for binding target and as a
	 * destination name by default.
	 * @return the binding target name
	 */
	String value() default "";

}

/**
 * Enables the binding of targets annotated with {@link Input} and {@link Output} to a
 * broker, according to the list of interfaces passed as value to the annotation.
 *
 * @author Dave Syer
 * @author Marius Bogoevici
 * @author David Turanski
 * @author Soby Chacko
 *
 * @deprecated as of 3.1 in favor of functional programming model
 */
@Target({ ElementType.TYPE, ElementType.ANNOTATION_TYPE })
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@Configuration
@Import({ BindingBeansRegistrar.class, BinderFactoryAutoConfiguration.class })
@EnableIntegration
@Deprecated
public @interface EnableBinding {

	/**
	 * A list of interfaces having methods annotated with {@link Input} and/or
	 * {@link Output} to indicate binding targets.
	 * @return list of interfaces
	 */
	Class<?>[] value() default {};

}

Producing and Consuming Messages — SpringCloud 官方文档

@EnableBinding @deprecated as of 3.1 in favor of functional programming model — stackoverflow

The type EnableBinding is deprecated — stackoverflow

因此，所有采用命令式编程的API，在升级到新版本时，官方推荐使用更加简单的函数式编程API

完整代码样例

FJiayang/spring-cloud-stream-rabbit-example — GitHub项目地址

升级方式

命令式编程模型

以简单多Topic消息为例，原本的编程方式如下

生产者

@SpringBootApplication
public class ProducerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ProducerApplication.class, args);
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 17:57
 */
@RestController
@EnableBinding(MySource.class)
public class Producer {

 @Autowired
 private MySource channel;

 @RequestMapping("/send")
 public String send() {
 channel.output().send(MessageBuilder.withPayload(new Date()).build());
 return "success";
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 18:01
 */
public interface MySource {
 String OUTPUT = "output";

 @Output(MySource.OUTPUT)
 MessageChannel output();
}

配置文件

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin
 cloud:
 stream:
 bindings:
 output:
 destination: my-test-channel
server:
 port: 8082

消费者

@SpringBootApplication
public class ConsumerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ConsumerApplication.class, args);
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 18:10
 */
@EnableBinding(MySink.class)
public class Consumer {
 @StreamListener(MySink.INPUT)
 public void receive(Message<String> message) {
 System.out.println("接收到MQ消息:" + message.getPayload());
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 18:07
 */
public interface MySink {
 String INPUT = "input";

 @Input(MySink.INPUT)
 SubscribableChannel input();
}

配置文件

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin
 cloud:
 stream:
 bindings:
 input:
 destination: my-test-channel
server:
 port: 8081

函数式编程模型

从命令式升级到函数式编程，首先需要升级SpringCloud版本至3.1以上，此处升级到最新版2020.0.1

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>

 <groupId>top.fjy8018</groupId>
 <artifactId>cloud-stream</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>pom</packaging>

 <name>cloud-stream</name>
 <description>Demo project for Spring Boot</description>

 <modules>
 <module>producer</module>
 <module>consumer</module>
 </modules>

 <parent>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-parent</artifactId>
 <version>2.4.2</version>
 <relativePath/> <!-- lookup parent from repository -->
 </parent>

 <properties>
 <spring-cloud.version>2020.0.1</spring-cloud.version>
 </properties>

 <dependencyManagement>
 <dependencies>
 <dependency>
 <groupId>org.springframework.cloud</groupId>
 <artifactId>spring-cloud-dependencies</artifactId>
 <version>${spring-cloud.version}</version>
 <type>pom</type>
 <scope>import</scope>
 </dependency>
 </dependencies>
 </dependencyManagement>
</project>

目前国内有清晰的升级指引的还比较少，目前只能参考官方文档材料

Programming Model — SpringCloud官方文档

模型1——系统定时驱动的消息队列

官方文档描述

Suppliers (Sources)
Function and Consumer are pretty straightforward when it comes to how their invocation is triggered. They are triggered based on data (events) sent to the destination they are bound to. In other words, they are classic event-driven components.

However, Supplier is in its own category when it comes to triggering. Since it is, by definition, the source (the origin) of the data, it does not subscribe to any in-bound destination and, therefore, has to be triggered by some other mechanism(s). There is also a question of Supplier implementation, which could be imperative or reactive and which directly relates to the triggering of such suppliers.

Consider the following sample:
@SpringBootApplication
public static class SupplierConfiguration {

	@Bean
	public Supplier<String> stringSupplier() {
		return () -> "Hello from Supplier";
	}
}
The preceding Supplier bean produces a string whenever its get() method is invoked. However, who invokes this method and how often? The framework provides a default polling mechanism (answering the question of “Who?”) that will trigger the invocation of the supplier and by default it will do so every second (answering the question of “How often?”). In other words, the above configuration produces a single message every second and each message is sent to an output destination that is exposed by the binder.

其大意为，若采用默认的函数式编程模型，会每秒钟发送一条消息到消费者，定时发送参数可以配置

按照文档升级后如下

生产者

@SpringBootApplication
public class ProducerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ProducerApplication.class, args);
 }

 @Bean
 public Supplier<Date> source1() {
 return () -> new Date();
 }
}

函数式编程不需要在代码中定义监听队列，只需要按照约定写在配置文件中即可

官方文档描述

Functional binding names
Unlike the explicit naming required by annotation-based support (legacy) used in the previous versions of spring-cloud-stream, the functional programming model defaults to a simple convention when it comes to binding names, thus greatly simplifying application configuration. Let’s look at the first example:
@SpringBootApplication
public class SampleApplication {

	@Bean
	public Function<String, String> uppercase() {
	 return value -> value.toUpperCase();
	}
}
In the preceding example we have an application with a single function which acts as message handler. As a Function it has an input and output. The naming convention used to name input and output bindings is as follows:

input - <functionName> + -in- + <index>

output - <functionName> + -out- + <index>

The in and out corresponds to the type of binding (such as input or output). The index is the index of the input or output binding. It is always 0 for typical single input/output function, so it’s only relevant for Functions with multiple input and output arguments.

So if for example you would want to map the input of this function to a remote destination (e.g., topic, queue etc) called “my-topic” you would do so with the following property:
--spring.cloud.stream.bindings.uppercase-in-0.destination=my-topic
Note how uppercase-in-0 is used as a segment in property name. The same goes for uppercase-out-0.

配置文件

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin

 cloud:
 stream:
 bindings:
 source1-out-0:
 destination: test1
 function:
 definition: source1

server:
 port: 8083

由此可见，整体配置简化了很多，多为规约大于配置的方式进行定义，特别注意function配置的位置，是在stream配置的同级

消费者

@SpringBootApplication
public class ConsumerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ConsumerApplication.class, args);
 }

 @Bean
 public Consumer<Date> sink1() {
 return System.out::println;
 }
}

配置文件同理

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin

 cloud:
 stream:
 bindings:
 sink1-in-0:
 destination: test1
 function:
 definition: sink1


server:
 port: 8081

运行结果

生产者每隔一秒发送一次消息

模型2——`StreamBridge`

但实际生产中，更多是由业务场景触发，故不能使用该模式进行配置，SpringCloud还提供了另外一种StreamBridge模式

官方文档介绍

Sending arbitrary data to an output (e.g. Foreign event-driven sources)
There are cases where the actual source of data may be coming from the external (foreign) system that is not a binder. For example, the source of the data may be a classic REST endpoint. How do we bridge such source with the functional mechanism used by spring-cloud-stream?

Spring Cloud Stream provides two mechanisms, so let’s look at them in more details

Here, for both samples we’ll use a standard MVC endpoint method called delegateToSupplier bound to the root web context, delegating incoming requests to stream via two different mechanisms - imperative (via StreamBridge) and reactive (via EmitterProcessor).

大意是，实际的数据源可以是外部事件驱动，例如通过Rest接口

官方样例

Using StreamBridge
@SpringBootApplication
@Controller
public class WebSourceApplication {

	public static void main(String[] args) {
		SpringApplication.run(WebSourceApplication.class, "--spring.cloud.stream.source=toStream");
	}

	@Autowired
	private StreamBridge streamBridge;

	@RequestMapping
	@ResponseStatus(HttpStatus.ACCEPTED)
	public void delegateToSupplier(@RequestBody String body) {
		System.out.println("Sending " + body);
		streamBridge.send("toStream-out-0", body);
	}
}
Here we autowire a StreamBridge bean which allows us to send data to an output binding effectively bridging non-stream application with spring-cloud-stream. Note that preceding example does not have any source functions defined (e.g., Supplier bean) leaving the framework with no trigger to create source bindings, which would be typical for cases where configuration contains function beans. So to trigger the creation of source binding we use spring.cloud.stream.source property where you can declare the name of your sources. The provided name will be used as a trigger to create a source binding. So in the preceding example the name of the output binding will be toStream-out-0 which is consistent with the binding naming convention used by functions (see Binding and Binding names). You can use ; to signify multiple sources (e.g., --spring.cloud.stream.source=foo;bar)

参考官方文档改造后

生产者

这里直接贴出多主题的消费模式，实际生产中多数也是订阅多个主题

@SpringBootApplication
public class ProducerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ProducerApplication.class, args);
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 17:57
 */
@RestController
public class Producer {

 @Autowired
 private StreamBridge streamBridge;

 @RequestMapping("/send1")
 public String send1() {
 streamBridge.send("source1-out-0", new Date());
 return "success1";
 }

 @RequestMapping("/send2")
 public String send2() {
 streamBridge.send("source2-out-0", LocalDateTime.now().format(DateTimeFormatter.ISO_LOCAL_DATE_TIME));
 return "success2";
 }

}

配置文件

spring:
 rabbitmq:
 host: 192.168.133.128
 username: dev-user
 password: devpassword

 cloud:
 stream:
 bindings:
 source1-out-0:
 destination: test2
 source2-out-0:
 destination: test3
 function:
 definition: source1;source2


server:
 port: 8083

消费者

@SpringBootApplication
public class ConsumerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ConsumerApplication.class, args);
 }

 @Bean
 public Consumer<Date> sink1() {
 return System.out::println;
 }

 @Bean
 public Consumer<String> sink2() {
 return System.out::println;
 }
}

配置文件

spring:
 rabbitmq:
 host: 192.168.133.128
 username: dev-user
 password: devpassword

 cloud:
 stream:
 bindings:
 sink1-in-0:
 destination: test2
 sink2-in-0:
 destination: test3
 function:
 definition: sink1;sink2

server:
 port: 8081

运行结果

触发/send1

触发/send2

可见消息被正确的路由

总结

整体来看，函数式编程比命令式编程更加简洁，而且加上SpringCloud规约大于配置的特点，可以明显减少配置的代码量，是未来发展的趋势。

Mapstruct爬坑记录

Sat, 20 Feb 2021 22:45:11 +0800

背景

众所周知，Spring框架提供了著名的类拷贝工具BeanUtils，其使用简单、方便。

但是其内部实现基于运行时反射进行同名属性拷贝，对于需要进行字段映射和高性能运行时场景都无法满足要求，因此，开源社区衍生出了以下的类拷贝框架

使用

目前第二代密码管理平台采用的是mica重写后的BeanUtil类拷贝方法，整体来说性能可以满足要求，但为了进一步提升查询时类转换性能，同时考虑到应用中本身就包含了swagger依赖，而swagger中本身就引入了mapstruct依赖包，故无需引入新依赖。

因此，最终考虑将业务场景下的类拷贝改成Mapstruct实现，由此开始mapstruct爬坑之路。。

软件包版本

SpringBoot:2.4.2

SpringCloud:2020.0.1

lombok: 1.18.16

mapstruct: 1.3.1.Final

新增依赖

由于需要由mapstruct根据定义的接口自动生成类转换实现，故需要引入mapstruct接口处理包，有两种方式，一种通过maven插件的形式，一种通过maven依赖的形式

maven插件的形式（官方推荐）

<build>
 <plugins>
 <plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-compiler-plugin</artifactId>
 <version>3.5.1</version>
 <configuration>
 <source>1.8</source>
 <target>1.8</target>
 <annotationProcessorPaths>
 <path>
 <groupId>org.mapstruct</groupId>
 <artifactId>mapstruct-processor</artifactId>
 <version>${org.mapstruct.version}</version>
 </path>
 </annotationProcessorPaths>
 </configuration>
 </plugin>
 </plugins>
</build>

maven依赖的形式

<dependency>
 <groupId>org.mapstruct</groupId>
 <artifactId>mapstruct-processor</artifactId>
 <version>${mapstruct.version}</version>
</dependency>

考虑到平台的maven模块数量较大，互相依赖关系比较复杂，遗漏插件依赖容易导致难以排查的bug，最终采用第二种方式，减少编码量。

新增映射类

/**
 * @author F嘉阳
 * @date 2021/2/18 12:06
 */
@Mapper(componentModel = "spring", unmappedTargetPolicy = ReportingPolicy.IGNORE)
public interface OssEntityMapper {

	OssVO ossToOssVO(Oss oss);

	OssDTO ossToOssDto(Oss oss);

	/**
	 * bladeFile转Oss
	 *
	 * @param bladeFile
	 * @return
	 */
	@Mapping(target = "fileName", source = "name")
	Oss bladeFileToOss(BladeFile bladeFile);
}

新增单元测试

由于首次使用mapstruct，编写单元测试检测代码是否成功

/**
 * @author F嘉阳
 * @date 2021/2/18 14:23
 */
@Slf4j
@ExtendWith(BladeSpringExtension.class)
@SpringBootTest(classes = ResourceApplication.class)
@BladeBootTest(appName = AppConstant.APPLICATION_RESOURCE_NAME, profile = "test", enableLoader = true)
class OssEntityMapperTest {

	@Autowired
	private OssEntityMapper mapper;

	@Test
	void ossToOssVO() {
		Oss oss = new Oss();
		oss.setFileName("filename");
		oss.setLink("link");
		oss.setOriginalName("origin");
		oss.setOssType(OssType.QI_NIU_OSS);
		oss.setIsDeleted(BladeConstant.DB_NOT_DELETED);
		OssVO ossVO = mapper.ossToOssVO(oss);
		assertEquals("filename", ossVO.getFileName());
		assertEquals("link", ossVO.getLink());
		assertEquals("origin", ossVO.getOriginalName());
	}

	@Test
	void ossToOssDto() {
		Oss oss = new Oss();
		oss.setFileName("filename");
		oss.setLink("link");
		oss.setOriginalName("origin");
		oss.setOssType(OssType.QI_NIU_OSS);
		oss.setIsDeleted(BladeConstant.DB_NOT_DELETED);
		OssDTO ossDTO = mapper.ossToOssDto(oss);
		assertEquals("filename", ossDTO.getFileName());
		assertEquals("link", ossDTO.getLink());
		assertEquals("origin", ossDTO.getOriginalName());
	}

	@Test
	void bladeFileToOss() {
		BladeFile file = new BladeFile();
		file.setName("file");
		file.setLink("link");
		file.setOriginalName("origin");
		Oss oss = mapper.bladeFileToOss(file);
		assertEquals("file", oss.getFileName());
		assertEquals("link", oss.getLink());
		assertEquals("origin", oss.getOriginalName());
	}
}

编译结果

编译结果，报错

这就很奇怪了，这是按官网最简单的方式写的，而且IDEA也安装了mapstruct插件，也没报错，能正确识别属性，怎么编译就不通过了

查看生成的代码，确实没有属性映射

爬坑之路

由于编译报错，考虑以下方法解决

移除spring容器依赖，采用类引用的方式编写。——无效

移除属性映射，尝试编译，可以编译通过，但生成的代码没有同名属性映射，只会返回空值。——无效

@Override
public Oss bladeFileToOss(BladeFile bladeFile) {
 if ( bladeFile == null ) {
 return null;
 }

 Oss oss = new Oss();

 return oss;
}

更换类名后测试通过，多次测试仍然有不通过的情况。——可能和编译环境有关？
相同代码放在单独的工程（非maven多模块）测试通过。——可能和maven多模块依赖有关？
根据网上检索的结果，普遍反映的lombok版本太低导致的，经过检查，当前的lombok版本应该是满足最低版本要求的。——无效
考虑是否和spring某些组件冲突，单独新开maven多模块工程测试，依然测试不通过。——果然和maven有关？

解决方法

最终通过给官方仓库提交issue咨询得到答案

原来是lombok版本太高导致的。。

最终降低lombok版本问题解决。

GitHub DevOps工作流构建Java项目并上传到DockerHub

Sun, 07 Feb 2021 11:58:43 +0800

工程准备

此处使用fork自alibaba/Sentinel的FJiayang/Sentinel项目实践

工程地址

Sentinel — Sentinel仓库源码

准备Dockerfile

该工程为标准Java项目，故准备好的Dockerfile文件如下

FROM anapsix/alpine-java:8_server-jre_unlimited

MAINTAINER fjy8018@gmail.com

ENV PARAM ""
ENV VERSION 1.8.0

RUN mkdir -p /sentinel

WORKDIR /sentinel

ADD ./sentinel-dashboard/target/sentinel-dashboard.jar ./app-${VERSION}.jar

# 设置时区，默认为UTC
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN echo 'Asia/Shanghai' > /etc/timezone

ENTRYPOINT ["java", "${PARAM}", "-jar", "app-${VERSION}.jar"]

编写GitHub工作流

GitHub DevOps工作流又叫GitHub Actions 是2019年末推出的自动化CICD持续集成工作流，有详细的样例文档

此处参考以下文档编写

使用 Maven 构建和测试 Java — GitHub Actions 指南

安全配置

由于需要建构建后的制品推送到DockerHub中，但又不能在工作流中直接使用用户名和密码登陆，故还需要配置环境变量

在DockerHub中新增秘钥

在构建工程的设置中新增用户名和密码变量

编写后的工作流如下

name: Publish Docker image

on: [push]

jobs:
 build:

 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v2
 - name: Set up Maven Central Repository
 uses: actions/setup-java@v1
 with:
 java-version: 1.8

 - name: Publish package
 run: mvn --batch-mode clean package -Dmaven.test.skip=true

 - name: Push to Docker Hub
 uses: docker/build-push-action@v1
 with:
 username: ${{ secrets.DOCKER_USERNAME }}
 password: ${{ secrets.DOCKER_PASSWORD }}
 repository: fjy8018/sentinel
 tag_with_ref: true

触发构建

当前的触发方式配置为PUSH，修改好文件后会自动触发流水线工作

Maven打包，速度很快

打包完成

检查结果

登录到DockerHub中检查推送结果，可见成功推送

总结

整体来看，GitHub工作流效率高，文档齐全，编码上手速度快，相比GitLab CICD和DockerHub CICD来说，GitHub CICD配置更加简单、友好和高效。

RocketMQ学习笔记6——集群实战

Wed, 23 Dec 2020 19:15:00 +0800

组建高性能MQ服务

双主双从模式
保障无单点故障问题的高可用集群

集群拓扑图

IP	资源	服务
10.139.12.52	1C2G20GB	rocketmq-nameserver1、rocketmq-master1
10.139.12.53	1C2G20GB	rocketmq-nameserver2、rocketmq-master2
10.139.12.54	1C2G20GB	rocketmq-nameserver3、rocketmq-master1-slave
10.139.12.55	1C2G20GB	rocketmq-nameserver4、rocketmqmaster2-slave

配置文件最优设置解析

修改host文件映射

10.139.12.52 rocketmq-nameserver1 rocketmq-master1
10.139.12.53 rocketmq-nameserver2 rocketmq-master2
10.139.12.54 rocketmq-nameserver3 rocketmq-master1-slave
10.139.12.55 rocketmq-nameserver4 rocketmq-master2-slave

将4个节点的hosts均改为如上内容，并将4.4.0版本包放入环境中解压

创建存储目录

$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index

建立软连接

$ cd ~/rocketmq-test
$ ln -s rocketmq-all-4.4.0-bin-release rocketmq

修改配置文件

解压后默认的配置文件如下

[user@301 ~/rocketmq-test/rocketmq/conf]$ll
total 36
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync
drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave
-rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf
-rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml
-rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml
-rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml
-rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml
-rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml

若想保障生产环境高可用，推荐使用双主双从同步双写模式，可以让一个主节点对应多个从节点，但从节点越多，同步效率越慢

本次采用双主双从同步双写模式，一个主节点对应一个从节点

[user@301 ~/rocketmq-test/rocketmq/conf/2m-2s-sync]$ll
total 16
-rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-a.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties
-rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-b.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties

用途如下

IP	资源	服务	配置文件
10.139.12.52	1C2G20GB	rocketmq-nameserver1、rocketmq-master1	broker-a.properties
10.139.12.53	1C2G20GB	rocketmq-nameserver2、rocketmq-master2	broker-b.properties
10.139.12.54	1C2G20GB	rocketmq-nameserver3、rocketmq-slave1	broker-a-s.properties
10.139.12.55	1C2G20GB	rocketmq-nameserver4、rocketmq-slave2	broker-b-s.properties

`broker-a.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=false
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=false
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间，最大100%
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-b.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-b
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=false
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=false
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间，最大100%
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-a-s.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-b-s.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对
brokerName=broker-b
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

分发配置文件

$ scp broker-* ocmuser@rocketmq-master2-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync
$ scp broker-* ocmuser@rocketmq-master2:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync
$ scp broker-* ocmuser@rocketmq-master1-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync

修改日志配置文件

$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq/logs
$ cd /home/ocmuser/rocketmq-test/rocketmq/conf && sed -i 's#${user.home}#/home/ocmuser/rocketmq-test/#g' *.xml

修改脚本启动参数

/home/ocmuser/rocketmq-test/rocketmq/bin/runbroker.sh内容如下，至少保障1g内存

#======================================================================================
# 开发环境JVM Configuration
#======================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"

/home/ocmuser/rocketmq-test/rocketmq/bin/runserver.sh内容如下

#======================================================================================
# JVM Configuration
#======================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

启动 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin
$ nohup sh mqnamesrv &

启动 Master1:BrokerServerA

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master2:BrokerServerB

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master1-Slave:BrokerServerC

$ cd cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a-s.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master2-Slave:BrokerServerD

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b-s.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

查看进程和日志，可见部署完成

角色为同步双写，异步刷盘，4个节点通信正常

控制台监控验证

修改连接配置

rocketmq.config.namesrvAddr=10.139.12.51:9876;10.139.12.52:9876;10.139.12.53:9876;10.139.12.54:9876

连接正常

查看各节点角色和状态

服务停止

首先关闭 4 个 BrokerServer，再关闭 4 个 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ sh mqshutdown broker 
$ sh mqshutdown namesrv

双主双从环境构建与调优

为取得可靠性与效率的平衡，建议采用同步双写，异步刷盘的模式

RocketMQ学习笔记5——原理解析

Wed, 23 Dec 2020 19:13:00 +0800

消息存储结构

CommitLog：消息主体以及元数据的存储主体，存储Producer端写入的消息主体内容,消息内容不是定长的。消息主要是顺序写入日志文件，当文件满了，写入下一个文件；其日志信息科用于数据恢复，作为MQ数据保障兜底策略。
ConsumerQueue：消息消费队列，存储消息在commit log 中所在的这个位置，也可以说是一个偏移量.引入的目的主要是提高消息消费的性能。
IndexFile：IndexFile（索引文件）提供了一种可以通过key或时间区间来查询消息的方法。
每一个topic下的每一个message queue都对应一个ConsumerQueue文件，具体存储位置通过配置文件配置。
RocketMQ采用的是混合型的存储结构，即为Broker单个实例下所有的队列（ConsumerQueue）共用一个日志数据文件（即为CommitLog）来存储。
刷盘采用顺序写随机读的模式。
消费端采用“0拷贝”模式提升性能。

同步\异步刷盘

消息存储：内存+磁盘存储，两种刷盘方式

异步刷盘

能够充分利用OS的PageCache的优势，只要消息写入PageCache（内存）即可将成功的ACK（成功信息）返回给Producer端。
消息堆积到一定量才触发刷盘。
消息刷盘采用后台异步线程提交的方式进行，降低了读写延迟，提高了MQ的性能和吞吐量。

同步刷盘

只有在消息真正持久化至磁盘后RocketMQ的Broker端才会真正返回给Producer端一个成功的ACK响应。
同步刷盘对MQ消息可靠性来说是一种不错的保障，但是性能上会有较大影响，一般适用于金融业务应用该模式较多。

同步\异步复制

针对集群模式设计，同一组Broker有Master-Slave角色
推荐采用同步双写，异步复制的模式保持性能与可靠性的平衡

同步复制

Master与Slave都成功才返回ACK（成功信息）给Producer

异步复制

Master节点刷盘成功后立即返回ACK（成功信息）给Producer
再通过异步线程将信息同步给其他Slave节点
高性能，但存在数据丢失风险

同步双写

每个Master配置一个Slave，有多对Master-Slave，HA采用同步双写方式，即只有主备都写成功，才向应用返回成功，这种模式的优缺点如下：

优点：数据与服务都无单点故障，Master宕机情况下，消息无延迟，服务可用性与数据可用性都非常高；
缺点：性能比异步复制模式略低（大约低10%左右），发送单个消息的RT会略高，且目前版本在主节点宕机后，备机不能自动切换为主机。

高可用机制

Master-Slave搭配实现高可用机制
通过BrokerId区分节点
Master支持读写，Slave只读
当Master繁忙或者不可用时，可以自动切换到Slave读取消息

最佳实践

创建Topic时，将Topic的Message Queue创建到多个Broker组中，使得逻辑上高可用
由于RocketMQ不支持主从切换，故可以考虑使用监控程序监听主节点的有效性，若连接超时，则将Slave节点的配置改为Master并重启，升级为主节点，实现高可用。

NameServer协调者

NameServer是一个非常简单的Topic路由注册中心，其角色类似Dubbo中的zookeeper，支持Broker的动态注册与发现，是整个集群的状态服务器。
主要包括两个功能：
- Broker管理，NameServer接受Broker集群的注册信息并且保存下来作为路由信息的基本数据。然后提供心跳检测机制，检查Broker是否还存活；
- 路由信息管理，每个NameServer将保存关于Broker集群的整个路由信息和用于客户端查询的队列信息。然后Producer和Conumser通过NameServer就可以知道整个Broker集群的路由信息，从而进行消息的投递和消费。
NameServer通常也是集群的方式部署，各实例间相互不进行信息通讯。Broker是向每一台NameServer注册自己的路由信息，所以每一个NameServer实例上面都保存一份完整的路由信息。
当某个NameServer因某种原因下线了，Broker仍然可以向其它NameServer同步其路由信息，Producer,Consumer仍然可以动态感知Broker的路由的信息。

为什么不用zookeeper

RocketMQ 3.x以前使用zookeeper实现，后来改为nameserver实现
zookeeper提供主从选举，广播原子性等特性在MQ中用不上，功能冗余
zk使用主从自动切换提供高可用保障，nameserver使用多副本节点保障高可用
nameserver只提供路由等基本状态信息维护，代码量小，更加轻量，便于维护。

NameServer信息维护源码

package org.apache.rocketmq.namesrv.routeinfo;

public class RouteInfoManager {
 private final ReadWriteLock lock = new ReentrantReadWriteLock();
 private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;
 private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
 private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
 private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
 private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;
}

RocketMQ学习笔记4——消费者核心

Wed, 23 Dec 2020 19:11:00 +0800

配置参数详解

PushConsumer核心参数

ConsumeFromWhere源码，支持三种参数配置，默认是CONSUME_FROM_LAST_OFFSET

package org.apache.rocketmq.common.consumer;

public enum ConsumeFromWhere {
 /**
 * 第一次启动时从队列最后开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_LAST_OFFSET,

 @Deprecated
 CONSUME_FROM_LAST_OFFSET_AND_FROM_MIN_WHEN_BOOT_FIRST,
 @Deprecated
 CONSUME_FROM_MIN_OFFSET,
 @Deprecated
 CONSUME_FROM_MAX_OFFSET,
 /**
 * 从队列最开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_FIRST_OFFSET,
 /**
 * 指定具体时间点开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_TIMESTAMP,
}

AllocateMessageQueueStrategy接口用于集群模式下消息策略，默认配置为平均消息分配器AllocateMessageQueueAveragely

默认配置源码

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Constructor specifying consumer group and enabled msg trace flag.
 *
 * @param consumerGroup Consumer group.
 * @param enableMsgTrace Switch flag instance for message trace.
 */
 public DefaultMQPushConsumer(final String consumerGroup, boolean enableMsgTrace) {
 this(null, consumerGroup, null, new AllocateMessageQueueAveragely(), enableMsgTrace, null);
 }
}

subscription订阅参数，传递两个值，一个是主题，一个是表达式tag过滤

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Subscription relationship
 */
 private Map<String /* topic */, String /* sub expression */> subscription = new HashMap<String, String>();
}

样例：

consumer.subscribe("test_model_topic2", "TagA||TagB");

consumer.subscribe("test_model_topic2", "TagA");

consumer.subscribe("test_model_topic2", "*");

offsetStore配置核心偏移量，有两个实现，用于支持远程存储和本地存储，即支持集群和单机模式

consumeThreadMin与consumeThreadMax线程池自动调整参数，默认为20

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Minimum consumer thread number
 */
 private int consumeThreadMin = 20;

 /**
 * Max consumer thread number
 */
 private int consumeThreadMax = 20;
}

流控参数consumeConcurrentlyMaxSpan与pullThresholdForQueue

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Concurrently max span offset.it has no effect on sequential consumption
 * 单个队列并行消费的最大跨度
 */
 private int consumeConcurrentlyMaxSpan = 2000;

 /**
 * Flow control threshold on queue level, each message queue will cache at most 1000 messages by default,
 * Consider the {@code pullBatchSize}, the instantaneous value may exceed the limit
 * 一个队列最大的消息消费个数
 */
 private int pullThresholdForQueue = 1000;
}

消息拉取参数pullInterval与pullBatchSize

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Message pull Interval
 * 消息拉取的时间间隔
 */
 private long pullInterval = 0;

 /**
 * Batch pull size
 * 一次拉取的数据量
 */
 private int pullBatchSize = 32;
}

consumeMessageBatchMaxSize一次消息最大拉取的数据数量配置

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Batch consumption size
 * 一次消息最大拉取的数据数量
 */
 private int consumeMessageBatchMaxSize = 1;
}

MessageListener用于消息监听配置，包括并发、顺序读取等

MessageModel配置消息模式，广播或者集群，默认为集群模式

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Message model defines the way how messages are delivered to each consumer clients.
 * </p>
 *
 * RocketMQ supports two message models: clustering and broadcasting. If clustering is set, consumer clients with
 * the same {@link #consumerGroup} would only consume shards of the messages subscribed, which achieves load
 * balances; Conversely, if the broadcasting is set, each consumer client will consume all subscribed messages
 * separately.
 * </p>
 *
 * This field defaults to clustering.
 */
 private MessageModel messageModel = MessageModel.CLUSTERING;
}

集群与广播模式

集群模式

MessageModel源码支持两种模式，集群和广播

package org.apache.rocketmq.common.protocol.heartbeat;

/**
 * Message model
 */
public enum MessageModel {
 /**
 * broadcast
 */
 BROADCASTING("BROADCASTING"),
 /**
 * clustering（默认）
 */
 CLUSTERING("CLUSTERING");

 private String modeCN;

 MessageModel(String modeCN) {
 this.modeCN = modeCN;
 }

 public String getModeCN() {
 return modeCN;
 }
}

集群模式下GroupName用于把多个Consumer组织到一起，所以同一个集群，GroupName必须相同
相同的GroupName的Consumer只消费所订阅消息的一部分，即支持负载均衡
优势：集群模式下增加节点能动态增加负载均衡
一条消息只会被一个Consumer消费

样例

消息消费与奇偶性无关，随机负载均衡，并且负载均衡存在一定的差值，并非100%平分
为提高附在均衡度，可以用2个consume监听4个队列，或者3个consume监听6个队列
Producer只能保证平均分配到各个队列，不能保证被平均的消费

生产者

public class Producer {
	
 public static void main(String[] args) throws MQClientException, InterruptedException {
 	String group_name = "test_model_producer_name";
 DefaultMQProducer producer = new DefaultMQProducer(group_name);
 producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 producer.start();
 
 for (int i = 0; i < 10; i++) {
 try {
 	String tag = (i % 2 ==0) ? "TagA":"TagB";
 Message msg = new Message("test_model_topic2",// topic
 		tag,// tag
 ("信息内容" + i).getBytes()// body
 );
 SendResult sendResult = producer.send(msg);
 System.out.println(sendResult);
 }
 catch (Exception e) {
 e.printStackTrace();
 Thread.sleep(1000);
 }
 }
 
 producer.shutdown();
 }
}

消费者1

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.CLUSTERING);
//			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					//if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					//}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
		
	}
}

消费者2

public class Consumer2 {

	public Consumer2() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.CLUSTERING);
//			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					//if(tags.equals("TagB")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					//}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer2 c2 = new Consumer2();
		System.out.println("c2 start..");
		
	}
}

发送10条消息结果

消费者1结果

消费者2结果

注意，若使用集群模式，则tag表达式应该为*号，才能支持多consumer消费，若集群模式下指定tag表达式，则会导致消费结果不确定，原因如下

consumerA监听4条队列中的2条，consumerB监听4条队列中的另外2条。
Producer将包含TagA和TagB的消息均分到4条队列中。
consumerA消费TagA的消息，那么此时consumerA监听的2条队列中的TagB消息则永远不会被消费

样例

consumer1改成监听TagA，其他内容不变

consumer.subscribe("test_model_topic2", "TagA");

此时发送10条消息，consumer1监听的结果

此时将consumer2改成监听TagB并启动

consumer.subscribe("test_model_topic2", "TagB");

此时发送10条消息，consumer1监听的结果

consumer2监听结果为空，大量消息丢失

经过实测发现，只要集群模式下指定了tag，那么只有一个consumer能正常进行消费，其他consumer都为空

广播模式

同一个ConsumeGroup里的Consumer都消费订阅Topic全部信息
也就是一条消息会被每个Consume消费

样例

将consumer1和2改成广播模式，并监听所有tag

consumer.setMessageModel(MessageModel.BROADCASTING);
consumer.subscribe("test_model_topic2", "*");

发送10条消息consumer1消费结果

consumer2消费结果

注意广播模式下使用服务端tag过滤仍然有bug

将consumer1设置为只监听tagA消息

consumer.subscribe("test_model_topic2", "TagA");

consumer2设置为只监听tag2消息

consumer.subscribe("test_model_topic2", "TagB");

发送10条消息consumer1消费结果

consumer2消费结果

结果只有其中一个consumer能正常消费，与集群模式一样的问题

目前（4.4.0）若想要实现同一个topic的tag过滤，则只能让consumer监听所有消息并主动判断

样例：

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
	}
}

发送10条消息结果

消费者1结果

消费者2结果

对于consumer消费指定队列消息，推荐做法是将消息投递到专属的队列中，由同一个consumer监听，即相当于用groupname+topic唯一定位一个consumer，该方法集群与广播模式均适用

样例

此时consumer1变为test_model_consumer_name1

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name1";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "TagA");
//			consumer.setMessageModel(MessageModel.CLUSTERING);
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
	}
}

consumer2变为test_model_consumer_name2

public class Consumer2 {

	public Consumer2() {
		try {
			String group_name = "test_model_consumer_name2";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "TagB");
//			consumer.setMessageModel(MessageModel.CLUSTERING);
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagB")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer2 c2 = new Consumer2();
		System.out.println("c2 start..");
	}
}

发送10条消息结果

消费者1结果

消费者2结果

消费存储核心-OffSet

Offset是消费者消费进度的核心
Offset指某个topic下的一条消息在某个MessageQueue里的位置
通过Offset可以进行定位到这条消息
Offset的存储实现分为远程文件类型和本地文件类型两种

集群模式-`RemoteBrokerOffsetStore`

默认集群模式clustering，采用远程文件存储Offset
本质上因为多消费模式，每个Consumer消费所订阅主题的一部分

广播模式-`LocalFileOffsetStore`

广播模式下，由于每个Consumer都会收到消息且消费
各个Consumer之间没有任何干扰，独立线程消费
所以使用LocalFileOffsetStore，也就是把Offset存储到本地

PushConsumer长轮询模式分析

DefaultMQPushConsumer是使用长轮询模式进行实现的
通常主流的消息获取模式：Push消息推送模式与Pull消息拉取模式
Push模式消耗broker端性能，若消费端消费速度不一致，可能导致消息堆积
pull模式消耗消费端性能，消费端不知道消息达到时间
RocketMQ采用长轮询机制（消费端主动获取）

长轮询过程

客户端发起获取消息请求，假设一次请求超时时间为15s
服务端判断是否有消息，有则直接返回
服务端无消息，则等待5s（阈值），5s后再次检查是否有消息，有则返回，无则继续等待
15s后循环检查结束，若仍然无消息，则返回给客户端

优势：

主动权在客户端，当服务端消息堆积，不会全推送到客户端

缺点：

连接返回时间较长，占用客户端资源（请求阻塞，等待服务端返回）

源码分析

源码实现位置

PullRequestHoldService继承自Runnable接口

核心类PullRequestHoldService源码全貌

package org.apache.rocketmq.broker.longpolling;

public class PullRequestHoldService extends ServiceThread {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
 private static final String TOPIC_QUEUEID_SEPARATOR = "@";
 private final BrokerController brokerController;
 private final SystemClock systemClock = new SystemClock();
 private ConcurrentMap<String/* topic@queueId */, ManyPullRequest> pullRequestTable =
 new ConcurrentHashMap<String, ManyPullRequest>(1024);

 public PullRequestHoldService(final BrokerController brokerController) {
 this.brokerController = brokerController;
 }

 public void suspendPullRequest(final String topic, final int queueId, final PullRequest pullRequest) {
 String key = this.buildKey(topic, queueId);
 ManyPullRequest mpr = this.pullRequestTable.get(key);
 if (null == mpr) {
 mpr = new ManyPullRequest();
 ManyPullRequest prev = this.pullRequestTable.putIfAbsent(key, mpr);
 if (prev != null) {
 mpr = prev;
 }
 }

 mpr.addPullRequest(pullRequest);
 }

 private String buildKey(final String topic, final int queueId) {
 StringBuilder sb = new StringBuilder();
 sb.append(topic);
 sb.append(TOPIC_QUEUEID_SEPARATOR);
 sb.append(queueId);
 return sb.toString();
 }

 @Override
 public void run() {
 log.info("{} service started", this.getServiceName());
 while (!this.isStopped()) {
 try {
 // 判断是否启用长轮询
 if (this.brokerController.getBrokerConfig().isLongPollingEnable()) {
 // 等待5s
 this.waitForRunning(5 * 1000);
 } else {
 this.waitForRunning(this.brokerController.getBrokerConfig().getShortPollingTimeMills());
 }

 long beginLockTimestamp = this.systemClock.now();
 // 检查请求
 this.checkHoldRequest();
 long costTime = this.systemClock.now() - beginLockTimestamp;
 if (costTime > 5 * 1000) {
 log.info("[NOTIFYME] check hold request cost {} ms.", costTime);
 }
 } catch (Throwable e) {
 log.warn(this.getServiceName() + " service has exception. ", e);
 }
 }

 log.info("{} service end", this.getServiceName());
 }

 @Override
 public String getServiceName() {
 return PullRequestHoldService.class.getSimpleName();
 }

 private void checkHoldRequest() {
 // 遍历所有请求的channel
 for (String key : this.pullRequestTable.keySet()) {
 String[] kArray = key.split(TOPIC_QUEUEID_SEPARATOR);
 if (2 == kArray.length) {
 String topic = kArray[0];
 int queueId = Integer.parseInt(kArray[1]);
 final long offset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
 try {
 // 唤醒
 this.notifyMessageArriving(topic, queueId, offset);
 } catch (Throwable e) {
 log.error("check hold request failed. topic={}, queueId={}", topic, queueId, e);
 }
 }
 }
 }

 public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset) {
 notifyMessageArriving(topic, queueId, maxOffset, null, 0, null, null);
 }

 public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset, final Long tagsCode,
 long msgStoreTime, byte[] filterBitMap, Map<String, String> properties) {
 String key = this.buildKey(topic, queueId);
 // 取出等待的channel（请求）
 ManyPullRequest mpr = this.pullRequestTable.get(key);
 if (mpr != null) {
 List<PullRequest> requestList = mpr.cloneListAndClear();
 if (requestList != null) {
 List<PullRequest> replayList = new ArrayList<PullRequest>();
 // 循环所有的请求
 for (PullRequest request : requestList) {
 long newestOffset = maxOffset;
 if (newestOffset <= request.getPullFromThisOffset()) {
 newestOffset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
 }

 if (newestOffset > request.getPullFromThisOffset()) {
 boolean match = request.getMessageFilter().isMatchedByConsumeQueue(tagsCode,
 new ConsumeQueueExt.CqExtUnit(tagsCode, msgStoreTime, filterBitMap));
 // match by bit map, need eval again when properties is not null.
 if (match && properties != null) {
 match = request.getMessageFilter().isMatchedByCommitLog(null, properties);
 }
 // 如果匹配上
 if (match) {
 try {
 // 唤醒request
 this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
 request.getRequestCommand());
 } catch (Throwable e) {
 log.error("execute request when wakeup failed.", e);
 }
 continue;
 }
 }

 if (System.currentTimeMillis() >= (request.getSuspendTimestamp() + request.getTimeoutMillis())) {
 try {
 this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
 request.getRequestCommand());
 } catch (Throwable e) {
 log.error("execute request when wakeup failed.", e);
 }
 continue;
 }

 replayList.add(request);
 }

 if (!replayList.isEmpty()) {
 mpr.addPullRequest(replayList);
 }
 }
 }
 }
}

ManyPullRequest里存储的实际为PullRequest数组

public class ManyPullRequest {
 private final ArrayList<PullRequest> pullRequestList = new ArrayList<>();
	...
}

PullRequest实际存储NIO请求

public class PullRequest {
 private final RemotingCommand requestCommand;
 /** NIO channel */
 private final Channel clientChannel;
 private final long timeoutMillis;
 private final long suspendTimestamp;
 private final long pullFromThisOffset;
 private final SubscriptionData subscriptionData;
 private final MessageFilter messageFilter;
	...
}

`DefaultMQPullConsumer`消息拉取模式

实现类DefaultMQPullConsumer
主要逻辑
- 获取Message Queue并遍历
- 维护OffsetStore
- 根据不同消息状态做不同处理

生产者样例

public class Producer {
 public static void main(String[] args) throws MQClientException, InterruptedException {
 	
 	String group_name = "test_pull_producer_name";
 DefaultMQProducer producer = new DefaultMQProducer(group_name);
 producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 producer.start();
 
 for (int i = 0; i < 10; i++) {
 try {
 Message msg = new Message("test_pull_topic",// topic
 		"TagA",// tag
 		("Hello RocketMQ " + i).getBytes()// body
 );
 // 配置延迟
 msg.setDelayTimeLevel(2);
 
 SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
 // 外部方法的arg会传递到Object中
 @Override
 public MessageQueue select(List<MessageQueue> list, Message message, Object o) {
 Integer queueNumber = (Integer) o;
 return list.get(queueNumber);
 }
 },1);

 SendStatus status = sendResult.getSendStatus();
 System.out.println(status);

 Thread.sleep(1000);
 }
 catch (Exception e) {
 e.printStackTrace();
 Thread.sleep(3000);
 }
 }
 producer.shutdown();
 }
}

发送消息后的结果，在队列1有45条消息

使用普通Pull轮询模式

public class PullConsumer {
	//Map<key, value> key为指定的队列，value为这个队列拉取数据的最后位置
 private static final Map<MessageQueue, Long> offseTable = new HashMap<MessageQueue, Long>();
 
 public static void main(String[] args) throws MQClientException {
 	
 	String group_name = "test_pull_consumer_name";
 DefaultMQPullConsumer consumer = new DefaultMQPullConsumer(group_name);
 consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 consumer.start();
 System.err.println("consumer start");
 //	从TopicTest这个主题去获取所有的队列（默认会有4个队列）
 Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("test_pull_topic");
 //	遍历每一个队列，进行拉取数据
 for (MessageQueue mq : mqs) {
 System.out.println("Consume from the queue: " + mq);
 
 SINGLE_MQ: while (true) {
 try {
 	//	从queue中获取数据，从什么位置开始拉取数据 单次对多拉取32条记录
 PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
 System.out.println(pullResult);
 System.out.println(pullResult.getPullStatus());
 System.out.println();
 // 客户端自行维护Offset
 putMessageQueueOffset(mq, pullResult.getNextBeginOffset());
 switch (pullResult.getPullStatus()) {
	 case FOUND:
	 	List<MessageExt> list = pullResult.getMsgFoundList();
	 	for(MessageExt msg : list){
	 		System.out.println(new String(msg.getBody()));
	 	}
	 break;
	 case NO_MATCHED_MSG:
	 break;
	 case NO_NEW_MSG:
	 	System.out.println("没有新的数据啦...");
	 break SINGLE_MQ;
	 case OFFSET_ILLEGAL:
	 break;
	 default:
	 break;
 }
 }
 catch (Exception e) {
 e.printStackTrace();
 }
 }
 }
 consumer.shutdown();
 }
 
 
 private static void putMessageQueueOffset(MessageQueue mq, long offset) {
 offseTable.put(mq, offset);
 }
 
 
 private static long getMessageQueueOffset(MessageQueue mq) {
 Long offset = offseTable.get(mq);
 if (offset != null)
 return offset;
 return 0;
 }
}

消费结果，其会随机选择队列读取，导致服务端积压的消息得不到有效的消费

而且该模式需要客户端编码实现Offset存储

Pull定时任务实现样例

public class PullScheduleService {

 public static void main(String[] args) throws MQClientException {
 	
 	String group_name = "test_pull_consumer_name";
 	// 使用定时任务
 final MQPullConsumerScheduleService scheduleService = new MQPullConsumerScheduleService(group_name);
 
 scheduleService.getDefaultMQPullConsumer().setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 
 scheduleService.setMessageModel(MessageModel.CLUSTERING);
 
 scheduleService.registerPullTaskCallback("test_pull_topic", new PullTaskCallback() {
 
 @Override
 public void doPullTask(MessageQueue mq, PullTaskContext context) {
 MQPullConsumer consumer = context.getPullConsumer();
 System.err.println("-------------- queueId: " + mq.getQueueId() + "-------------");
 try {
 // 获取从哪里拉取
 long offset = consumer.fetchConsumeOffset(mq, false);
 if (offset < 0)
 offset = 0;
 
 PullResult pullResult = consumer.pull(mq, "*", offset, 32);
 switch (pullResult.getPullStatus()) {
 case FOUND:
 	List<MessageExt> list = pullResult.getMsgFoundList();
 	for(MessageExt msg : list){
 		//消费数据...
 		System.out.println(new String(msg.getBody()));
 	}
 break;
 case NO_MATCHED_MSG:
 break;
 case NO_NEW_MSG:
 case OFFSET_ILLEGAL:
 break;
 default:
 break;
 }
 // 客户端自动更新Offset，无需本地持久化维护
 consumer.updateConsumeOffset(mq, pullResult.getNextBeginOffset());
 // 设置再过3000ms后重新拉取
 context.setPullNextDelayTimeMillis(3000);
 
 }
 catch (Exception e) {
 e.printStackTrace();
 }
 }
 });
 
 scheduleService.start();
 }
}

运行结果，由于采用多线程读取所有队列，所以当服务端有消息时可以及时获取

实现原理

MQPullConsumerScheduleService中将PullTaskCallback方法作为Map存储
并使用ScheduledThreadPoolExecutor作为线程池，配置线程数为10
线程通过调度taskTable中的PullTaskImpl任务，实现方法回调

package org.apache.rocketmq.client.consumer;

public class MQPullConsumerScheduleService {
 private final InternalLogger log = ClientLogger.getLog();
 private final MessageQueueListener messageQueueListener = new MessageQueueListenerImpl();
 private final ConcurrentMap<MessageQueue, PullTaskImpl> taskTable =
 new ConcurrentHashMap<MessageQueue, PullTaskImpl>();
 private DefaultMQPullConsumer defaultMQPullConsumer;
 private int pullThreadNums = 20;
 private ConcurrentMap<String /* topic */, PullTaskCallback> callbackTable =
 new ConcurrentHashMap<String, PullTaskCallback>();
 private ScheduledThreadPoolExecutor scheduledThreadPoolExecutor;

 public MQPullConsumerScheduleService(final String consumerGroup) {
 this.defaultMQPullConsumer = new DefaultMQPullConsumer(consumerGroup);
 this.defaultMQPullConsumer.setMessageModel(MessageModel.CLUSTERING);
 }

 public void putTask(String topic, Set<MessageQueue> mqNewSet) {
 Iterator<Entry<MessageQueue, PullTaskImpl>> it = this.taskTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<MessageQueue, PullTaskImpl> next = it.next();
 if (next.getKey().getTopic().equals(topic)) {
 if (!mqNewSet.contains(next.getKey())) {
 next.getValue().setCancelled(true);
 it.remove();
 }
 }
 }

 for (MessageQueue mq : mqNewSet) {
 if (!this.taskTable.containsKey(mq)) {
 // 需要调度的任务
 PullTaskImpl command = new PullTaskImpl(mq);
 this.taskTable.put(mq, command);
 // 使用多线程调度
 this.scheduledThreadPoolExecutor.schedule(command, 0, TimeUnit.MILLISECONDS);

 }
 }
 }

 public void start() throws MQClientException {
 final String group = this.defaultMQPullConsumer.getConsumerGroup();
 this.scheduledThreadPoolExecutor = new ScheduledThreadPoolExecutor(
 this.pullThreadNums,
 new ThreadFactoryImpl("PullMsgThread-" + group)
 );

 this.defaultMQPullConsumer.setMessageQueueListener(this.messageQueueListener);

 this.defaultMQPullConsumer.start();

 log.info("MQPullConsumerScheduleService start OK, {} {}",
 this.defaultMQPullConsumer.getConsumerGroup(), this.callbackTable);
 }

 public void registerPullTaskCallback(final String topic, final PullTaskCallback callback) {
 // 放入回调任务集合
 this.callbackTable.put(topic, callback);
 this.defaultMQPullConsumer.registerMessageQueueListener(topic, null);
 }

 class MessageQueueListenerImpl implements MessageQueueListener {
 @Override
 public void messageQueueChanged(String topic, Set<MessageQueue> mqAll, Set<MessageQueue> mqDivided) {
 MessageModel messageModel =
 MQPullConsumerScheduleService.this.defaultMQPullConsumer.getMessageModel();
 switch (messageModel) {
 case BROADCASTING:
 MQPullConsumerScheduleService.this.putTask(topic, mqAll);
 break;
 case CLUSTERING:
 MQPullConsumerScheduleService.this.putTask(topic, mqDivided);
 break;
 default:
 break;
 }
 }
 }

 class PullTaskImpl implements Runnable {
 private final MessageQueue messageQueue;
 private volatile boolean cancelled = false;

 public PullTaskImpl(final MessageQueue messageQueue) {
 this.messageQueue = messageQueue;
 }

 @Override
 public void run() {
 String topic = this.messageQueue.getTopic();
 if (!this.isCancelled()) {
 // 获取回调任务
 PullTaskCallback pullTaskCallback =
 MQPullConsumerScheduleService.this.callbackTable.get(topic);
 if (pullTaskCallback != null) {
 final PullTaskContext context = new PullTaskContext();
 context.setPullConsumer(MQPullConsumerScheduleService.this.defaultMQPullConsumer);
 try {
 // 执行客户端实现的回调任务
 pullTaskCallback.doPullTask(this.messageQueue, context);
 } catch (Throwable e) {
 context.setPullNextDelayTimeMillis(1000);
 log.error("doPullTask Exception", e);
 }

 if (!this.isCancelled()) {
 MQPullConsumerScheduleService.this.scheduledThreadPoolExecutor.schedule(this,
 context.getPullNextDelayTimeMillis(), TimeUnit.MILLISECONDS);
 } else {
 log.warn("The Pull Task is cancelled after doPullTask, {}", messageQueue);
 }
 } else {
 log.warn("Pull Task Callback not exist , {}", topic);
 }
 } else {
 log.warn("The Pull Task is cancelled, {}", messageQueue);
 }
 }
 }
}

RocketMQ学习笔记3——生产者核心

Wed, 23 Dec 2020 19:09:00 +0800

配置参数

producerGroup：组名
createTopicKey：创建Topic（不允许使用，应当由架构设计定义）
defaultTopicQueueNums：创建topic时默认的队列数量（默认为4）
sendMsgTimeout：发送消息时的超时时间（默认值：3000，单位：毫秒）
compressMsgBodyOverHowmuch ：消息压缩字节（默认4096）
retryTimesWhenSendFailed ：同步模式下，在返回发送失败之前，内部尝试重新发送消息的最大次数。（默认值：2）
retryTimesWhenSendAsyncFailed：异步模式下，在发送失败之前，内部尝试重新发送消息的最大次数。（默认值：2）
retryAnotherBrokerWhenNotStoreOK：同步模式下，消息保存失败时是否重试其他broker 。（默认值：false ）
maxMessageSize：最大消息限制。（默认值：1024 * 1024 * 4，单位：字节）

主从同步机制

主从同步
同步信息：数据内容+元数据信息
元数据（配置信息）同步：Broker角色识别，为Slave则启动同步任务
- 通过Netty传输
- 实质为定时任务
消息同步（通过Socket）：HAService、HAconnection、WaitNotifyObject
- 通过原生Socket传输，性能高，数据包支持容量大

元数据同步源码分析

元信息同步工程源码位于broker中

发起定时任务源码BrokerController

package org.apache.rocketmq.broker;

public class BrokerController {
 public boolean initialize() throws CloneNotSupportedException {
 ...
 if (result) {
 ...
 // 进行角色判断
 if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) {
 if (this.messageStoreConfig.getHaMasterAddress() != null && this.messageStoreConfig.getHaMasterAddress().length() >= 6) {
 this.messageStore.updateHaMasterAddress(this.messageStoreConfig.getHaMasterAddress());
 this.updateMasterHAServerAddrPeriodically = false;
 } else {
 this.updateMasterHAServerAddrPeriodically = true;
 }
 // 启动定时任务进行元信息同步
 this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

 @Override
 public void run() {
 try {
 BrokerController.this.slaveSynchronize.syncAll();
 } catch (Throwable e) {
 log.error("ScheduledTask syncAll slave exception", e);
 }
 }
 }, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);
 } else {
 this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

 @Override
 public void run() {
 try {
 BrokerController.this.printMasterAndSlaveDiff();
 } catch (Throwable e) {
 log.error("schedule printMasterAndSlaveDiff error.", e);
 }
 }
 }, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);
 }
 }
 ...
 }
}

syncAll方法实现

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 public void syncAll() {
 // 同步Topic配置信息
 this.syncTopicConfig();
 // 同步消费者偏移量
 this.syncConsumerOffset();
 // 同步延迟偏移量
 this.syncDelayOffset();
 // 同步订阅组配置信息
 this.syncSubscriptionGroupConfig();
 }
 ...
}

同步Topic配置信息源码

syncTopicConfig方法实现如下

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 private void syncTopicConfig() {
 String masterAddrBak = this.masterAddr;
 if (masterAddrBak != null) {
 try {
 TopicConfigSerializeWrapper topicWrapper =
 this.brokerController.getBrokerOuterAPI().getAllTopicConfig(masterAddrBak);
 if (!this.brokerController.getTopicConfigManager().getDataVersion()
 .equals(topicWrapper.getDataVersion())) {

 this.brokerController.getTopicConfigManager().getDataVersion()
 .assignNewOne(topicWrapper.getDataVersion());
 this.brokerController.getTopicConfigManager().getTopicConfigTable().clear();
 this.brokerController.getTopicConfigManager().getTopicConfigTable()
 .putAll(topicWrapper.getTopicConfigTable());
 this.brokerController.getTopicConfigManager().persist();

 log.info("Update slave topic config from master, {}", masterAddrBak);
 }
 } catch (Exception e) {
 log.error("SyncTopicConfig Exception, {}", masterAddrBak, e);
 }
 }
 }
 ...
}

其中getAllTopicConfig方法实现如下

package org.apache.rocketmq.broker.out;

public class BrokerOuterAPI {
 ...
 public TopicConfigSerializeWrapper getAllTopicConfig(
 final String addr) throws RemotingConnectException, RemotingSendRequestException,
 RemotingTimeoutException, InterruptedException, MQBrokerException {
 RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_TOPIC_CONFIG, null);
 // 发起远程同步调用
 RemotingCommand response = this.remotingClient.invokeSync(MixAll.brokerVIPChannel(true, addr), request, 3000);
 assert response != null;
 switch (response.getCode()) {
 case ResponseCode.SUCCESS: {
 return TopicConfigSerializeWrapper.decode(response.getBody(), TopicConfigSerializeWrapper.class);
 }
 default:
 break;
 }

 throw new MQBrokerException(response.getCode(), response.getRemark());
 }
 ...
}

invokeSync通过调用封装过的Netty客户端实现

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingClient extends NettyRemotingAbstract implements RemotingClient {
 ...
 /**
 * 同步请求
 * @param addr 请求地址
 * @param request 请求命令
 * @param timeoutMillis 超时时间
 */
 @Override
 public RemotingCommand invokeSync(String addr, final RemotingCommand request, long timeoutMillis)
 throws InterruptedException, RemotingConnectException, RemotingSendRequestException, RemotingTimeoutException {
 long beginStartTime = System.currentTimeMillis();
 final Channel channel = this.getAndCreateChannel(addr);
 if (channel != null && channel.isActive()) {
 try {
 doBeforeRpcHooks(addr, request);
 // 算出极限耗时代价
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTime) {
 // 如果同步等待时长 < 极限耗时 说明请求不成立，抛出异常
 throw new RemotingTimeoutException("invokeSync call timeout");
 }
 // 同步请求实现
 RemotingCommand response = this.invokeSyncImpl(channel, request, timeoutMillis - costTime);
 doAfterRpcHooks(RemotingHelper.parseChannelRemoteAddr(channel), request, response);
 return response;
 } catch (RemotingSendRequestException e) {
 log.warn("invokeSync: send request exception, so close the channel[{}]", addr);
 this.closeChannel(addr, channel);
 throw e;
 } catch (RemotingTimeoutException e) {
 if (nettyClientConfig.isClientCloseSocketIfTimeout()) {
 this.closeChannel(addr, channel);
 log.warn("invokeSync: close socket because of timeout, {}ms, {}", timeoutMillis, addr);
 }
 log.warn("invokeSync: wait response timeout exception, the channel[{}]", addr);
 throw e;
 }
 } else {
 this.closeChannel(addr, channel);
 throw new RemotingConnectException(addr);
 }
 }
 ...
}

invokeSyncImpl实现逻辑

package org.apache.rocketmq.remoting.netty;

public abstract class NettyRemotingAbstract {
 ...
 public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
 final long timeoutMillis)
 throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
 final int opaque = request.getOpaque();

 try {
 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
 this.responseTable.put(opaque, responseFuture);
 final SocketAddress addr = channel.remoteAddress();
 // 使用netty channel
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 // 轮训监听
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 } else {
 responseFuture.setSendRequestOK(false);
 }
 // 异常情况
 responseTable.remove(opaque);
 responseFuture.setCause(f.cause());
 responseFuture.putResponse(null);
 log.warn("send a request command to channel <" + addr + "> failed.");
 }
 });

 RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
 if (null == responseCommand) {
 if (responseFuture.isSendRequestOK()) {
 throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
 responseFuture.getCause());
 } else {
 throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
 }
 }

 return responseCommand;
 } finally {
 this.responseTable.remove(opaque);
 }
 }
	...
}

同步消费者偏移量源码

其中syncConsumerOffset方法实现如下

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 private void syncConsumerOffset() {
 String masterAddrBak = this.masterAddr;
 if (masterAddrBak != null) {
 try {
 ConsumerOffsetSerializeWrapper offsetWrapper =
 this.brokerController.getBrokerOuterAPI().getAllConsumerOffset(masterAddrBak);
 this.brokerController.getConsumerOffsetManager().getOffsetTable()
 .putAll(offsetWrapper.getOffsetTable());
 this.brokerController.getConsumerOffsetManager().persist();
 log.info("Update slave consumer offset from master, {}", masterAddrBak);
 } catch (Exception e) {
 log.error("SyncConsumerOffset Exception, {}", masterAddrBak, e);
 }
 }
 }
 ...
}

getAllConsumerOffset实现逻辑

package org.apache.rocketmq.broker.out;

public class BrokerOuterAPI {
 ...
 public ConsumerOffsetSerializeWrapper getAllConsumerOffset(
 final String addr) throws InterruptedException, RemotingTimeoutException,
 RemotingSendRequestException, RemotingConnectException, MQBrokerException {
 // 封装CMD指令向主节点发起同步offset信息请求，同步时间上限为3000ms
 RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_CONSUMER_OFFSET, null);
 RemotingCommand response = this.remotingClient.invokeSync(addr, request, 3000);
 assert response != null;
 switch (response.getCode()) {
 case ResponseCode.SUCCESS: {
 return ConsumerOffsetSerializeWrapper.decode(response.getBody(), ConsumerOffsetSerializeWrapper.class);
 }
 default:
 break;
 }

 throw new MQBrokerException(response.getCode(), response.getRemark());
 }
 ...
}

封装的CMD指令

package org.apache.rocketmq.common.protocol;

public class RequestCode {
 ...
 public static final int NOTIFY_CONSUMER_IDS_CHANGED = 40;

 public static final int LOCK_BATCH_MQ = 41;

 public static final int UNLOCK_BATCH_MQ = 42;
 public static final int GET_ALL_CONSUMER_OFFSET = 43;

 public static final int GET_ALL_DELAY_OFFSET = 45;
 ...
}

远程命令对象构建，构建后通过remotingClient发送即可

package org.apache.rocketmq.remoting.protocol;

public class RemotingCommand {
 ...
	public static RemotingCommand createRequestCommand(int code, CommandCustomHeader customHeader) {
 RemotingCommand cmd = new RemotingCommand();
 cmd.setCode(code);
 cmd.customHeader = customHeader;
 setCmdVersion(cmd);
 return cmd;
 }
 ...
}

消息同步源码

消息同步工程源码位于store中

HAConnection实现消息读写

HAservice包含节点连接实现HAClient

逻辑

Master节点：

AcceptSocketService：接收Slave节点连接
HAConnection：
- ReadSocketService：读取来自Slave节点的数据
- WriteSocketService：写入Slave节点的数据

Slave节点：

HAService：
- HAClient：对Master节点连接，读写数据

通信协议：Master节点与Slave节点通信协议较简单，主要如下

对象	用途	第几位	字段	数据类型	字节数	说明
`Slave`=>`Master`	上报`CommitLog`已经同步的物理位置	0	`maxPhyOffset`	Long	8	`CommitLog`最大物理位置
`Master`=>`Slave`	传输新的`CommitLog`数据	0	`fromPhyOffset`	Long	8	传输`CommitLog`开始的物理位置
		1	`size`	Int	4	传输`CommitLog`数据长度
		2	`body`	Bytes		传输`CommitLog`数据

连接Master源码

package org.apache.rocketmq.store.ha;

public class HAService {
 ...
 class HAClient extends ServiceThread {
 /**
 * 客户端连接Master，使用NIO函数，提高效率
 * @return
 * @throws ClosedChannelException
 */
 private boolean connectMaster() throws ClosedChannelException {
 if (null == socketChannel) {
 String addr = this.masterAddress.get();
 if (addr != null) {

 SocketAddress socketAddress = RemotingUtil.string2SocketAddress(addr);
 if (socketAddress != null) {
 this.socketChannel = RemotingUtil.connect(socketAddress);
 if (this.socketChannel != null) {
 this.socketChannel.register(this.selector, SelectionKey.OP_READ);
 }
 }
 }

 this.currentReportedOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();

 this.lastWriteTimestamp = System.currentTimeMillis();
 }

 return this.socketChannel != null;
 }
 
 /**
 * Client(Slave)主循环，实现不断从Master传输CommitLog数据
 * 上传Master自己本地的CommitLog已经同步物理位置
 */
 @Override
 public void run() {
 log.info(this.getServiceName() + " service started");

 while (!this.isStopped()) {
 try {
 if (this.connectMaster()) {

 if (this.isTimeToReportOffset()) {
 // 满足上报间隔，上报到Master进度
 // 固定间隔（默认为5s）向Master上报Slave本地CommitLog已经同步到的物理位置
 boolean result = this.reportSlaveMaxOffset(this.currentReportedOffset);
 if (!result) {
 this.closeMaster();
 }
 }

 this.selector.select(1000);
 // 处理Master传输给Slave的CommitLog数据
 boolean ok = this.processReadEvent();
 if (!ok) {
 this.closeMaster();
 }
 // 若进度有变化，上报到Master进度
 if (!reportSlaveMaxOffsetPlus()) {
 continue;
 }
 // Master超时未返回，关闭连接
 long interval =
 HAService.this.getDefaultMessageStore().getSystemClock().now()
 - this.lastWriteTimestamp;
 if (interval > HAService.this.getDefaultMessageStore().getMessageStoreConfig()
 .getHaHousekeepingInterval()) {
 log.warn("HAClient, housekeeping, found this connection[" + this.masterAddress
 + "] expired, " + interval);
 this.closeMaster();
 log.warn("HAClient, master not response some time, so close connection");
 }
 } else {
 this.waitForRunning(1000 * 5);
 }
 } catch (Exception e) {
 log.warn(this.getServiceName() + " service has exception. ", e);
 this.waitForRunning(1000 * 5);
 }
 }

 log.info(this.getServiceName() + " service end");
 }
 ...
 }
 ...
}

RemotingUtil使用原生Socket实现

package org.apache.rocketmq.remoting.common;

public class RemotingUtil {
 public static SocketAddress string2SocketAddress(final String addr) {
 String[] s = addr.split(":");
 InetSocketAddress isa = new InetSocketAddress(s[0], Integer.parseInt(s[1]));
 return isa;
 }
 
 public static SocketChannel connect(SocketAddress remote, final int timeoutMillis) {
 SocketChannel sc = null;
 try {
 // 打开channel
 sc = SocketChannel.open();
 // 设置同步阻塞，可能存在bug
 sc.configureBlocking(true);
 // 设置关闭Socket的延迟事件，那么当线程执行Socket的close方法时，会进入阻塞状态
 // 直到Socket发送完所有剩余数据，或者超过了方法设置的延迟时间，才从close方法返回
 sc.socket().setSoLinger(false, -1);
 // 禁止使用Nagle算法，使用小数据传输
 // Nagle算法通过将缓冲区的小封包自动相连，组成较大的包
 sc.socket().setTcpNoDelay(true);
 // 设置接收缓冲区
 sc.socket().setReceiveBufferSize(1024 * 64);
 // 设置发送缓冲区
 sc.socket().setSendBufferSize(1024 * 64);
 // 连接
 sc.socket().connect(remote, timeoutMillis);
 // bug？又设置非同步阻塞了
 sc.configureBlocking(false);
 return sc;
 } catch (Exception e) {
 if (sc != null) {
 try {
 sc.close();
 } catch (IOException e1) {
 e1.printStackTrace();
 }
 }
 }

 return null;
 }
 ...
}

核心读请求实现

package org.apache.rocketmq.store.ha;

public class HAService {
 ...
 class HAClient extends ServiceThread {
 private boolean processReadEvent() {
 int readSizeZeroTimes = 0;
 while (this.byteBufferRead.hasRemaining()) {
 try {
 int readSize = this.socketChannel.read(this.byteBufferRead);
 if (readSize > 0) {
 lastWriteTimestamp = HAService.this.defaultMessageStore.getSystemClock().now();
 readSizeZeroTimes = 0;
 boolean result = this.dispatchReadRequest();
 if (!result) {
 log.error("HAClient, dispatchReadRequest error");
 return false;
 }
 } else if (readSize == 0) {
 if (++readSizeZeroTimes >= 3) {
 break;
 }
 } else {
 log.info("HAClient, processReadEvent read socket < 0");
 return false;
 }
 } catch (IOException e) {
 log.info("HAClient, processReadEvent read socket exception", e);
 return false;
 }
 }

 return true;
 }
 
 /**
 * 核心读请求实现
 * 
 * 读取Master传输的CommitLog数据，并返回是否正确
 * 如果读取到数据，写入CommitLog
 * 异常原因：
 * 1. Master传输来的数据开始位置offset(masterPhyOffset)不等于Slave的CommitLog数最大offset(slavePhyOffset)
 * 2. 上报到Master进度失败
 * @return 是否正确
 */
 private boolean dispatchReadRequest() {
 final int msgHeaderSize = 8 + 4; // phyoffset + size
 int readSocketPos = this.byteBufferRead.position();

 while (true) {
 // 读取到请求数据
 int diff = this.byteBufferRead.position() - this.dispatchPostion;
 if (diff >= msgHeaderSize) {
 // 读取masterPhyOffset、bodySize
 long masterPhyOffset = this.byteBufferRead.getLong(this.dispatchPostion);
 int bodySize = this.byteBufferRead.getInt(this.dispatchPostion + 8);
 // 设置Slave节点CommitLog文件最大的offset位置
 long slavePhyOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();
 // 校验Master传输来的数据开始位置offset和slave的commitLog数最大offset是否相同
 if (slavePhyOffset != 0) {
 if (slavePhyOffset != masterPhyOffset) {
 log.error("master pushed offset not equal the max phy offset in slave, SLAVE: "
 + slavePhyOffset + " MASTER: " + masterPhyOffset);
 return false;
 }
 }
 // 读取消息
 if (diff >= (msgHeaderSize + bodySize)) {
 // 写入CommitLog
 byte[] bodyData = new byte[bodySize];
 this.byteBufferRead.position(this.dispatchPostion + msgHeaderSize);
 this.byteBufferRead.get(bodyData);

 HAService.this.defaultMessageStore.appendToCommitLog(masterPhyOffset, bodyData);
 // 设置处理到的位置
 this.byteBufferRead.position(readSocketPos);
 this.dispatchPostion += msgHeaderSize + bodySize;
 // 上报到Master进度
 if (!reportSlaveMaxOffsetPlus()) {
 return false;
 }
 // 继续读数据
 continue;
 }
 }
 // 空间写满，重新分配空间
 if (!this.byteBufferRead.hasRemaining()) {
 this.reallocateByteBuffer();
 }

 break;
 }

 return true;
 }
 ...
 }
 ...
}

同步\异步消息发送

实现分类

同步消息发送
- 发送：producer.send(Message msg)
- 核心实现：DefaultMQProducerImpl
异步消息发送
- 发送：producer.send(Message msg,SendCallback sendCallback)
- 核心实现：DefaultMQProducerImpl

实例

public class Producer {

	public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
		
		DefaultMQProducer producer = new DefaultMQProducer("test_quick_producer_name");
		
	producer.setNamesrvAddr(Const.NAMESRV_ADDR_MASTER_SLAVE);
		
		producer.start();
		
		for(int i = 0 ; i <5; i ++) {
			//	1.	创建消息
			Message message = new Message("test_quick_topic",	//	主题
					"TagA", //	标签
					"key" + i,	// 	用户自定义的key ,唯一的标识
					("Hello RocketMQ" + i).getBytes());	//	消息内容实体（byte[]）
			// 异步发送消息
			producer.send(message, new SendCallback() {
				@Override
				public void onSuccess(SendResult sendResult) {
					System.err.println("msgId: " + sendResult.getMsgId() + ", status: " + sendResult.getSendStatus());
					producer.shutdown();
				}
				@Override
				public void onException(Throwable e) {
					e.printStackTrace();
					System.err.println("------发送失败");
				}
			});
		}	
	}
}

同步消息发送源码分析

发送消息方法

package org.apache.rocketmq.client.producer;

public class DefaultMQProducer extends ClientConfig implements MQProducer {
 ...
 @Override
 public SendResult send(Message msg,
 long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 return this.defaultMQProducerImpl.send(msg, timeout);
 }
}

模式枚举

package org.apache.rocketmq.client.impl;

public enum CommunicationMode {
 SYNC,
 ASYNC,
 // 单向，不需要响应
 ONEWAY,
}

消息发送实现类

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public SendResult send(Message msg,
 long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 return this.sendDefaultImpl(msg, CommunicationMode.SYNC, null, timeout);
 }
 
 private SendResult sendDefaultImpl(
 Message msg,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final long timeout
 ) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 // 校验运行状态
 this.makeSureStateOK();
 // 检查消息格式
 Validators.checkMessage(msg, this.defaultMQProducer);
 // 回调ID，用户标记是否同一次消息
 final long invokeID = random.nextLong();
 // 开始发送消息的时间
 long beginTimestampFirst = System.currentTimeMillis();
 long beginTimestampPrev = beginTimestampFirst;
 long endTimestamp = beginTimestampFirst;
 // 查找topic
 TopicPublishInfo topicPublishInfo = this.tryToFindTopicPublishInfo(msg.getTopic());
 if (topicPublishInfo != null && topicPublishInfo.ok()) {
 boolean callTimeout = false;
 // 要发送的消息队列
 MessageQueue mq = null;
 Exception exception = null;
 SendResult sendResult = null;
 // 重试次数，从配置参数中读取
 int timesTotal = communicationMode == CommunicationMode.SYNC ? 1 + this.defaultMQProducer.getRetryTimesWhenSendFailed() : 1;
 int times = 0;
 // 存储每次发送消息选择的broker名称
 String[] brokersSent = new String[timesTotal];
 for (; times < timesTotal; times++) {
 // 最后一次消息发送的brokerName进行存储
 String lastBrokerName = null == mq ? null : mq.getBrokerName();
 // 消息队列选择策略
 MessageQueue mqSelected = this.selectOneMessageQueue(topicPublishInfo, lastBrokerName);
 if (mqSelected != null) {
 mq = mqSelected;
 brokersSent[times] = mq.getBrokerName();
 try {
 beginTimestampPrev = System.currentTimeMillis();
 // 计算代价
 long costTime = beginTimestampPrev - beginTimestampFirst;
 if (timeout < costTime) {
 callTimeout = true;
 break;
 }
 // 消息发送实现
 sendResult = this.sendKernelImpl(msg, mq, communicationMode, sendCallback, topicPublishInfo, timeout - costTime);
 endTimestamp = System.currentTimeMillis();
 // 更新broker可用信息（容错）
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
 switch (communicationMode) {
 case ASYNC:
 return null;
 case ONEWAY:
 return null;
 case SYNC:
 // 同步发送等待返回结果
 if (sendResult.getSendStatus() != SendStatus.SEND_OK) {
 if (this.defaultMQProducer.isRetryAnotherBrokerWhenNotStoreOK()) {
 continue;
 }
 }

 return sendResult;
 default:
 break;
 }
 } catch (RemotingException e) {
 endTimestamp = System.currentTimeMillis();
 // 抛异常则更新broker容错信息
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 continue;
 } catch (MQClientException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 continue;
 } catch (MQBrokerException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 // 抛出如下异常则进行重试
 switch (e.getResponseCode()) {
 case ResponseCode.TOPIC_NOT_EXIST:
 case ResponseCode.SERVICE_NOT_AVAILABLE:
 case ResponseCode.SYSTEM_ERROR:
 case ResponseCode.NO_PERMISSION:
 case ResponseCode.NO_BUYER_ID:
 case ResponseCode.NOT_IN_CURRENT_UNIT:
 continue;
 default:
 if (sendResult != null) {
 return sendResult;
 }

 throw e;
 }
 } catch (InterruptedException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
 log.warn(String.format("sendKernelImpl exception, throw exception, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());

 log.warn("sendKernelImpl exception", e);
 log.warn(msg.toString());
 throw e;
 }
 } else {
 break;
 }
 }

 if (sendResult != null) {
 return sendResult;
 }

 String info = String.format("Send [%d] times, still failed, cost [%d]ms, Topic: %s, BrokersSent: %s",
 times,
 System.currentTimeMillis() - beginTimestampFirst,
 msg.getTopic(),
 Arrays.toString(brokersSent));

 info += FAQUrl.suggestTodo(FAQUrl.SEND_MSG_FAILED);

 MQClientException mqClientException = new MQClientException(info, exception);
 if (callTimeout) {
 throw new RemotingTooMuchRequestException("sendDefaultImpl call timeout");
 }

 if (exception instanceof MQBrokerException) {
 mqClientException.setResponseCode(((MQBrokerException) exception).getResponseCode());
 } else if (exception instanceof RemotingConnectException) {
 mqClientException.setResponseCode(ClientErrorCode.CONNECT_BROKER_EXCEPTION);
 } else if (exception instanceof RemotingTimeoutException) {
 mqClientException.setResponseCode(ClientErrorCode.ACCESS_BROKER_TIMEOUT);
 } else if (exception instanceof MQClientException) {
 mqClientException.setResponseCode(ClientErrorCode.BROKER_NOT_EXIST_EXCEPTION);
 }

 throw mqClientException;
 }

 List<String> nsList = this.getmQClientFactory().getMQClientAPIImpl().getNameServerAddressList();
 if (null == nsList || nsList.isEmpty()) {
 throw new MQClientException(
 "No name server address, please set it." + FAQUrl.suggestTodo(FAQUrl.NAME_SERVER_ADDR_NOT_EXIST_URL), null).setResponseCode(ClientErrorCode.NO_NAME_SERVER_EXCEPTION);
 }

 throw new MQClientException("No route info of this topic, " + msg.getTopic() + FAQUrl.suggestTodo(FAQUrl.NO_TOPIC_ROUTE_INFO),
 null).setResponseCode(ClientErrorCode.NOT_FOUND_TOPIC_EXCEPTION);
 }
}

查找topic方法实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 private TopicPublishInfo tryToFindTopicPublishInfo(final String topic) {
 // 使用Map缓存
 TopicPublishInfo topicPublishInfo = this.topicPublishInfoTable.get(topic);
 if (null == topicPublishInfo || !topicPublishInfo.ok()) {
 // 当无可用topic发布信息时，主动向NameServer拉取一次
 this.topicPublishInfoTable.putIfAbsent(topic, new TopicPublishInfo());
 this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
 topicPublishInfo = this.topicPublishInfoTable.get(topic);
 }
 // 若可用
 if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) {
 return topicPublishInfo;
 } else {
 // 否则根据设置判断是否新建Topic
 this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, true, this.defaultMQProducer);
 topicPublishInfo = this.topicPublishInfoTable.get(topic);
 return topicPublishInfo;
 }
 }
}

客户端获取路由信息实现

package org.apache.rocketmq.client.impl.factory;

public class MQClientInstance {
 public boolean updateTopicRouteInfoFromNameServer(final String topic) {
 return updateTopicRouteInfoFromNameServer(topic, false, null);
 }
 
 public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault,
 DefaultMQProducer defaultMQProducer) {
 try {
 if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
 try {
 TopicRouteData topicRouteData;
 if (isDefault && defaultMQProducer != null) {
 topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(),
 1000 * 3);
 if (topicRouteData != null) {
 for (QueueData data : topicRouteData.getQueueDatas()) {
 int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
 data.setReadQueueNums(queueNums);
 data.setWriteQueueNums(queueNums);
 }
 }
 } else {
 topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, 1000 * 3);
 }
 if (topicRouteData != null) {
 TopicRouteData old = this.topicRouteTable.get(topic);
 boolean changed = topicRouteDataIsChange(old, topicRouteData);
 if (!changed) {
 changed = this.isNeedUpdateTopicRouteInfo(topic);
 } else {
 log.info("the topic[{}] route info changed, old[{}] ,new[{}]", topic, old, topicRouteData);
 }

 if (changed) {
 TopicRouteData cloneTopicRouteData = topicRouteData.cloneTopicRouteData();

 for (BrokerData bd : topicRouteData.getBrokerDatas()) {
 this.brokerAddrTable.put(bd.getBrokerName(), bd.getBrokerAddrs());
 }

 // Update Pub info
 {
 TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData);
 publishInfo.setHaveTopicRouterInfo(true);
 Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<String, MQProducerInner> entry = it.next();
 MQProducerInner impl = entry.getValue();
 if (impl != null) {
 impl.updateTopicPublishInfo(topic, publishInfo);
 }
 }
 }

 // Update sub info
 {
 Set<MessageQueue> subscribeInfo = topicRouteData2TopicSubscribeInfo(topic, topicRouteData);
 Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<String, MQConsumerInner> entry = it.next();
 MQConsumerInner impl = entry.getValue();
 if (impl != null) {
 impl.updateTopicSubscribeInfo(topic, subscribeInfo);
 }
 }
 }
 log.info("topicRouteTable.put. Topic = {}, TopicRouteData[{}]", topic, cloneTopicRouteData);
 this.topicRouteTable.put(topic, cloneTopicRouteData);
 return true;
 }
 } else {
 log.warn("updateTopicRouteInfoFromNameServer, getTopicRouteInfoFromNameServer return null, Topic: {}", topic);
 }
 } catch (Exception e) {
 if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX) && !topic.equals(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC)) {
 log.warn("updateTopicRouteInfoFromNameServer Exception", e);
 }
 } finally {
 this.lockNamesrv.unlock();
 }
 } else {
 log.warn("updateTopicRouteInfoFromNameServer tryLock timeout {}ms", LOCK_TIMEOUT_MILLIS);
 }
 } catch (InterruptedException e) {
 log.warn("updateTopicRouteInfoFromNameServer Exception", e);
 }

 return false;
 }
}

消息队列选择策略实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
 return this.mqFaultStrategy.selectOneMessageQueue(tpInfo, lastBrokerName);
 }
}

负载均衡消息选择策略实现

package org.apache.rocketmq.client.latency;

public class MQFaultStrategy {
 private final static InternalLogger log = ClientLogger.getLog();
 private final LatencyFaultTolerance<String> latencyFaultTolerance = new LatencyFaultToleranceImpl();
 // 发送消息延迟容错开关
 private boolean sendLatencyFaultEnable = false;
 // 消息发送时长的延迟级别数组
 private long[] latencyMax = {50L, 100L, 550L, 1000L, 2000L, 3000L, 15000L};
 // broker预估不可用时长级别数组
 private long[] notAvailableDuration = {0L, 0L, 30000L, 60000L, 120000L, 180000L, 600000L};
 /**
 * 负载均衡消息选择策略
 */
 public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
 if (this.sendLatencyFaultEnable) {
 try {
 // 获取可用队列
 int index = tpInfo.getSendWhichQueue().getAndIncrement();
 for (int i = 0; i < tpInfo.getMessageQueueList().size(); i++) {
 int pos = Math.abs(index++) % tpInfo.getMessageQueueList().size();
 if (pos < 0)
 pos = 0;
 MessageQueue mq = tpInfo.getMessageQueueList().get(pos);
 if (latencyFaultTolerance.isAvailable(mq.getBrokerName())) {
 if (null == lastBrokerName || mq.getBrokerName().equals(lastBrokerName))
 return mq;
 }
 }
 // 选择最优队列
 final String notBestBroker = latencyFaultTolerance.pickOneAtLeast();
 int writeQueueNums = tpInfo.getQueueIdByBroker(notBestBroker);
 if (writeQueueNums > 0) {
 final MessageQueue mq = tpInfo.selectOneMessageQueue();
 if (notBestBroker != null) {
 mq.setBrokerName(notBestBroker);
 mq.setQueueId(tpInfo.getSendWhichQueue().getAndIncrement() % writeQueueNums);
 }
 return mq;
 } else {
 latencyFaultTolerance.remove(notBestBroker);
 }
 } catch (Exception e) {
 log.error("Error occurred when selecting message queue", e);
 }

 return tpInfo.selectOneMessageQueue();
 }

 return tpInfo.selectOneMessageQueue(lastBrokerName);
 }
}

更新broker可用信息（容错）实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) {
 this.mqFaultStrategy.updateFaultItem(brokerName, currentLatency, isolation);
 }
}

与负载均衡消息选择策略位于相同类中

package org.apache.rocketmq.client.latency;

public class MQFaultStrategy {
 	 public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) {
 if (this.sendLatencyFaultEnable) {
 long duration = computeNotAvailableDuration(isolation ? 30000 : currentLatency);
 this.latencyFaultTolerance.updateFaultItem(brokerName, currentLatency, duration);
 }
 } 
}

消息发送实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 private SendResult sendKernelImpl(final Message msg,
 final MessageQueue mq,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final TopicPublishInfo topicPublishInfo,
 final long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 long beginStartTime = System.currentTimeMillis();
 // 获得地址
 String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
 if (null == brokerAddr) {
 tryToFindTopicPublishInfo(mq.getTopic());
 brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
 }

 SendMessageContext context = null;
 if (brokerAddr != null) {
 brokerAddr = MixAll.brokerVIPChannel(this.defaultMQProducer.isSendMessageWithVIPChannel(), brokerAddr);
 // 记录消息内容，因为下面逻辑可能改变消息内容，例如消息压缩
 byte[] prevBody = msg.getBody();
 try {
 //for MessageBatch,ID has been set in the generating process
 if (!(msg instanceof MessageBatch)) {
 MessageClientIDSetter.setUniqID(msg);
 }

 int sysFlag = 0;
 boolean msgBodyCompressed = false;
 if (this.tryToCompressMessage(msg)) {
 sysFlag |= MessageSysFlag.COMPRESSED_FLAG;
 msgBodyCompressed = true;
 }

 final String tranMsg = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
 if (tranMsg != null && Boolean.parseBoolean(tranMsg)) {
 sysFlag |= MessageSysFlag.TRANSACTION_PREPARED_TYPE;
 }
 // 消息发送前校验
 if (hasCheckForbiddenHook()) {
 CheckForbiddenContext checkForbiddenContext = new CheckForbiddenContext();
 checkForbiddenContext.setNameSrvAddr(this.defaultMQProducer.getNamesrvAddr());
 checkForbiddenContext.setGroup(this.defaultMQProducer.getProducerGroup());
 checkForbiddenContext.setCommunicationMode(communicationMode);
 checkForbiddenContext.setBrokerAddr(brokerAddr);
 checkForbiddenContext.setMessage(msg);
 checkForbiddenContext.setMq(mq);
 checkForbiddenContext.setUnitMode(this.isUnitMode());
 this.executeCheckForbiddenHook(checkForbiddenContext);
 }
 // 消息发送前回调处理
 if (this.hasSendMessageHook()) {
 context = new SendMessageContext();
 context.setProducer(this);
 context.setProducerGroup(this.defaultMQProducer.getProducerGroup());
 context.setCommunicationMode(communicationMode);
 context.setBornHost(this.defaultMQProducer.getClientIP());
 context.setBrokerAddr(brokerAddr);
 context.setMessage(msg);
 context.setMq(mq);
 String isTrans = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
 if (isTrans != null && isTrans.equals("true")) {
 context.setMsgType(MessageType.Trans_Msg_Half);
 }

 if (msg.getProperty("__STARTDELIVERTIME") != null || msg.getProperty(MessageConst.PROPERTY_DELAY_TIME_LEVEL) != null) {
 context.setMsgType(MessageType.Delay_Msg);
 }
 this.executeSendMessageHookBefore(context);
 }
 // 构建消息发送请求头
 SendMessageRequestHeader requestHeader = new SendMessageRequestHeader();
 requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup());
 requestHeader.setTopic(msg.getTopic());
 requestHeader.setDefaultTopic(this.defaultMQProducer.getCreateTopicKey());
 requestHeader.setDefaultTopicQueueNums(this.defaultMQProducer.getDefaultTopicQueueNums());
 requestHeader.setQueueId(mq.getQueueId());
 requestHeader.setSysFlag(sysFlag);
 requestHeader.setBornTimestamp(System.currentTimeMillis());
 requestHeader.setFlag(msg.getFlag());
 requestHeader.setProperties(MessageDecoder.messageProperties2String(msg.getProperties()));
 requestHeader.setReconsumeTimes(0);
 requestHeader.setUnitMode(this.isUnitMode());
 requestHeader.setBatch(msg instanceof MessageBatch);
 if (requestHeader.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
 String reconsumeTimes = MessageAccessor.getReconsumeTime(msg);
 if (reconsumeTimes != null) {
 requestHeader.setReconsumeTimes(Integer.valueOf(reconsumeTimes));
 MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_RECONSUME_TIME);
 }

 String maxReconsumeTimes = MessageAccessor.getMaxReconsumeTimes(msg);
 if (maxReconsumeTimes != null) {
 requestHeader.setMaxReconsumeTimes(Integer.valueOf(maxReconsumeTimes));
 MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_MAX_RECONSUME_TIMES);
 }
 }

 SendResult sendResult = null;
 // 消息发送逻辑
 switch (communicationMode) {
 // 异步发送
 case ASYNC:
 Message tmpMessage = msg;
 if (msgBodyCompressed) {
 //If msg body was compressed, msgbody should be reset using prevBody.
 //Clone new message using commpressed message body and recover origin massage.
 //Fix bug:https://github.com/apache/rocketmq-externals/issues/66
 tmpMessage = MessageAccessor.cloneMessage(msg);
 msg.setBody(prevBody);
 }
 long costTimeAsync = System.currentTimeMillis() - beginStartTime;
 if (timeout < costTimeAsync) {
 throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
 }
 sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
 brokerAddr,
 mq.getBrokerName(),
 tmpMessage,
 requestHeader,
 timeout - costTimeAsync,
 communicationMode,
 sendCallback,
 topicPublishInfo,
 this.mQClientFactory,
 this.defaultMQProducer.getRetryTimesWhenSendAsyncFailed(),
 context,
 this);
 break;
 case ONEWAY:
 // 同步发送
 case SYNC:
 long costTimeSync = System.currentTimeMillis() - beginStartTime;
 if (timeout < costTimeSync) {
 throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
 }
 sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
 brokerAddr,
 mq.getBrokerName(),
 msg,
 requestHeader,
 timeout - costTimeSync,
 communicationMode,
 context,
 this);
 break;
 default:
 assert false;
 break;
 }

 if (this.hasSendMessageHook()) {
 // 发送结果放入上下文后返回
 context.setSendResult(sendResult);
 this.executeSendMessageHookAfter(context);
 }

 return sendResult;
 } catch (RemotingException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } catch (MQBrokerException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } catch (InterruptedException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } finally {
 msg.setBody(prevBody);
 }
 }

 throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
 }
}

使用MQClientAPIImpl真正发起Netty网络请求

package org.apache.rocketmq.client.impl;

public class MQClientAPIImpl {
 ...
 public SendResult sendMessage(
 final String addr,
 final String brokerName,
 final Message msg,
 final SendMessageRequestHeader requestHeader,
 final long timeoutMillis,
 final CommunicationMode communicationMode,
 final SendMessageContext context,
 final DefaultMQProducerImpl producer
 ) throws RemotingException, MQBrokerException, InterruptedException {
 return sendMessage(addr, brokerName, msg, requestHeader, timeoutMillis, communicationMode, null, null, null, 0, context, producer);
 }

 /**
 * 封装 RemotingCommand 对象后真正发起网络请求
 */
 public SendResult sendMessage(
 final String addr,
 final String brokerName,
 final Message msg,
 final SendMessageRequestHeader requestHeader,
 final long timeoutMillis,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final TopicPublishInfo topicPublishInfo,
 final MQClientInstance instance,
 final int retryTimesWhenSendFailed,
 final SendMessageContext context,
 final DefaultMQProducerImpl producer
 ) throws RemotingException, MQBrokerException, InterruptedException {
 long beginStartTime = System.currentTimeMillis();
 RemotingCommand request = null;
 if (sendSmartMsg || msg instanceof MessageBatch) {
 SendMessageRequestHeaderV2 requestHeaderV2 = SendMessageRequestHeaderV2.createSendMessageRequestHeaderV2(requestHeader);
 request = RemotingCommand.createRequestCommand(msg instanceof MessageBatch ? RequestCode.SEND_BATCH_MESSAGE : RequestCode.SEND_MESSAGE_V2, requestHeaderV2);
 } else {
 request = RemotingCommand.createRequestCommand(RequestCode.SEND_MESSAGE, requestHeader);
 }
 // 设置请求体
 request.setBody(msg.getBody());

 switch (communicationMode) {
 case ONEWAY:
 this.remotingClient.invokeOneway(addr, request, timeoutMillis);
 return null;
 case ASYNC:
 final AtomicInteger times = new AtomicInteger();
 long costTimeAsync = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTimeAsync) {
 throw new RemotingTooMuchRequestException("sendMessage call timeout");
 }
 this.sendMessageAsync(addr, brokerName, msg, timeoutMillis - costTimeAsync, request, sendCallback, topicPublishInfo, instance,
 retryTimesWhenSendFailed, times, context, producer);
 return null;
 case SYNC:
 long costTimeSync = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTimeSync) {
 throw new RemotingTooMuchRequestException("sendMessage call timeout");
 }
 return this.sendMessageSync(addr, brokerName, msg, timeoutMillis - costTimeSync, request);
 default:
 assert false;
 break;
 }

 return null;
 }

 private SendResult sendMessageSync(
 final String addr,
 final String brokerName,
 final Message msg,
 final long timeoutMillis,
 final RemotingCommand request
 ) throws RemotingException, MQBrokerException, InterruptedException {
 // 同步发网络请求，返回响应结果
 RemotingCommand response = this.remotingClient.invokeSync(addr, request, timeoutMillis);
 assert response != null;
 return this.processSendResponse(brokerName, msg, response);
 }
}

底层Netty通信

类图

`RemotingClient`源码

package org.apache.rocketmq.remoting;

public interface RemotingClient extends RemotingService {

 /**
 * 更新协调服务地址列表
 * @param addrs
 */
 void updateNameServerAddressList(final List<String> addrs);

 /**
 * 更新协调服务地址列表
 * @return
 */
 List<String> getNameServerAddressList();

 /**
 * 同步消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 RemotingCommand invokeSync(final String addr, final RemotingCommand request,
 final long timeoutMillis) throws InterruptedException, RemotingConnectException,
 RemotingSendRequestException, RemotingTimeoutException;

 /**
 * 异步消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeAsync(final String addr, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback) throws InterruptedException, RemotingConnectException,
 RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException;

 /**
 * 单次消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeOneway(final String addr, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingConnectException, RemotingTooMuchRequestException,
 RemotingTimeoutException, RemotingSendRequestException;

 /**
 *用于注册一些底层通信服务，比如进行元数据信息的同步工作，commitlog的同步等
 * @param requestCode 底层业务通信规则码
 * @param processor 注册器（单线程）
 * @param executor 线程池，用于执行注册器的业务逻辑
 */
 void registerProcessor(final int requestCode, final NettyRequestProcessor processor,
 final ExecutorService executor);

 /**
 * 回调函数执行线程设置，用于生产者发送消息后的回调函数
 * @param callbackExecutor
 */
 void setCallbackExecutor(final ExecutorService callbackExecutor);

 /**
 * 获取回调池：NameServer地址
 * @return
 */
 ExecutorService getCallbackExecutor();
 
 /**
 * 用于远程服务地址判断是否可以进行通信
 * @param addr
 * @return
 */
 boolean isChannelWritable(final String addr);
}

registerProcessor方法在实现类中用于初始化时进行注册逻辑

package org.apache.rocketmq.client.impl;

public class MQClientAPIImpl {
 /**
 * Netty 服务启动时初始化工作
 * @param nettyClientConfig 配置信息
 * @param clientRemotingProcessor
 * @param rpcHook
 * @param clientConfig
 */
 public MQClientAPIImpl(final NettyClientConfig nettyClientConfig,
 final ClientRemotingProcessor clientRemotingProcessor,
 RPCHook rpcHook, final ClientConfig clientConfig) {
 this.clientConfig = clientConfig;
 topAddressing = new TopAddressing(MixAll.getWSAddr(), clientConfig.getUnitName());
 this.remotingClient = new NettyRemotingClient(nettyClientConfig, null);
 this.clientRemotingProcessor = clientRemotingProcessor;

 this.remotingClient.registerRPCHook(rpcHook);
 this.remotingClient.registerProcessor(RequestCode.CHECK_TRANSACTION_STATE, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.NOTIFY_CONSUMER_IDS_CHANGED, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.RESET_CONSUMER_CLIENT_OFFSET, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.GET_CONSUMER_STATUS_FROM_CLIENT, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.GET_CONSUMER_RUNNING_INFO, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.CONSUME_MESSAGE_DIRECTLY, this.clientRemotingProcessor, null);
 }
 ...
}

setCallbackExecutor方法用于回调函数执行线程设置，用于生产者发送消息后的回调线程池，例如DefaultMQProducerImpl中的使用方式

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public void setCallbackExecutor(final ExecutorService callbackExecutor) {
	 this.mQClientFactory.getMQClientAPIImpl().getRemotingClient().setCallbackExecutor(callbackExecutor);
 }
 
 /**
 * It will be removed at 4.4.0 cause for exception handling and the wrong Semantics of timeout.
 * A new one will be provided in next version
 * @param msg
 * @param sendCallback
 * @param timeout the <code>sendCallback</code> will be invoked at most time
 * @throws RejectedExecutionException
 */
 @Deprecated
 public void send(final Message msg, final SendCallback sendCallback, final long timeout)
 throws MQClientException, RemotingException, InterruptedException {
 final long beginStartTime = System.currentTimeMillis();
 // 获取回调线程池
 ExecutorService executor = this.getCallbackExecutor();
 try {
 executor.submit(new Runnable() {
 @Override
 public void run() {
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeout > costTime) {
 try {
 sendDefaultImpl(msg, CommunicationMode.ASYNC, sendCallback, timeout - costTime);
 } catch (Exception e) {
 sendCallback.onException(e);
 }
 } else {
 sendCallback.onException(
 new RemotingTooMuchRequestException("DEFAULT ASYNC send call timeout"));
 }
 }

 });
 } catch (RejectedExecutionException e) {
 throw new MQClientException("executor rejected ", e);
 }

 }
	...
}

`RemotingServer`源码

package org.apache.rocketmq.remoting;

public interface RemotingServer extends RemotingService {

 /**
 * 用于注册一些底层通信服务，比如进行元数据信息的同步工作，commitlog的同步等
 * @param requestCode 底层业务通信规则码
 * @param processor 注册器（单线程）
 * @param executor 线程池，用于执行注册器的业务逻辑
 */
 void registerProcessor(final int requestCode, final NettyRequestProcessor processor,
 final ExecutorService executor);

 /**
 * 默认注册方法：比如注册：{@link org.apache.rocketmq.broker.processor.AdminBrokerProcessor}管理Broker工作线程或者namesrv有关的流程器，
 * 比如同步地址、路由topic等信息
 * @param processor
 * @param executor
 */
 void registerDefaultProcessor(final NettyRequestProcessor processor, final ExecutorService executor);

 /**
 * 获取端口
 * @return
 */
 int localListenPort();

 /**
 * 根据业务规则获取Pair<执行器，线程池>
 * @param requestCode 业务规则码
 * @return
 */
 Pair<NettyRequestProcessor, ExecutorService> getProcessorPair(final int requestCode);

 /**
 * 同步消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 RemotingCommand invokeSync(final Channel channel, final RemotingCommand request,
 final long timeoutMillis) throws InterruptedException, RemotingSendRequestException,
 RemotingTimeoutException;

 /**
 * 异步消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeAsync(final Channel channel, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback) throws InterruptedException,
 RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException;

 /**
 * 单向消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeOneway(final Channel channel, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException,
 RemotingSendRequestException;

}

`NettyRemotingServer`源码

其唯一实现类NettyRemotingServer，主要实现Netty编程模型

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingServer extends NettyRemotingAbstract implements RemotingServer {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);
 /**
 * NettyServer辅助类
 */
 private final ServerBootstrap serverBootstrap;
 /**
 * selector组（用于注册并监听event）
 */
 private final EventLoopGroup eventLoopGroupSelector;
 /**
 * 主线程组（用于接收TCP连接）
 */
 private final EventLoopGroup eventLoopGroupBoss;
 /**
 * NettyServer配置
 */
 private final NettyServerConfig nettyServerConfig;

 /**
 * 主要使用在registerProcessor方法中
 */
 private final ExecutorService publicExecutor;

 /**
 * channel事件监听
 */
 private final ChannelEventListener channelEventListener;

 public NettyRemotingServer(final NettyServerConfig nettyServerConfig,
 final ChannelEventListener channelEventListener) {
 super(nettyServerConfig.getServerOnewaySemaphoreValue(), nettyServerConfig.getServerAsyncSemaphoreValue());
 this.serverBootstrap = new ServerBootstrap();
 this.nettyServerConfig = nettyServerConfig;
 this.channelEventListener = channelEventListener;

 int publicThreadNums = nettyServerConfig.getServerCallbackExecutorThreads();
 if (publicThreadNums <= 0) {
 publicThreadNums = 4;
 }

 this.publicExecutor = Executors.newFixedThreadPool(publicThreadNums, new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyServerPublicExecutor_" + this.threadIndex.incrementAndGet());
 }
 });

 this.eventLoopGroupBoss = new NioEventLoopGroup(1, new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyBoss_%d", this.threadIndex.incrementAndGet()));
 }
 });

 // 如果是Linux平台，并且开启了native epoll，就用EpollEventLoopGroup，这个也就是用JNI，调用c写的epoll；否则就用java NIO的NioEventLoopGroup
 if (useEpoll()) {
 this.eventLoopGroupSelector = new EpollEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);
 private int threadTotal = nettyServerConfig.getServerSelectorThreads();

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyServerEPOLLSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
 }
 });
 } else {
 this.eventLoopGroupSelector = new NioEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);
 private int threadTotal = nettyServerConfig.getServerSelectorThreads();

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyServerNIOSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
 }
 });
 }

 loadSslContext();
 }
 
 @Override
 public void start() {
 // 根据配置创建工作线程组
 this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
 nettyServerConfig.getServerWorkerThreads(),
 new ThreadFactory() {

 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyServerCodecThread_" + this.threadIndex.incrementAndGet());
 }
 });
 // Netty编程模型
 ServerBootstrap childHandler =
 this.serverBootstrap.group(this.eventLoopGroupBoss, this.eventLoopGroupSelector)
 .channel(useEpoll() ? EpollServerSocketChannel.class : NioServerSocketChannel.class)
 // sync + accept = backlog
 .option(ChannelOption.SO_BACKLOG, 1024)
 // 允许重复使用本地地址和端口
 .option(ChannelOption.SO_REUSEADDR, true)
 .option(ChannelOption.SO_KEEPALIVE, false)
 .childOption(ChannelOption.TCP_NODELAY, true)
 .childOption(ChannelOption.SO_SNDBUF, nettyServerConfig.getServerSocketSndBufSize())
 .childOption(ChannelOption.SO_RCVBUF, nettyServerConfig.getServerSocketRcvBufSize())
 .localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort()))
 .childHandler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) throws Exception {
 ch.pipeline()
 .addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME,
 new HandshakeHandler(TlsSystemConfig.tlsMode))
 .addLast(defaultEventExecutorGroup,
 new NettyEncoder(),
 new NettyDecoder(),
 new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()),
 new NettyConnectManageHandler(),
 new NettyServerHandler()
 );
 }
 });

 if (nettyServerConfig.isServerPooledByteBufAllocatorEnable()) {
 childHandler.childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
 }

 try {
 ChannelFuture sync = this.serverBootstrap.bind().sync();
 InetSocketAddress addr = (InetSocketAddress) sync.channel().localAddress();
 this.port = addr.getPort();
 } catch (InterruptedException e1) {
 throw new RuntimeException("this.serverBootstrap.bind().sync() InterruptedException", e1);
 }

 if (this.channelEventListener != null) {
 this.nettyEventExecutor.start();
 }

 this.timer.scheduleAtFixedRate(new TimerTask() {

 @Override
 public void run() {
 try {
 NettyRemotingServer.this.scanResponseTable();
 } catch (Throwable e) {
 log.error("scanResponseTable exception", e);
 }
 }
 }, 1000 * 3, 1000);
 }
 ...
}

`NettyRemotingAbstract`源码

NettyRemotingAbstract具体实现三种发送方法

package org.apache.rocketmq.remoting.netty;

public abstract class NettyRemotingAbstract {
 /**
 * 异步调用具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
 long beginStartTime = System.currentTimeMillis();
 final int opaque = request.getOpaque();
 boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
 if (acquired) {
 final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync);
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTime) {
 throw new RemotingTooMuchRequestException("invokeAsyncImpl call timeout");
 }

 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis - costTime, invokeCallback, once);
 this.responseTable.put(opaque, responseFuture);
 try {
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 }
 requestFail(opaque);
 log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel));
 }
 });
 } catch (Exception e) {
 responseFuture.release();
 log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e);
 throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
 }
 } else {
 if (timeoutMillis <= 0) {
 throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast");
 } else {
 String info =
 String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
 timeoutMillis,
 this.semaphoreAsync.getQueueLength(),
 this.semaphoreAsync.availablePermits()
 );
 log.warn(info);
 throw new RemotingTimeoutException(info);
 }
 }
 }
 /**
 * 单向发送具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 public void invokeOnewayImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
 request.markOnewayRPC();
 boolean acquired = this.semaphoreOneway.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
 if (acquired) {
 final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreOneway);
 try {
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 once.release();
 if (!f.isSuccess()) {
 log.warn("send a request command to channel <" + channel.remoteAddress() + "> failed.");
 }
 }
 });
 } catch (Exception e) {
 once.release();
 log.warn("write send a request command to channel <" + channel.remoteAddress() + "> failed.");
 throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
 }
 } else {
 if (timeoutMillis <= 0) {
 throw new RemotingTooMuchRequestException("invokeOnewayImpl invoke too fast");
 } else {
 String info = String.format(
 "invokeOnewayImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
 timeoutMillis,
 this.semaphoreOneway.getQueueLength(),
 this.semaphoreOneway.availablePermits()
 );
 log.warn(info);
 throw new RemotingTimeoutException(info);
 }
 }
 }
 /**
 * 同步调用具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
 final long timeoutMillis)
 throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
 final int opaque = request.getOpaque();

 try {
 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
 this.responseTable.put(opaque, responseFuture);
 final SocketAddress addr = channel.remoteAddress();
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 } else {
 responseFuture.setSendRequestOK(false);
 }

 responseTable.remove(opaque);
 responseFuture.setCause(f.cause());
 responseFuture.putResponse(null);
 log.warn("send a request command to channel <" + addr + "> failed.");
 }
 });

 RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
 if (null == responseCommand) {
 if (responseFuture.isSendRequestOK()) {
 throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
 responseFuture.getCause());
 } else {
 throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
 }
 }

 return responseCommand;
 } finally {
 this.responseTable.remove(opaque);
 }
 }
 ...
}

Netty编码解码源码解析——`RemotingCommand`源码

netty消息数据包结构

RemotingCommand中编码、解码方法体现了其数据包格式和用法

package org.apache.rocketmq.remoting.protocol;

public class RemotingCommand {
 
 public ByteBuffer encode() {
 // 1> header length size
 int length = 4;

 // 2> header data length
 byte[] headerData = this.headerEncode();
 length += headerData.length;

 // 3> body data length
 if (this.body != null) {
 length += body.length;
 }

 ByteBuffer result = ByteBuffer.allocate(4 + length);

 // length
 result.putInt(length);

 // header length
 result.put(markProtocolType(headerData.length, serializeTypeCurrentRPC));

 // header data
 result.put(headerData);

 // body data;
 if (this.body != null) {
 result.put(this.body);
 }

 result.flip();

 return result;
 }
 public static RemotingCommand decode(final ByteBuffer byteBuffer) {
 int length = byteBuffer.limit();
 int oriHeaderLen = byteBuffer.getInt();
 int headerLength = getHeaderLength(oriHeaderLen);

 byte[] headerData = new byte[headerLength];
 // 此处使用的是java nio buffer
 byteBuffer.get(headerData);
 // header封装，长度+类型 对应数据包结构第二位
 RemotingCommand cmd = headerDecode(headerData, getProtocolType(oriHeaderLen));

 int bodyLength = length - 4 - headerLength;
 byte[] bodyData = null;
 if (bodyLength > 0) {
 bodyData = new byte[bodyLength];
 byteBuffer.get(bodyData);
 }
 cmd.body = bodyData;

 return cmd;
 }
 ...
}

其支持的序列化类型有两个

SerializeType序列化类型枚举

package org.apache.rocketmq.remoting.protocol;

public enum SerializeType {
 JSON((byte) 0),
 ROCKETMQ((byte) 1);
 ...
}

在NettyRemotingClient中的Netty编程模型也有编码、解码配置，其使用的是netty buffer

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingClient extends NettyRemotingAbstract implements RemotingClient {
 @Override
 public void start() {
 this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
 nettyClientConfig.getClientWorkerThreads(),
 new ThreadFactory() {

 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyClientWorkerThread_" + this.threadIndex.incrementAndGet());
 }
 });

 Bootstrap handler = this.bootstrap.group(this.eventLoopGroupWorker).channel(NioSocketChannel.class)
 .option(ChannelOption.TCP_NODELAY, true)
 .option(ChannelOption.SO_KEEPALIVE, false)
 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, nettyClientConfig.getConnectTimeoutMillis())
 .option(ChannelOption.SO_SNDBUF, nettyClientConfig.getClientSocketSndBufSize())
 .option(ChannelOption.SO_RCVBUF, nettyClientConfig.getClientSocketRcvBufSize())
 .handler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) throws Exception {
 ChannelPipeline pipeline = ch.pipeline();
 if (nettyClientConfig.isUseTLS()) {
 if (null != sslContext) {
 pipeline.addFirst(defaultEventExecutorGroup, "sslHandler", sslContext.newHandler(ch.alloc()));
 log.info("Prepend SSL handler");
 } else {
 log.warn("Connections are insecure as SSLContext is null!");
 }
 }
 pipeline.addLast(
 // 使用工作线程
 defaultEventExecutorGroup,
 // 编码
 new NettyEncoder(),
 // 解码
 new NettyDecoder(),
 new IdleStateHandler(0, 0, nettyClientConfig.getClientChannelMaxIdleTimeSeconds()),
 // 连接管理器
 new NettyConnectManageHandler(),
 new NettyClientHandler());
 }
 });

 this.timer.scheduleAtFixedRate(new TimerTask() {
 @Override
 public void run() {
 try {
 NettyRemotingClient.this.scanResponseTable();
 } catch (Throwable e) {
 log.error("scanResponseTable exception", e);
 }
 }
 }, 1000 * 3, 1000);

 if (this.channelEventListener != null) {
 this.nettyEventExecutor.start();
 }
 }
}

NettyEncoder源码

package org.apache.rocketmq.remoting.netty;

import io.netty.buffer.ByteBuf;
import java.nio.ByteBuffer;

public class NettyEncoder extends MessageToByteEncoder<RemotingCommand> {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);

 @Override
 public void encode(ChannelHandlerContext ctx, RemotingCommand remotingCommand, ByteBuf out)
 throws Exception {
 try {
 ByteBuffer header = remotingCommand.encodeHeader();
 out.writeBytes(header);
 byte[] body = remotingCommand.getBody();
 if (body != null) {
 out.writeBytes(body);
 }
 } catch (Exception e) {
 log.error("encode exception, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e);
 if (remotingCommand != null) {
 log.error(remotingCommand.toString());
 }
 RemotingUtil.closeChannel(ctx.channel());
 }
 }
}

NettyDecoder源码

其继承LengthFieldBasedFrameDecoder，支持灵活配置编解码参数

package org.apache.rocketmq.remoting.netty;

import io.netty.buffer.ByteBuf;
import io.netty.handler.codec.LengthFieldBasedFrameDecoder;
import java.nio.ByteBuffer;

public class NettyDecoder extends LengthFieldBasedFrameDecoder {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);

 private static final int FRAME_MAX_LENGTH =
 Integer.parseInt(System.getProperty("com.rocketmq.remoting.frameMaxLength", "16777216"));

 public NettyDecoder() {
 super(FRAME_MAX_LENGTH, 0, 4, 0, 4);
 }

 @Override
 public Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception {
 ByteBuf frame = null;
 try {
 frame = (ByteBuf) super.decode(ctx, in);
 if (null == frame) {
 return null;
 }

 ByteBuffer byteBuffer = frame.nioBuffer();

 return RemotingCommand.decode(byteBuffer);
 } catch (Exception e) {
 log.error("decode exception, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e);
 RemotingUtil.closeChannel(ctx.channel());
 } finally {
 if (null != frame) {
 frame.release();
 }
 }

 return null;
 }
}

消息返回状态详解

SEND_OK：消息发送成功。
FLUSH_DISK_TIMEOUT：消息发送成功，但消息刷盘失败，此时服务器宕机则消息丢失
FLUSH_SLAVE_TIMEOUT：主从同步消息时超时，从节点已经收到消息，若此时从节点宕机，则从节点消息丢失
SLAVE_NOT_AVAILABLE：从节点不可用，此时主节点宕机，则消息丢失

对于可靠消息投递，后三种情况均应该考虑补偿。

延迟消息投递

场景：消息发送到Broker后，要特定的时间才会被Consumer消费

特点：目前只支持固定精度的定时消息

固定精度消息原因：支持任意精度消息时，对消息持久化、排序产生较大性能开销

使用方法：对message设置setDelayTimeLevel级别即可

核心类：MessageStoreConfig和ScheduleMessageService

MessageStoreConfig源码

package org.apache.rocketmq.store.config;

public class MessageStoreConfig {
 // 延时投递级别，固定精度
 private String messageDelayLevel = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h";
 ...
}

自定义消息发送规则

场景：把消息发送到指定的队列（Message Queue）

处理类：MessageQueueSelector

用法：调用send(Message msg, MessageQueueSelector selector, Object arg)方法即可

实现代码片段，指定发送到id为1的队列中

SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
 // 外部方法的arg会传递到Object中
 @Override
 public MessageQueue select(List<MessageQueue> list, Message message, Object o) {
 Integer queueNumber = (Integer) o;
 return list.get(queueNumber);
 }
},1);

RocketMQ学习笔记2——入门

Wed, 23 Dec 2020 19:06:00 +0800

生产者模型使用

步骤

创建生产者对象DefaultMQProducer
设置NamesrvAddr
启动生产者服务
创建消息并发送

简单消息发送

创建测试主题

常量定义

public class Const {

	public static final String NAMESRV_ADDR_SINGLE = "10.139.12.52:9876";
}

public class Producer {

	public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
		
		DefaultMQProducer producer = new DefaultMQProducer("test_quick_producer_name");
		
		producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
		producer.start();
		
		for(int i = 0 ; i <5; i ++) {
			//	1.	创建消息
			Message message = new Message("test_quick_topic",	//	主题
					"TagA", //	标签
					"key" + i,	// 	用户自定义的key ,唯一的标识
					("Hello RocketMQ" + i).getBytes());	//	消息内容实体（byte[]）
			SendResult sr = producer.send(message, new MessageQueueSelector() {
				@Override
				public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
					Integer queueNumber = (Integer)arg;
					return mqs.get(queueNumber);
				}
			}, 2);
			System.err.println(sr);
			System.err.println("消息发出: " + sr);
		}
		producer.shutdown();
	}
}

运行结果

SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4]
10:45:15.129 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10911] result: true
10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10909] result: true
10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:9876] result: true

消费者模型使用

步骤

创建消费者对象DefaultMQPushConsumer
设置NamesrvAddr及消费位置ConsumeFromWhere
进行主题订阅Subscribe
注册监听并消费registerMessageListener

简单消息消费

public class Consumer {

	public static void main(String[] args) throws MQClientException {
		// 采用Push方式
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test_quick_consumer_name");
		
		consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
		// 匹配所有Tag
		consumer.subscribe("test_quick_topic", "*");
		
		consumer.registerMessageListener(new MessageListenerConcurrently() {
			
			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
				MessageExt me = msgs.get(0);
				try {
					String topic = me.getTopic();
					String tags = me.getTags();
					String keys = me.getKeys();
					
					String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET);
					System.err.println("topic: " + topic + ",tags: " + tags + ", keys: " + keys + ",body: " + msgBody);
				} catch (Exception e) {
					e.printStackTrace();
 // 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
					return ConsumeConcurrentlyStatus.RECONSUME_LATER;
				}
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		consumer.start();
		System.err.println("consumer start...");
	}
}

模拟异常重试

public class Consumer {

	public static void main(String[] args) throws MQClientException {
		// 采用Push方式
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test_quick_consumer_name");
		
		consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
		consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
		// 匹配所有Tag
		consumer.subscribe("test_quick_topic", "*");
		
		consumer.registerMessageListener(new MessageListenerConcurrently() {
			
			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
				MessageExt me = msgs.get(0);
				try {
					String topic = me.getTopic();
					String tags = me.getTags();
					String keys = me.getKeys();
					// 模拟异常
					if(keys.equals("key1")) {
						System.err.println("消息消费失败..");
						int a = 1/0;
					}
					
					String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET);
					System.err.println("topic: " + topic + ",tags: " + tags + ", keys: " + keys + ",body: " + msgBody);
				} catch (Exception e) {
					e.printStackTrace();
					// 重发次数
					int recousumeTimes = me.getReconsumeTimes();
					System.err.println("recousumeTimes: " + recousumeTimes);
					// 超过3次不再重发
					if(recousumeTimes == 3) {
						//		记录日志....
						// 	做补偿处理
						return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
					}
					// 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
					return ConsumeConcurrentlyStatus.RECONSUME_LATER;
				}
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		consumer.start();
		System.err.println("consumer start...");
	}
}

日志输出

consumer start...
topic: test_quick_topic,tags: TagA, keys: key0,body: Hello RocketMQ0
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 0
topic: test_quick_topic,tags: TagA, keys: key2,body: Hello RocketMQ2
topic: test_quick_topic,tags: TagA, keys: key3,body: Hello RocketMQ3
topic: test_quick_topic,tags: TagA, keys: key4,body: Hello RocketMQ4
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 1
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 2
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 3

四种集群环境

集群环境拓扑图

分类

单点模式（不推荐）
主从模式
双主模式（不推荐）
双主双从、多主多从模式

主从集群环境构建

优势：

主从模式可以保障消息的即时性与可靠性
当投递一条消息后关闭主节点，从节点可以继续提供消费者进行消费，但不能接受消息
主节点上线后会进行消费进度offset同步

基础环境

选择2台服务器进行部署验证

IP	IP
GY11-SyOCM33-101	10.139.12.52	主
GY11-SyOCM33-201	10.139.12.53	从

53节点按照单点部署时操作好对应目录和日志配置

修改主节点配置

52节点修改主节点配置文件broker-a.properties如下

修改了namesrvAddr

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876;10.139.12.53:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

修改从节点配置

修改从节点配置文件broker-a-s.properties如下

拷贝主节点配置后修改id和brokerRole

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876;10.139.12.53:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

主从配置同步到53节点即可

启动NameSrv

启动Namesrv（主从节点均执行）

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin
$ nohup sh mqnamesrv &

启动Broker

主节点启动

$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &

从节点启动

$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a-s.properties >/dev/null 2>&1 &

启动控制台

修改控制台配置地址参数为

rocketmq.config.namesrvAddr=10.139.12.52:9876;10.139.12.53:9876

数据高可用机制故障演练

投递一条消息

关闭主节点，剩余一个slave

启动消费者，消费正常

主节点上线，同步消息位点（offset）使消费端不会重复消费

RocketMQ学习笔记1——初探

Wed, 23 Dec 2020 19:04:00 +0800

RocketMQ整体认知

优势

支持集群模型、负载均衡、水平扩展能力
亿级别的消息堆积能力
采用零拷贝的原理、顺序写盘、随机读
丰富的API使用
代码优秀，底层通信框架采用Netty NIO框架
NameServer代替Zookeeper
强调集群无单点，可扩展，任意一点高可用，水平可扩展
消息失败重试机制，消息可查询
开源社区活跃，成熟度高（经过双十一考验）

核心概念模型

基本概念

消息生产者（Producer）：负责生产消息，一般由业务系统负责生产消息。
消息消费者（Consumer）：负责消费消息，一般是后台系统负责异步消费。
推动式消费（Push Consumer）： Consumer消费的一种类型，该模式下Broker收到数据后会主动推送给消费端。
拉取式消费（Pull Consumer）： Consumer消费的一种类型，应用通常主动调用Consumer的拉消息方法从Broker服务器拉消息。
生产者组（Producer Group）：同一类Producer的集合，这类Producer发送同一类消息且发送逻辑一致。
消费者组（Consumer Group）：同一类Consumer的集合，这类Consumer通常消费同一类消息且消费逻辑一致。
Broker：MQ消息服务（中转角色，用于消息存储于生产消息转发）

源码结构

源码分析以4.4.0为例

源码地址： https://github.com/apache/rocketmq

结构：

结构解析：

rocketmq-common：公用数据结构
rocketmq-distribution：编译模块，编译输出
rocketmq-filter：进行Broker过滤的不感兴趣的消息传输，减小带宽压力
rocketmq-logappender、rocketmq-logging：日志服务
rocketmq-namesrv：Namesrv服务，用于服务协调
rocketmq-openmessaging：对外提供服务
rocketmq-remoting：远程调用接口，封装Netty底层通信
rocketmq-srvutil：提供一些公用的工具方法，比如命令行解析参数
rocketmq-store：消息存储
rocketmq-test、rocketmq-example
rocketmq-tools：管理工具，比如有名的mqadmin工具

编译部署&环境搭建

选择1台服务器进行部署验证

IP	IP
GY11-SyOCM33-101	10.139.12.52

编译部署

在源码目录执行一下命令编译工程

$ mvn -Prelease-all -DskipTests clean package

编译成功后在rocketmq\distribution\target目录下生成apache-rocketmq.tar.gz包，使用该包解压与从官网下载一致，其他步骤参考二进制部署。

二进制包部署（单点）

此处从官网下载4.4.0版本并上传至服务器解压

下载地址

解压到~/rocketmq-test/rocketmq-all-4.4.0-bin-release/如下

$ ll
total 36
drwxr-xr-x 2 ocmuser gpocm 83 Jan 17 2019 benchmark
drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 bin
drwxr-xr-x 5 ocmuser gpocm 196 Jan 17 2019 conf
drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 lib
-rw-r--r-- 1 ocmuser gpocm 17336 Jan 17 2019 LICENSE
-rw-r--r-- 1 ocmuser gpocm 1337 Jan 17 2019 NOTICE
-rw-r--r-- 1 ocmuser gpocm 2481 Jan 17 2019 README.md

创建存储目录

$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index

RocketMQ 配置文件

进入提供好的样例配置文件目录，其中包含双主双从异步复制、双主双从同步复制和双主集群配置文件，此处选择双主双从异步复制作为配置模板

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf]$ll
total 36
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync
drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave
-rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf
-rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml
-rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml
-rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml
-rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml
-rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml

选择broker-a配置文件

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async]$ll
total 16
-rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-a.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties
-rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-b.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties

删除原有配置，加入如下配置（单点）

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

修改日志配置文件

$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs 
$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf && sed -i 's#${user.home}#/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release#g' *.xml

修改启动脚本参数

将默认8g的内存配置改成1g

$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runbroker.sh

runbroker.sh

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"
...

同理

$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runserver.sh

runserver.sh

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"
...

启动 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ nohup sh mqnamesrv &

检查进程

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin]$jps
65322 NamesrvStartup
65534 Jps

启动 BrokerServer A

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &

$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/namesrv.log

停止与数据清理

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ sh mqshutdown broker 
$ sh mqshutdown namesrv 
# --等待停止 
$ rm -rf /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index 
# --按照上面步骤重启 NameServer 与 BrokerServer

SpringBoot 控制台讲解

克隆工程：https://github.com/apache/rocketmq-externals.git

进入控制台配置文件rocketmq-externals\rocketmq-console\src\main\resources\application.properties修改配置

server.contextPath=
server.port=9090

## SSL setting
#server.ssl.key-store=classpath:rmqcngkeystore.jks
#server.ssl.key-store-password=rocketmq
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=rmqcngkey

#spring.application.index=true
spring.application.name=rocketmq-console
spring.http.encoding.charset=UTF-8
spring.http.encoding.enabled=true
spring.http.encoding.force=true
logging.config=classpath:logback.xml
#if this value is empty,use env value rocketmq.config.namesrvAddr NAMESRV_ADDR | now, you can set it in ops page.default localhost:9876
rocketmq.config.namesrvAddr=10.139.12.52:9876
#if you use rocketmq version < 3.5.8, rocketmq.config.isVIPChannel should be false.default true
rocketmq.config.isVIPChannel=
#rocketmq-console's data path:dashboard/monitor
rocketmq.config.dataPath=./data
#set it false if you don't want use dashboard.default true
rocketmq.config.enableDashBoardCollect=true
#set the message track trace topic if you don't want use the default one

SpringBoot源码分析2——走向自动装配

Thu, 10 Dec 2020 17:28:00 +0800

Spring Framework 手动装配

Spring模式注解装配

定义:一种用于声明在应用中扮演“组件”角色的注解
举例:@Component、@Service、@Configuration等
装配:< context: component-scan>或@Componentscan

官方仓库说明

Stereotype Annotations
A *stereotype annotation* is an annotation that is used to declare the role that a component plays within the application. For example, the @Repository annotation in the Spring Framework is a marker for any class that fulfills the role or stereotype of a repository (also known as Data Access Object or DAO).

@Component is a generic stereotype for any Spring-managed component. Any component annotated with @Component is a candidate for component scanning. Similarly, any component annotated with an annotation that is itself meta-annotated with @Component is also a candidate for component scanning. For example, @Service is meta-annotated with @Component.

Core Spring provides several stereotype annotations out of the box, including but not limited to: @Component, @Service, @Repository, @Controller, @RestController, and @Configuration. @Repository, @Service, etc. are specializations of @Component.

模式注解是一种用于声明在应用中扮演“组件”角色的注解。如 Spring Framework 中的 @Repository 标注在任何类上，用于扮演仓储角色的模式注解。

@Component 作为一种由 Spring 容器托管的通用模式组件，任何被 @Component 标准的组件均为组件扫描的候选对象。类似地，凡是被 @Component 元标注（meta-annotated）的注解，如 @Service ，当任何组件标注它时，也被视作组件扫描的候选对象（派生性）

模式注解举例

Spring Framework 注解	场景说明	起始版本
`@Repository`	数据仓储模式注解	2.0
`@Component`	通用组件模式注解	2.5
`@Service`	服务模式注解	2.5
`@Controller`	Web控制器模式注解	2.5
`@Configuration`	配置类模式注解	3.0

装配方式

<context:component-scan> 方式

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:context="http://www.springframework.org/schema/context"
 xsi:schemaLocation="http://www.springframework.org/schema/beans 
http://www.springframework.org/schema/beans/spring-beans.xsd
 http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-
context.xsd">
 
 <!-- 激活注解驱动特性 -->
 <context:annotation-config />
 
 <!-- 找寻被 @Component 或者其派生 Annotation 标记的类（Class），将它们注册为 Spring Bean -->
 <context:component-scan base-package="top.fjy8018.spring.boot" />
 
</beans>

@ComponentScan 方式

@ComponentScan(basePackages = "top.fjy8018.spring.boot")
public class SpringConfiguration {
 ...
}

Spring Framework手动装配

自定义模式注解

`@Component` “派生性”

package top.fjy8018.springboot.annotation;

import org.springframework.stereotype.Component;
import org.springframework.stereotype.Repository;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 10:36
 */
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Repository
public @interface FirstLevelRepository {

 /**
 * 保证签名一致性
 */
 String value() default "";
}

三个级别并无明显区别，体现注解派生性，即使用@FirstLevelRepository和使用@Component效果一致

派生关系

@Component
- @Repository
  - @FirstLevelRepository

package org.springframework.stereotype;

/**
 * @since 2.5
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Indexed
public @interface Component {

 /**
 * Component的签名
 */
 String value() default "";

}

Repository中元标注了@Component

package org.springframework.stereotype;

/**
 * @since 2.0
 * @see Component
 */
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Component
public @interface Repository {

 /**
 * 签名与Component一致
 */
 @AliasFor(annotation = Component.class)
 String value() default "";
}

`@Component` “层次性”

package top.fjy8018.springboot.annotation;

import org.springframework.stereotype.Component;
import org.springframework.stereotype.Repository;

import java.lang.annotation.*;

/**
 * Component层次性
 * 顶级 {@link Component}
 * 一级 {@link Repository @Repository}
 * 二级 {@link FirstLevelRepository}
 * 三级 {@link SecondLevelRepository}
 *
 * @author F嘉阳
 * @date 2018-09-09 10:54
 */
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@FirstLevelRepository
public @interface SecondLevelRepository {

 /**
 * 层次性、派生性都必须保证签名一致
 */
 String value() default "";
}

@Component
- @Repository
  - FirstLevelRepository
    - SecondLevelRepository

注解目标类，value 为bean的名称，此处不管用@FirstLevelRepository还是用SecondLevelRepository，效果都和使用@Component一致

package top.fjy8018.springboot.repository;

import top.fjy8018.springboot.annotation.FirstLevelRepository;

/**
 * @author F嘉阳
 * @date 2018-09-09 10:40
 */
//@Component("myRepository")
@FirstLevelRepository(value = "myRepository")
//@SecondLevelRepository(value = "myRepository")
public class RepositoryDemo {
}

仓储引导类，使用@ComponentScan指定扫描Bean路径

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.ComponentScan;
import top.fjy8018.springboot.repository.RepositoryDemo;

/**
 * @author F嘉阳
 * @date 2018-09-09 10:43
 */
@ComponentScan(basePackages = "top.fjy8018.springboot.repository")
public class RepositoryBootstrapDemo {

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(RepositoryBootstrapDemo.class)
 .web(WebApplicationType.NONE)
 .run(args);

 // 通过bean名称获取bean
 RepositoryDemo myRepository = applicationContext.getBean("myRepository", RepositoryDemo.class);

 // myRepository是否存在，若不存在则报错
 System.out.println("myRepository是否存在：" + myRepository);

 // 关闭上下文
 applicationContext.close();
 }
}

执行结果

SpringBoot源码中也有体现该层次关系

@SpringBootApplication
- @SpringBootConfiguration
  - @Configuration
    - @Component
- @EnableAutoConfiguration
  - @AutoConfigurationPackage

Spring @Enable 模块装配

Spring Framework 3.1 开始支持“@Enable 模块驱动”。所谓“模块”是指具备相同领域的功能组件集合，组合所形成一个独立的单元。比如 Web MVC 模块、AspectJ代理模块、Caching（缓存）模块、JMX（Java 管理扩展）模块、Async（异步处理）模块等。

举例:@EnableWebmvc、@EnableAutoConfiguration等
实现:注解方式、编程方式

@Enable 注解模块举例

框架实现	`@Enable` 注解模块	激活模块
Spring Framework	`@EnableWebMvc`	Web
	`@EnableTransactionManagement`	事务管理模块
	`@EnableCaching`	Caching 模块
	`@EnableMBeanExport`	JMX 模块
	`@EnableAsync`	异步处理模块
	`EnableWebFlux`	Web Flux 模块
	`@EnableAspectJAutoProxy`	AspectJ 代理模块

Spring Boot	`@EnableAutoConfiguration`	自动装配模块
	`@EnableManagementContext`	Actuator 管理模块
	`@EnableConfigurationProperties`	配置属性绑定模块
	`@EnableOAuth2Sso`	OAuth2 单点登录模块

Spring Cloud	`@EnableEurekaServer`	Eureka服务器模块
	`@EnableConfigServer`	配置服务器模块
	`@EnableFeignClients`	Feign客户端模块
	`@EnableZuulProxy`	服务网关
	`@EnableCircuitBreaker`	服务熔断模块

实现方式

注解驱动方式

Spring源码，使用@Import注入配置Bean

package org.springframework.web.servlet.config.annotation;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Import(DelegatingWebMvcConfiguration.class)
public @interface EnableWebMvc {}

接口编程方式

Spring源码——激活缓存注解

package org.springframework.cache.annotation;

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Import(CachingConfigurationSelector.class)
public @interface EnableCaching {...}

CachingConfigurationSelector类图

ImportSelector接口源码，该接口在3.1引入，目的在于增加灵活度

selectImports方法返回数组，即可以死是多个配置Bean

package org.springframework.context.annotation;

/**
 * @since 3.1
 */
public interface ImportSelector {
 
 String[] selectImports(AnnotationMetadata importingClassMetadata);
}

@Enable注解方式自定义实现

激活HelloWorld模块，使用@Import(HelloWorldConfiguration.class)直接注入HelloWorldConfiguration模块

package top.fjy8018.springboot.annotation;

import org.springframework.context.annotation.Import;
import top.fjy8018.springboot.configuration.HelloWorldConfiguration;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:35
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Import(HelloWorldConfiguration.class)
public @interface EnableHelloWorld {}

目标类，其注解@Configuration在Spring3.0时引入

package top.fjy8018.springboot.configuration;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:31
 */
@Configuration
public class HelloWorldConfiguration {

 /**
 * 方法名称即Bean名称
 * @return Hello World!
 */
 @Bean
 public String helloWorld() {
 return "Hello world!";
 }
}

package org.springframework.context.annotation;

/**
 * @since 3.0
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Component
public @interface Configuration {
 @AliasFor(annotation = Component.class)
 String value() default "";
}

@Enable接口编程方式自定义实现

参考Spring包名路径创建自定义Selector，通过HelloWorldImportSelector实现HelloWorldConfiguration的配置Bean导入

package top.fjy8018.springboot.annotation;

import org.springframework.context.annotation.ImportSelector;
import org.springframework.core.type.AnnotationMetadata;
import top.fjy8018.springboot.configuration.HelloWorldConfiguration;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:28
 */
public class HelloWorldImportSelector implements ImportSelector {

 /**
 * @param importingClassMetadata 元数据信息
 * @return 一个或多个配置Bean
 */
 @Override
 public String[] selectImports(AnnotationMetadata importingClassMetadata) {
 return new String[]{HelloWorldConfiguration.class.getName()};
 }
}

激活模块方式变化，使用@Import(HelloWorldImportSelector.class)注入HelloWorldImportSelector模块，在Selector里加载并返回 HelloWorldConfiguration 模块

package top.fjy8018.springboot.annotation;

import org.springframework.context.annotation.Import;
import top.fjy8018.springboot.configuration.HelloWorldConfiguration;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:35
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
//@Import(HelloWorldConfiguration.class)
@Import(HelloWorldImportSelector.class)
public @interface EnableHelloWorld {
}

实现逻辑

graph LR
A(EnableHelloWorld) --> B(HelloWorldImportSelector) 
 B-->C(HelloWorldConfiguration)
 C --> D(helloWorld)

此方法比直接注解驱动更有弹性，可增加switch条件返回不同的模块，源码样例CachingConfigurationSelector

package org.springframework.cache.annotation;
public class CachingConfigurationSelector extends AdviceModeImportSelector<EnableCaching> {
 
 @Override
 public String[] selectImports(AdviceMode adviceMode) {
 switch (adviceMode) {
 case PROXY:
 return getProxyImports();
 case ASPECTJ:
 return getAspectJImports();
 default:
 return null;
 }
 }
 ...
}

Spring条件装配

从 Spring Framework 3.1 开始，允许在 Bean 装配时增加前置条件判断

Spring 注解	场景说明	起始版本
`@Profile`	配置化条件装配	3.1
`@Conditional`	编程条件装配	4.0

条件注解实现方式

配置方式 - `@Profile`

在spring4后实现方式发生变化，依赖@Conditional和ProfileCondition实现

package org.springframework.context.annotation;

/**
* @since 3.1
*/
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(ProfileCondition.class)
public @interface Profile {...}

package org.springframework.context.annotation;

/**
 * @since 4.0
 */
class ProfileCondition implements Condition {...}

通过上下文对象和注解元信息判断

package org.springframework.context.annotation;

/**
 * @since 4.0
 */
@FunctionalInterface
public interface Condition {
 boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata);
}

编程方式 - `@Conditional`

ConditionalOnClass用于判断标识类在当前classpath下是否存在，该注解使用@Conditional搭配OnClassCondition实现

package org.springframework.boot.autoconfigure.condition;

/**
 * {@link Conditional} that only matches when the specified classes are on the classpath.
 */
@Target({ ElementType.TYPE, ElementType.METHOD })
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(OnClassCondition.class)
public @interface ConditionalOnClass {...}

package org.springframework.boot.autoconfigure.condition;

/**
 * {@link Condition} that checks if properties are defined in environment.
 * @since 1.1.0
 * @see ConditionalOnProperty
 */
@Order(Ordered.HIGHEST_PRECEDENCE + 40)
class OnPropertyCondition extends SpringBootCondition {
 @Override
 public ConditionOutcome getMatchOutcome(ConditionContext context,
 AnnotatedTypeMetadata metadata) {
 // 获取元信息
 List<AnnotationAttributes> allAnnotationAttributes = annotationAttributesFromMultiValueMap(
 metadata.getAllAnnotationAttributes(
 ConditionalOnProperty.class.getName()));
 List<ConditionMessage> noMatch = new ArrayList<>();
 List<ConditionMessage> match = new ArrayList<>();
 for (AnnotationAttributes annotationAttributes : allAnnotationAttributes) {
 ConditionOutcome outcome = determineOutcome(annotationAttributes,
 context.getEnvironment());
 (outcome.isMatch() ? match : noMatch).add(outcome.getConditionMessage());
 }
 if (!noMatch.isEmpty()) {
 return ConditionOutcome.noMatch(ConditionMessage.of(noMatch));
 }
 return ConditionOutcome.match(ConditionMessage.of(match));
 }
 ...
}

OnPropertyCondition为抽象类，实际仍然是Condition接口的实现

package org.springframework.boot.autoconfigure.condition;

public abstract class SpringBootCondition implements Condition {...}

ConditionalOnProperty用于判断配置在上下文中是否存在

package org.springframework.boot.autoconfigure.condition;

/**
 * @since 1.1.0
 */
@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.TYPE, ElementType.METHOD })
@Documented
@Conditional(OnPropertyCondition.class)
public @interface ConditionalOnProperty {...}

自定义条件装配

目标服务

package top.fjy8018.springboot.service;

/**
 * 计算模块
 * @author F嘉阳
 * @date 2018-09-09 11:56
 */
public interface CalculateService {

 /**
 * 多整数求和
 * @param values 多个整数
 * @return 累加值
 */
 Integer sum(Integer... values);
}

基于配置方式

服务实现类，分别使用 for循环实现和Lambda实现，均标注@Service作为Bean自动注入

指定@Profile("Java7")，当条件为Java7时才装配

package top.fjy8018.springboot.service.impl;

import org.springframework.context.annotation.Profile;
import org.springframework.stereotype.Service;
import top.fjy8018.springboot.service.CalculateService;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:57
 */
@Profile("Java7")
@Service
public class Java7CalculateServiceImpl implements CalculateService {

 /**
 * 多整数求和
 *
 * @param values 多个整数
 * @return 累加值
 */
 @Override
 public Integer sum(Integer... values) {
 System.out.println("计算模块java7 for循环实现");
 int sum = 0;
 for (int i = 0; i < values.length; i++) {
 sum += values[i];
 }
 return sum;
 }
}

指定@Profile("Java8")，当条件为Java7时才装配

package top.fjy8018.springboot.service.impl;

import org.springframework.context.annotation.Profile;
import org.springframework.stereotype.Service;
import top.fjy8018.springboot.service.CalculateService;

import java.util.stream.Stream;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:57
 */
@Profile("Java8")
@Service
public class Java8CalculateServiceImpl implements CalculateService {

 /**
 * 多整数求和
 * @param values 多个整数
 * @return 累加值
 */
 @Override
 public Integer sum(Integer... values) {
 System.out.println("计算模块java8 Lambda实现");
 return Stream.of(values).reduce(0, Integer::sum);
 }
}

启动类，通过scanBasePackages指定报名扫描，该方法与@ComponentScan(basePackages = "xxx")效果一致

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import top.fjy8018.springboot.service.CalculateService;

/**
 * @author F嘉阳
 * @date 2018-09-09 12:06
 */
@SpringBootApplication(scanBasePackages = "top.fjy8018.springboot.service")
public class CalculateServiceDemo {

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(CalculateServiceDemo.class)
 .profiles("Java8")
 .web(WebApplicationType.NONE)
 .run(args);

 // 直接用类获取bean
 CalculateService calculateService = applicationContext.getBean(CalculateService.class);

 // calculateService Bean 是否存在，若不添加Profile则必有报错
 Integer[] values = new Integer[10];
 for (int i = 1; i <= values.length; i++) {
 values[i - 1] = i;
 }
 System.out.println("calculateService Bean 求和: " + calculateService.sum(values));

 // 关闭上下文
 applicationContext.close();
 }
}

执行结果——java8

执行结果——java7

若不指定配置则报错

基于编程方式

系统属性条件判断实现类

package top.fjy8018.springboot.condition;

import org.springframework.context.annotation.Condition;
import org.springframework.context.annotation.ConditionContext;
import org.springframework.core.type.AnnotatedTypeMetadata;

import java.util.Map;

/**
 * @author F嘉阳
 * @date 2018-09-09 17:15
 */
public class OnSystemPropertyCondition implements Condition {

 /**
 * 条件匹配规则，此处可实现复杂逻辑判断
 * @param context 上下文
 * @param metadata 元信息
 * @return 是否匹配
 */
 @Override
 public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {

 // 通过类名称取元信息
 Map<String, Object> annotationAttributes = metadata.getAnnotationAttributes(ConditionalOnSystemProperty.class.getName());

 String propertyName = String.valueOf(annotationAttributes.get("name"));
 String propertyValue = String.valueOf(annotationAttributes.get("value"));

 // 取系统属性
 String systemPropertyValue = System.getProperty(propertyName);

 return propertyValue.equals(systemPropertyValue);
 }

Java 系统属性条件判断类装配，系统属性通过Java API获取java.lang.System#getProperty(java.lang.String)方法获取的属性值

package top.fjy8018.springboot.condition;

import org.springframework.context.annotation.Conditional;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 17:14
 */
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(OnSystemPropertyCondition.class)
public @interface ConditionalOnSystemProperty {

 /**
 * Java 系统属性名称
 * @return 系统属性名称
 */
 String name();

 /**
 * 系统属性值
 * @return 属性值
 */
 String value();
}

启动类，helloWorld仅在系统条件满足时自动装配，注解若作用于类上语义相同

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.Bean;
import top.fjy8018.springboot.condition.ConditionalOnSystemProperty;

/**
 * 系统属性条件引导类
 *
 * @author F嘉阳
 * @date 2018-09-09 17:26
 */
public class SystemPropertyConditionBootstrap {

 @Bean
 @ConditionalOnSystemProperty(name = "user.name", value = "11860")
 public String helloWorld() {
 return "Hello SpringBoot2.0";
 }

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(SystemPropertyConditionBootstrap.class)
 .web(WebApplicationType.NONE)
 .run(args);

 // 通过类型和方法名获取Bean
 String helloWorldBean = applicationContext.getBean("helloWorld", String.class);
 // 若系统参数条件不匹配则报找不到Bean错误，以此实现编程方式条件加载
 System.out.println("ConditionalOnSystemProperty 加载内容：" + helloWorldBean);

 applicationContext.close();
 }
}

执行结果

条件不满足则报错

Spring Boot 自动装配

在 Spring Boot 场景下，基于规约大于配置的原则，实现 Spring 组件自动装配的目的（0配置）。

装配: 模式注解、@Eηable模块、条件装配、工厂加载机制
实现: 激活自动装配、实现自动装配、配置自动装配实现

底层装配技术

Spring 模式注解装配
Spring @Enable 模块装配
Spring 条件装配装配
Spring 工厂加载机制
- 实现类： SpringFactoriesLoader
- 配置资源： META-INF/spring.factories

package org.springframework.core.io.support;

/**
 * @since 3.2
 */
public abstract class SpringFactoriesLoader {

 public static final String FACTORIES_RESOURCE_LOCATION = "META-INF/spring.factories";
 
 public static <T> List<T> loadFactories(Class<T> factoryClass, @Nullable ClassLoader classLoader) {
 Assert.notNull(factoryClass, "'factoryClass' must not be null");
 // 获取类加载器
 ClassLoader classLoaderToUse = classLoader;
 if (classLoaderToUse == null) {
 classLoaderToUse = SpringFactoriesLoader.class.getClassLoader();
 }
 List<String> factoryNames = loadFactoryNames(factoryClass, classLoaderToUse);
 if (logger.isTraceEnabled()) {
 logger.trace("Loaded [" + factoryClass.getName() + "] names: " + factoryNames);
 }
 List<T> result = new ArrayList<>(factoryNames.size());
 // 迭代装配
 for (String factoryName : factoryNames) {
 result.add(instantiateFactory(factoryName, factoryClass, classLoaderToUse));
 }
 AnnotationAwareOrderComparator.sort(result);
 return result;
 }
 ...
}

自动装配举例

参考 META-INF/spring.factories

实现方法

激活自动装配 - @EnableAutoConfiguration
实现自动装配 - XXXAutoConfiguration
配置自动装配实现 - META-INF/spring.factories

目的在于使用META-INF/spring.factories文件配置替代@ComponentScan手动装配注解，从而实现自动装配

而Spring大多数自动装配都由starter模块负责

自定义自动装配

设计逻辑

HelloWorldAutoConfiguration

条件判断： user.name == "Mercy"
模式注解： @Configuration
@Enable 模块：流程图

引导类

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;

/**
 * @author F嘉阳
 * @date 2018-09-10 17:13
 */
@EnableAutoConfiguration
public class HelloWorldAutoConfigurationBootstrap {

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(HelloWorldAutoConfigurationBootstrap.class)
 .web(WebApplicationType.NONE)
 .run(args);

 // 通过bean名称获取helloWorld Bean
 String helloWorldBean = applicationContext.getBean("helloWorld", String.class);

 // helloWorld Bean 是否存在
 System.out.println("helloWorld Bean 内容: " + helloWorldBean);

 // 关闭上下文
 applicationContext.close();
 }
}

配置Bean，使用模式注解标注 @Configuration，该类实际上实现了条件装配、模式注解、自动装配

package top.fjy8018.springboot.configuration;

import org.springframework.context.annotation.Configuration;
import top.fjy8018.springboot.annotation.EnableHelloWorld;
import top.fjy8018.springboot.condition.ConditionalOnSystemProperty;

/**
 * @author F嘉阳
 * @date 2018-09-10 17:23
 */
@Configuration
@EnableHelloWorld
@ConditionalOnSystemProperty(name = "user.name", value = "11860")
public class HelloWorldAutoConfiguration {}

创建spring.factories，每行全类名顶格开始，防止在某些版本下产生不稳定的BUG

# 自动装配
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
top.fjy8018.springboot.configuration.HelloWorldAutoConfiguration

执行结果与之前一致

SpringBoot源码分析1——总览

Thu, 10 Dec 2020 17:11:00 +0800

SpringBoot 2.0 特点

组件自动装配

规约大于配置,专注核心业务

模式注解、@ Enable模块、条件装配、加载机制

外部化配置

一次构建、按需调配,到处运行

Environment抽象、生命周期、破坏性变更

嵌入式容器

内置容器、无需部署、独立运行

Servlet Web容器、 Reactive Web容器

Spring Boot Starter

简化依赖、按需装配、自我包含

依赖管理、装配条件、装配顺序

Production-Ready

一站式运维、生态无缝整合

健康检査、数据指标、@ Endpoint管控

SpringBoot 与Java EE规范

Web: Servlet( JSR-315, JSR-340）
SQL: JDBC(SR-221)
数据校验: Bean validation(JSR303、JSR-349)
缓存: Java Caching API(JSR-107)
Web Sockets: Java api for Web Socket(JSR-356
Web services: JAX-WS( JSR-224）
Java管理:JMX(JSR3)
消息:JMS(JSR-914)

核心特性介绍

三大特性

组件自动装配: Web mvc、 Web flux、JDBC等
嵌入式Web容器: Tomcat、 Jetty以及 Undertow
生产准备特性:指标、健康检査、外部化配置等

组件自动装配

激活： @EnableAutoConfiguration 配置： /META-INF/spring.factories 实质为key-value形式实现： XXXAutoConfiguration

官方文档样例

import org.springframework.boot.*;
import org.springframework.boot.autoconfigure.*;
import org.springframework.web.bind.annotation.*;

@RestController
@EnableAutoConfiguration
public class Example {

 @RequestMapping("/")
 String home() {
 return "Hello World!";
 }

 public static void main(String[] args) throws Exception {
 SpringApplication.run(Example.class, args);
 }

}

实际上@SpringBootApplication已经包含该注解

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@SpringBootConfiguration
@EnableAutoConfiguration
@ComponentScan(excludeFilters = {
 @Filter(type = FilterType.CUSTOM, classes = TypeExcludeFilter.class),
 @Filter(type = FilterType.CUSTOM, classes = AutoConfigurationExcludeFilter.class) })
public @interface SpringBootApplication {
}

在IDEA中使用ctrl+shift+n可查看Spring源码中包含大量spring.factories文件

样例文件：

org/springframework/boot/spring-boot-autoconfigure/2.0.4.RELEASE/spring-boot-autoconfigure-2.0.4.RELEASE.jar!/META-INF/spring.factories

# Initializers
org.springframework.context.ApplicationContextInitializer=\
org.springframework.boot.autoconfigure.SharedMetadataReaderFactoryContextInitializer,\
org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener

# Application Listeners
org.springframework.context.ApplicationListener=\
org.springframework.boot.autoconfigure.BackgroundPreinitializer

# Auto Configuration Import Listeners
org.springframework.boot.autoconfigure.AutoConfigurationImportListener=\
org.springframework.boot.autoconfigure.condition.ConditionEvaluationReportAutoConfigurationImportListener

# Auto Configuration Import Filters
org.springframework.boot.autoconfigure.AutoConfigurationImportFilter=\
org.springframework.boot.autoconfigure.condition.OnClassCondition

# 自动装配配置，自动装载Bean列表
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
org.springframework.boot.autoconfigure.admin.SpringApplicationAdminJmxAutoConfiguration,\
org.springframework.boot.autoconfigure.web.servlet.WebMvcAutoConfiguration,\
...

WebMvcAutoConfiguration源码，其包含大量条件装配，若未检测到该类的依赖，则不会被装配

@Configuration
@ConditionalOnWebApplication(type = Type.SERVLET)
@ConditionalOnClass({ Servlet.class, DispatcherServlet.class, WebMvcConfigurer.class })
@ConditionalOnMissingBean(WebMvcConfigurationSupport.class)
@AutoConfigureOrder(Ordered.HIGHEST_PRECEDENCE + 10)
@AutoConfigureAfter({ DispatcherServletAutoConfiguration.class,
 ValidationAutoConfiguration.class })
public class WebMvcAutoConfiguration {
...

嵌入式Web容器

Web servlet: Tomcat、 Jetty和 Undertow
Web reactive Netty Web Server

生产准备特性

指标:/ actuator/ metrics
健康检查:/ actuator/ health
外部化配置:/ actuator/ configprops

Web应用介绍

传统 Servlet应用概览

Servlet组件: Servlet、 Filter、 Listener
Servlet注册: Servlet注解、 Spring Bean、 RegistrationBean
异步非阻塞:异步 Servlet、非阻塞 Servlet

传统 Servelt 应用

加入springboot应用通用依赖前缀多为spring-boot-starter-xxx

加入Web依赖

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
</dependency>

Maven依赖变化

由此可看出默认的Web容器为Tomcat（内嵌Servlet容器）

Servlet 组件

Servlet
- 实现
  - @WebServlet
  - HttpServlet
  - 注册
- URL 映射
  - @WebServlet(urlPatterns = "/my/servlet")
- 注册 @ServletComponentScan(basePackages = "top.fjy8018.springboot.web.servlet")
Filter
Listener

Servlet 注册

Servlet 注解

@ServletComponentScan +
- @WebServlet
- @WebFilter
- @WebListener

Spring Bean

@Bean +
- Servlet
- Filter
- Listener
RegistrationBean
- ServletRegistrationBean
- FilterRegistrationBean
- ServletListenerRegistrationBean

Servlet 实现

参考源码包位置，由于出现了Reactive Web容器，SpringBoot 1.x的包路径与2.x有所不同，2.x包路径变更为org.springframework.boot.autoconfigure.web.servlet

HttpServlet

package top.fjy8018.springboot.web.servlet;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

/**
 * 结合servlet规范注册
 *
 * @author F嘉阳
 * @date 2018-09-08 19:17
 */
@WebServlet(urlPatterns = "/myservlet")
public class MyServlet extends HttpServlet {

 @Override
 protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
 resp.getWriter().print("hello world");
 }
}

Servlet 3.0规范要求必须实现HttpServlet
其内部提供多种方法实现（ctrl+F12）
urlPatterns = "/myservlet"指明浏览器映射

注册

在主类中注册（包扫描）

package top.fjy8018.springboot;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.servlet.ServletComponentScan;

@SpringBootApplication
@ServletComponentScan(basePackages = "top.fjy8018.springboot.web.servlet")
public class OverviewApplication {

 public static void main(String[] args) {
 SpringApplication.run(OverviewApplication.class, args);
 }
}

启动后从控制台可看出映射已经成功

异步非阻塞 Servlet

异步 Servlet

javax.servlet.ServletRequest#startAsync()
javax.servlet.AsyncContext

非阻塞 Servlet

javax.servlet.ServletInputStream#setReadListener
- javax.servlet.ReadListener
javax.servlet.ServletOutputStream#setWriteListener
- javax.servlet.WriteListener

异步非阻塞Servlet实现

package top.fjy8018.springboot.web.servlet;

import javax.servlet.AsyncContext;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

/**
 * 结合servlet规范注册
 * asyncSupported指定异步servlet，默认false
 *
 * @author F嘉阳
 * @date 2018-09-08 19:17
 */
@WebServlet(urlPatterns = "/myservlet", asyncSupported = true)
public class MyServlet extends HttpServlet {

 @Override
 protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
 AsyncContext asyncContext = req.startAsync();

 // 执行任务
 asyncContext.start(() -> {
 try {
 resp.getWriter().print("hello world");

 // 触发完成，必须步骤，否则客户端发生超时
 asyncContext.complete();
 } catch (IOException e) {
 e.printStackTrace();
 }
 });
 }
}

asyncSupported = true 启用异步支持，否则报500错误

Spring Web MVC 应用介绍

Web mvo视图:模板引擎、内容协商、异常处理等
Web mVc rest:资源服务、资源跨域、服务发现等

Web MVC 视图

ViewResolver
View

源码

package org.springframework.web.servlet;

import java.util.Locale;
import org.springframework.lang.Nullable;

public interface ViewResolver {
 @Nullable
 View resolveViewName(String viewName, Locale locale) throws Exception;

}

通过指定名称和路径加载视图

视图渲染View

package org.springframework.web.servlet;

public interface View {
 ...

 void render(@Nullable Map<String, ?> model, HttpServletRequest request, HttpServletResponse response)
 throws Exception;

}

提供接口供不同的模板引擎实现ViewResolver

模板引擎

Thymeleaf
Freemarker
JSP

若同时有多种模板引擎，则需要内容协商

内容协商

ContentNegotiationConfigurer
ContentNegotiationStrategy
ContentNegotiatingViewResolver

异常处理

@ExceptionHandler
HandlerExceptionResolver
- ExceptionHandlerExceptionResolver
BasicErrorController (Spring Boot默认白页404)

源码

@Controller
@RequestMapping("${server.error.path:${error.path:/error}}")
public class BasicErrorController extends AbstractErrorController {
 
 @RequestMapping(produces = "text/html")
 public ModelAndView errorHtml(HttpServletRequest request,
 HttpServletResponse response) {
 HttpStatus status = getStatus(request);
 Map<String, Object> model = Collections.unmodifiableMap(getErrorAttributes(
 request, isIncludeStackTrace(request, MediaType.TEXT_HTML)));
 response.setStatus(status.value());
 ModelAndView modelAndView = resolveErrorView(request, response, status, model);
 return (modelAndView != null) ? modelAndView : new ModelAndView("error", model);
 }
 
 @RequestMapping
 @ResponseBody
 public ResponseEntity<Map<String, Object>> error(HttpServletRequest request) {
 Map<String, Object> body = getErrorAttributes(request,
 isIncludeStackTrace(request, MediaType.ALL));
 HttpStatus status = getStatus(request);
 return new ResponseEntity<>(body, status);
 }
}

其根据浏览器类型提供两种方式渲染，分别为html和json

{
 "timestamp": "2019-03-13T04:47:18.445+0000",
 "status": 404,
 "error": "Not Found",
 "message": "Not Found",
 "path": "/hello"
}

Web MVC REST

资源服务

@RequestMapping
@GetMapping
@ResponseBody
@RequestBody

从源码分析可知@GetMapping是从@RequestMapping变化而来，通过委派（别名）方式实现，并且在SpringMVC 4.3后引入

package org.springframework.web.bind.annotation;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import org.springframework.core.annotation.AliasFor;

/**
 * Annotation for mapping HTTP {@code GET} requests onto specific handler
 * methods.
 *
 * <p>Specifically, {@code @GetMapping} is a <em>composed annotation</em> that
 * acts as a shortcut for {@code @RequestMapping(method = RequestMethod.GET)}.
 *
 *
 * @author Sam Brannen
 * @since 4.3
 * @see PostMapping
 * @see PutMapping
 * @see DeleteMapping
 * @see PatchMapping
 * @see RequestMapping
 */
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@RequestMapping(method = RequestMethod.GET)
public @interface GetMapping {
 /**
 * 委派目标 {@link RequestMapping#name}.
 */
 @AliasFor(annotation = RequestMapping.class)
 String name() default "";
 ...
}

资源跨域

CrossOrigin
WebMvcConfigurer#addCorsMappings
传统解决方案
- IFrame
- JSONP

源码

package org.springframework.web.servlet.config.annotation;

/**
 * @since 3.1
 */
public interface WebMvcConfigurer {
 /**
 * 跨域配置
 * @since 4.2
 */
 default void addCorsMappings(CorsRegistry registry) {
 }
}

注解实现方式

package org.springframework.web.bind.annotation;

/**
 * @since 4.2
 */
@Target({ ElementType.METHOD, ElementType.TYPE })
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface CrossOrigin {...}

两者均来自springframework，属于不同的实现维度（接口编程 / 注解驱动）

服务发现

HATEOS

核心组件

DispatcherServlet——Servlet实现请求转发到映射URL
HandlerMapping——处理映射URL方法
HandlerAdapter——方法处理
ViewResolver——交由视图处理
…

Spring WebFlux （Spring 5+）

Reactor 基础

Java Lambda —— Reactive Stream实现（JDK 9 flow api、RxJava）
Mono
Flux

Web Flux 核心

Web MVC 注解兼容

@Controller
@RequestMapping
@ResponseBody
@RequestBody
…

函数式声明

RouterFunction

异步非阻塞

Servlet 3.1 +
Netty Reactor

使用场景

页面渲染
REST 应用

性能测试

国外博客

结论：Webflux 性能不会显著提升，但Webflux提升吞吐量

Web Server 应用

切换 Web server
自定义 Servlet Web server
自定义 Reactive Web server

切换 Web Server

切换其他 Servlet 容器

Tomcat -> Jetty

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 <exclusions>
 <!-- tomcat加载优先级高于jetty，若不加排除，则仍然加载tomcat -->
 <exclusion>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-tomcat</artifactId>
 </exclusion>
 </exclusions>
</dependency>
<!-- 使用 Jetty 容器 -->
<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-jetty</artifactId>
</dependency>

替换 Servlet 容器

WebFlux

<!--使用webflux必须注释web依赖，当web和flux共存时，web优先级更高-->
<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

注释后相关类和的注解也已经失效，需要删除

@SpringBootApplication
//@ServletComponentScan(basePackages = "top.fjy8018.springboot.web.servlet")
public class SpringbootApplication {

 public static void main(String[] args) {
 SpringApplication.run(SpringbootApplication.class, args);
 }
}

依赖变化

启动后变为Netty容器

自定义 Servlet Web Server

WebServerFactoryCustomizer

SpringBoot 2.0 新增支持接口

package org.springframework.boot.web.server;

/**
 * @since 2.0.0
 * @see WebServerFactoryCustomizerBeanPostProcessor
 */
@FunctionalInterface
public interface WebServerFactoryCustomizer<T extends WebServerFactory> {
 
 void customize(T factory);
}

该接口为主接口，其下有大量实现，IDEA通过ctrl+alt+B或者ctrl+alt+左键点击查看

其中TomcatServletWebServerFactoryCustomizer就是实现类之一，但在启用Webflux后该类无效

package org.springframework.boot.autoconfigure.web.servlet;

/**
 * {@link WebServerFactoryCustomizer} to apply {@link ServerProperties} to Tomcat web
 * servers.
 *
 * @author Brian Clozel
 * @author Phillip Webb
 * @since 2.0.0
 */
public class TomcatServletWebServerFactoryCustomizer
 implements WebServerFactoryCustomizer<TomcatServletWebServerFactory>, Ordered {
 @Override
 public void customize(TomcatServletWebServerFactory factory) {
 ServerProperties.Tomcat tomcatProperties = this.serverProperties.getTomcat();
 if (!ObjectUtils.isEmpty(tomcatProperties.getAdditionalTldSkipPatterns())) {
 factory.getTldSkipPatterns()
 .addAll(tomcatProperties.getAdditionalTldSkipPatterns());
 }
 if (tomcatProperties.getRedirectContextRoot() != null) {
 customizeRedirectContextRoot(factory,
 tomcatProperties.getRedirectContextRoot());
 }
 if (tomcatProperties.getUseRelativeRedirects() != null) {
 customizeUseRelativeRedirects(factory,
 tomcatProperties.getUseRelativeRedirects());
 }
 }
 ...
}

自定义 Reactive Web Server

ReactiveWebServerFactoryCustomizer

其源码显示提供了最基础的实现，包括端口、SSL等，允许自行扩展实现

package org.springframework.boot.autoconfigure.web.reactive;

/**
 * {@link WebServerFactoryCustomizer} to apply {@link ServerProperties} to reactive
 * servers.
 * @since 2.0.0
 */
public class ReactiveWebServerFactoryCustomizer implements
 WebServerFactoryCustomizer<ConfigurableReactiveWebServerFactory>, Ordered {

 private final ServerProperties serverProperties;

 public ReactiveWebServerFactoryCustomizer(ServerProperties serverProperties) {
 this.serverProperties = serverProperties;
 }

 @Override
 public int getOrder() {
 return 0;
 }

 @Override
 public void customize(ConfigurableReactiveWebServerFactory factory) {
 PropertyMapper map = PropertyMapper.get().alwaysApplyingWhenNonNull();
 map.from(this.serverProperties::getPort).to(factory::setPort);
 map.from(this.serverProperties::getAddress).to(factory::setAddress);
 map.from(this.serverProperties::getSsl).to(factory::setSsl);
 map.from(this.serverProperties::getCompression).to(factory::setCompression);
 map.from(this.serverProperties::getHttp2).to(factory::setHttp2);
 }
}

数据相关介绍

关系型数据

JDBC:数据源、 JdbcTemplate、自动装配
JPA:实体映射关系、实体操作、自动装配
事务∶ Spring事务抽象、JDBC事务处理、自动装配

JDBC

依赖

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>

依赖变化，在SpringBoot2.0 后依赖新的连接池 HikariCP

数据源

javax.sql.DataSource
JdbcTemplate
自动装配
DataSourceAutoConfiguration

DataSource

连接池多数实现是对DataSource的二次封装，提高执行效率

package javax.sql;

/**
 * @since 1.4
 */

public interface DataSource extends CommonDataSource, Wrapper {
 Connection getConnection() throws SQLException;
 
 Connection getConnection(String username, String password)
 throws SQLException;
}

自动装配

JdbcTemplate自动装配，其依赖数据源配置项，故有@AutoConfigureAfter注解标识，该类执行于DataSourceAutoConfiguration之后，额外还有多个条件装配的标识

package org.springframework.boot.autoconfigure.jdbc;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for {@link JdbcTemplate} and
 * {@link NamedParameterJdbcTemplate}.
 * @since 1.4.0
 */
@Configuration
@ConditionalOnClass({ DataSource.class, JdbcTemplate.class })
@ConditionalOnSingleCandidate(DataSource.class)
@AutoConfigureAfter(DataSourceAutoConfiguration.class)
@EnableConfigurationProperties(JdbcProperties.class)
public class JdbcTemplateAutoConfiguration {...}

DataSourceAutoConfiguration

package org.springframework.boot.autoconfigure.jdbc;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for {@link DataSource}.
 */
@Configuration
@ConditionalOnClass({ DataSource.class, EmbeddedDatabaseType.class })
@EnableConfigurationProperties(DataSourceProperties.class)
@Import({ DataSourcePoolMetadataProvidersConfiguration.class,
 DataSourceInitializationConfiguration.class })
public class DataSourceAutoConfiguration {...}

该类属于自动装配，故必然在spring.factories中，而数据源下即JdbcTemplateAutoConfiguration

此时直接启动会得到缺少数据源的错误报告

***************************
APPLICATION FAILED TO START
***************************

Description:

Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.

Reason: Failed to determine a suitable driver class


Action:

Consider the following:
 If you want an embedded database (H2, HSQL or Derby), please put it on the classpath.
 If you have database settings to be loaded from a particular profile you may need to activate it (no profiles are currently active).

JPA

依赖

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>

其属于SpringData项目，基于hibernate实现，同时也自动依赖JDBC

实体映射关系

@javax.persistence.OneToOne
@javax.persistence.OneToMany
@javax.persistence.ManyToOne
@javax.persistence.ManyToMany
…

实体操作

javax.persistence.EntityManager

属于Java类库的API

package javax.persistence;

public interface EntityManager {

 /**
 * 存
 */
 public void persist(Object entity);

 /**
 * 更新
 */
 public <T> T merge(T entity);

 /**
 * 删除
 */
 public void remove(Object entity);

 /**
 * 查询以及多个重载方法
 */
 public <T> T find(Class<T> entityClass, Object primaryKey);

 public <T> T find(Class<T> entityClass, Object primaryKey,
 Map<String, Object> properties);
 ...
}

自动装配

HibernateJpaAutoConfiguration

该类也同属于自动装配，且需要JDBC提供数据源，故其加载顺序标识为DataSourceAutoConfiguration之后

package org.springframework.boot.autoconfigure.orm.jpa;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for Hibernate JPA.
 */
@Configuration
@ConditionalOnClass({ LocalContainerEntityManagerFactoryBean.class, EntityManager.class })
@Conditional(HibernateEntityManagerCondition.class)
@EnableConfigurationProperties(JpaProperties.class)
@AutoConfigureAfter({ DataSourceAutoConfiguration.class })
@Import(HibernateJpaConfiguration.class)
public class HibernateJpaAutoConfiguration {...}

事务（Transaction）

依赖

<dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-tx</artifactId>
</dependency>

Spring 事务抽象

PlatformTransactionManager

该接口属于事务根接口，不仅限于数据库事务，默认实现包含分布式事务JtaTransactionManager和数据库事务DataSourceTransactionManager

package org.springframework.transaction;

import org.springframework.lang.Nullable;

/**
 * <p>The default implementations of this strategy interface are
 * {@link org.springframework.transaction.jta.JtaTransactionManager} and
 * {@link org.springframework.jdbc.datasource.DataSourceTransactionManager}

 * @see org.springframework.transaction.support.TransactionTemplate
 * @see org.springframework.transaction.interceptor.TransactionInterceptor
 * @see org.springframework.transaction.interceptor.TransactionProxyFactoryBean
 */
public interface PlatformTransactionManager {...}

JDBC 事务处理

DataSourceTransactionManager

内部通过IOC自动注入数据源接口

package org.springframework.jdbc.datasource;

@SuppressWarnings("serial")
public class DataSourceTransactionManager extends AbstractPlatformTransactionManager
 implements ResourceTransactionManager, InitializingBean {

 @Nullable
 private DataSource dataSource;
 ...
}

自动装配

TransactionAutoConfiguration

自动装配的条件是Spring事务依赖存在PlatformTransactionManager

package org.springframework.boot.autoconfigure.transaction;

/**
 * {@link org.springframework.boot.autoconfigure.EnableAutoConfiguration
 * Auto-configuration} for Spring transaction.
 * @since 1.3.0
 */
@Configuration
@ConditionalOnClass(PlatformTransactionManager.class)
@AutoConfigureAfter({ JtaAutoConfiguration.class, HibernateJpaAutoConfiguration.class,
 DataSourceTransactionManagerAutoConfiguration.class,
 Neo4jDataAutoConfiguration.class })
@EnableConfigurationProperties(TransactionProperties.class)
public class TransactionAutoConfiguration {...}

功能扩展介绍

Spring Boot 应用

Spring Application:失败分析、应用特性、事件监听等
Spring Boot配置:外部化配置、 Profile、配置属性
Spring Boot starter: Starter开发、最佳实践

SpringApplication

失败分析

FailureAnalysisReporter

该类在Spring Boot 1.4后开始支持

package org.springframework.boot.diagnostics;

/**
 * Reports a {@code FailureAnalysis} to the user.
 * @since 1.4.0
 */
@FunctionalInterface
public interface FailureAnalysisReporter {...}

应用特性

SpringApplication Fluent API

原始默认启动方法为SpringApplication.run(SpringbootApplication.class, args)其实际返回可配置上下文对象ConfigurableApplicationContext，可通过Builder模式重写

package top.fjy8018.springboot;

@SpringBootApplication
public class SpringbootApplication {

 public static void main(String[] args) {
 // Fluent API
 // 等价建造者设计模式调用方法
 new SpringApplicationBuilder(SpringbootApplication.class)
 // 不使用web模式
 .web(WebApplicationType.NONE)
 // 自定义参数
 .properties("name=fjy")
 .run(args);
 }
}

2.0 版本下提供3种WEB容器类型

package org.springframework.boot;

/**
 * @since 2.0.0
 */
public enum WebApplicationType {
 NONE,
 SERVLET,
 REACTIVE
}

Spring Boot 配置

外部化配置

ConfigurationProperty
@Profile

从SpringBoot 2.0 后引入，Origin类型支持其他配资源引入，非传统仅支持键值对，由此支持多数据源

package org.springframework.boot.context.properties.source;

/**
 * @since 2.0.0
 */
public final class ConfigurationProperty
 implements OriginProvider, Comparable<ConfigurationProperty> {
 
 private final ConfigurationPropertyName name;

 private final Object value;

 private final Origin origin;
 ...
}

官方文档

此处官方文档存在一定疏漏，部分场景配置无法实现

Spring Boot uses a very particular PropertySource order that is designed to allow sensible overriding of values. Properties are considered in the following order:

Devtools global settings properties on your home directory (~/.spring-boot-devtools.properties when devtools is active).

@TestPropertySource annotations on your tests.

@SpringBootTest#properties annotation attribute on your tests.

Command line arguments.

Properties from SPRING_APPLICATION_JSON (inline JSON embedded in an environment variable or system property).

ServletConfig init parameters.

ServletContext init parameters.

JNDI attributes from java:comp/env.

Java System properties (System.getProperties()).

OS environment variables.

A RandomValuePropertySource that has properties only in random.*.

Profile-specific application properties outside of your packaged jar (application-{profile}.properties and YAML variants).

Profile-specific application properties packaged inside your jar (application-{profile}.properties and YAML variants).

Application properties outside of your packaged jar (application.properties and YAML variants).

Application properties packaged inside your jar (application.properties and YAML variants).

@PropertySource annotations on your @Configuration classes.

Default properties (specified by setting SpringApplication.setDefaultProperties).

Profile支持以注解方式条件开启配置，用于区分开发环境和生产环境等

package org.springframework.context.annotation;

/**
 * @since 3.1
 */
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(ProfileCondition.class)
public @interface Profile {
 
 String[] value();
}

配置属性

PropertySources

提供两种编程方式：接口、注解，支持保存多个配置，按照17种顺序排列，该接口在SpringBoot中重新设计变为ConfigurationProperty

package org.springframework.core.env;

/**
 * @since 3.1
 * @see PropertySource
 */
public interface PropertySources extends Iterable<PropertySource<?>> {...}

运维管理介绍

官方文档

Spring Boot Actuator

端点（Endpoints）

Web Endpoints
JMX Endpoints

依赖

<!--运维监控-->
<dependency>
 <groupId>org.springframework.boot</groupId> 
 <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

启动效果

访问健康检查接口/actuator/health返回应用状态

{
 "status": "UP"
}

访问根路径

{
 "_links": {
 "self": {
 "href": "http://localhost:8080/actuator",
 "templated": false
 },
 "health": {
 "href": "http://localhost:8080/actuator/health",
 "templated": false
 },
 "info": {
 "href": "http://localhost:8080/actuator/info",
 "templated": false
 }
 }
}

依据SpringBoot打印的进程号或者依据应用名称使用Jconsole监控，该操作使用的为JMX端点

健康检查（Health Checks）

Health
HealthIndicator

使用JMX在默认情况下能获取比Web端点更多的信息，原因是JMX支持使用安全连接，具有更好的安全性

官方文档指出默认状态只暴露health和info，可通过配置文件调整

28.4 Actuator Security
For security purposes, all actuators other than /health and /info are disabled by default. The management.endpoints.web.exposure.include property can be used to enable the actuators.

在application.properties文件中加入配置重启应用

# 暴露所有健康监控点
management.endpoints.web.exposure.include=*

从日志输出可知增加了更多的URL映射

但尽管配置后，JMX显示的信息仍然比Web端更丰富，例如在JMX中不仅看到状态，还能看到磁盘、内存等信息，而Web端仅看到状态

指标（Metrics）

内建 Metrics
Web Endpoint : /actuator/metrics
自定义 Metrics

Web端点列举支持指标

{
 "names": [
 "jvm.memory.max",
 "jvm.gc.memory.promoted",
 "jvm.memory.used",
 "jvm.gc.max.data.size",
 "jvm.gc.pause",
 "jvm.memory.committed",
 "system.cpu.count",
 "logback.events",
 "jvm.buffer.memory.used",
 "jvm.threads.daemon",
 "system.cpu.usage",
 "jvm.gc.memory.allocated",
 "jvm.threads.live",
 "jvm.threads.peak",
 "process.uptime",
 "http.server.requests",
 "process.cpu.usage",
 "jvm.classes.loaded",
 "jvm.classes.unloaded",
 "jvm.gc.live.data.size",
 "jvm.buffer.count",
 "jvm.buffer.total.capacity",
 "process.start.time"
 ]
}

访问/actuator/metrics/jvm.memory.max可查看相关指标，转换后可知最大内存为5.2G

{
 "name": "jvm.memory.max",
 "description": "The maximum amount of memory in bytes that can be used for memory management",
 "baseUnit": "bytes",
 "measurements": [
 {
 "statistic": "VALUE",
 "value": 5595725823
 }
 ],
 "availableTags": [
 {
 "tag": "area",
 "values": [
 "heap",
 "nonheap"
 ]
 },
 {
 "tag": "id",
 "values": [
 "Compressed Class Space",
 "PS Survivor Space",
 "PS Old Gen",
 "Metaspace",
 "PS Eden Space",
 "Code Cache"
 ]
 }
 ]
}

设计模式学习——行为型

Wed, 22 Jul 2020 16:50:17 +0800

模板方法模式

介绍

定义:定义了一个算法的骨架,并允许子类为一个或多个步骤提供实现

模板方法使得子类可以在不改变算法结构的情况下,重新定义算法的某些步骤

类型：行为型

适用场景

一次性实现一个算法的不变的部分,并将可变的行为留给子类来实现
各子类中公共的行为被提取出来并集中到一个公共父类中, 从而避免代码重复

优点

提高复用性
提高扩展性
符合开闭原则

缺点

类数目增加
增加了系统实现的复杂度
继承关系自身缺点,如果父类添加新的抽象方法,所有子类都要改一遍

分类和扩展

钩子方法
- 提供了默认的行为，子类可以在必要时进行扩展

相关设计模式

模板方法模式和工厂方法模式
- 工厂方法是模板方法的一种特殊实现
模板方法模式和策略模式
- 策略模式在于实现算法可以互相替换而不影响应用层，不改变算法流程
- 模板方法定义了一个算法的骨架，不明确的交给子类实现，可以改变算法流程

代码样例

类图

抽象课程

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;
import org.apache.ibatis.mapping.MappedStatement;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.Collection;

/**
 * 抽象课程
 * <p>
 * {@link java.util.AbstractList} 提供抽象List实现
 * 例如 {@link java.util.AbstractList#addAll(int, Collection)} 定义通过模板
 * {@link java.util.AbstractMap} 同理
 * <p>
 * {@link javax.servlet.http.HttpServlet#doGet(HttpServletRequest, HttpServletResponse)} 也是模板定义
 * {@link org.apache.ibatis.executor.BaseExecutor#doUpdate(MappedStatement, Object)} 等方法交由子类实现
 * 例如 {@link org.apache.ibatis.executor.SimpleExecutor#doUpdate(MappedStatement, Object)}
 *
 * @author F嘉阳
 * @date 2020/3/4 10:59
 */
@Slf4j
public abstract class ACourse {

 /**
 * 固定不变的行为
 * 定义模板，不允许子类覆盖，防止影响流程执行顺序
 */
 protected final void makeCourse() {
 this.makePPT();
 this.makeVideo();
 // 调用可选方法
 if (needWriteArticle()) {
 this.writeArticle();
 }
 this.packageCourse();
 }

 /**
 * 固定不变的行为
 * 满足所有子类需求
 */
 final void makePPT() {
 log.info("制作PPT");
 }

 final void makeVideo() {
 log.info("制作视频");
 }

 /**
 * 可选行为
 * <p>
 * 手记可写可不写，但写的手记的流程还是一致的，故也声明为final
 * 通过定义钩子方法，允许子类覆盖
 */
 final void writeArticle() {
 log.info("编写手记");
 }

 /**
 * 钩子方法，提供默认实现
 *
 * @return
 */
 protected boolean needWriteArticle() {
 return false;
 }

 /**
 * 必须实现的行为
 * 课程内容打包
 */
 abstract void packageCourse();
}

具体课程实现

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;

/**
 * 具体课程实现
 *
 * @author F嘉阳
 * @date 2020/3/4 11:05
 */
@Slf4j
public class JavaCourse extends ACourse {
 /**
 * 必须实现的行为
 * 课程内容打包
 */
 @Override
 void packageCourse() {
 log.info("提供Java学习课程源码");
 }
}

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;

/**
 * 具体课程实现
 * @author F嘉阳
 * @date 2020/3/4 11:06
 */
@Slf4j
public class VueCourse extends ACourse {

 private boolean needWriteArticleFlag = false;

 /**
 * 通过构造器开放给实例决定
 * @param needWriteArticleFlag
 */
 public VueCourse(boolean needWriteArticleFlag) {
 this.needWriteArticleFlag = needWriteArticleFlag;
 }

 /**
 * 必须实现的行为
 * 课程内容打包
 */
 @Override
 void packageCourse() {
 log.info("提供Vue学习课程源码");
 log.info("提供Vue学习课程多媒体素材");
 }

 /**
 * 钩子方法，提供默认实现
 *
 * @return
 */
 @Override
 protected boolean needWriteArticle() {
 return this.needWriteArticleFlag;
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/4 11:08
 */
@Slf4j
class ACourseTest {

 @Test
 void makeJavaCourse() {
 ACourse javaCourse = new JavaCourse();
 javaCourse.makeCourse();
 }

 @Test
 void makeVueCourse() {
 ACourse vueCourse = new VueCourse(true);
 vueCourse.makeCourse();
 }
}

输出

11:27:56.942 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作PPT
11:27:56.957 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作视频
11:27:56.957 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.JavaCourse - 提供Java学习课程源码


11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作PPT
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作视频
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 编写手记
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.VueCourse - 提供Vue学习课程源码
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.VueCourse - 提供Vue学习课程多媒体素材

迭代器模式

介绍

定义提供一种方法,顺序访问一个集合对象中的各个元素,而又不暴露该对象的内部表示

类型：行为型

适用场景

访问一个集合对象的内容而无需暴露它的内部表示
为遍历不同的集合结构提供一个统一的接口

优点

分离了集合对象的遍历行为

缺点

类的个数成对增加

相关设计模式

迭代器模式和访问者模式
- 访问者模式扩展开放的部分，再作用于对象的操作上
- 迭代器模式扩展开放的部分，再作用于对象的种类上

代码样例

迭代器一般使用现成的，而不会自己实现

类图

课程接口

package top.fjy8018.designpattern.pattern.behavior.iterator;

import lombok.AllArgsConstructor;
import lombok.Getter;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:49
 */
@AllArgsConstructor
@Getter
public class Course {

 private String name;

}

迭代器

package top.fjy8018.designpattern.pattern.behavior.iterator;

import java.util.ArrayList;

/**
 * JDK迭代器
 * {@link java.util.Iterator}
 * 集合内部类实现 {@link ArrayList.Itr#hasNext()}
 * 同时做了进一步扩展 {@link ArrayList.ListItr}
 *
 * mybatis中 {@link org.apache.ibatis.cursor.defaults.DefaultCursor}
 * 其持有游标迭代器对象 {@link org.apache.ibatis.cursor.defaults.DefaultCursor.CursorIterator}
 *
 * @author F嘉阳
 * @date 2020/3/4 14:49
 */
public interface CourseIterator {

 /**
 * 获取下一个课程
 *
 * @return
 */
 Course nextCourse();

 /**
 * 是否是最后一个课程
 * @return
 */
 boolean isLastCourse();
}

迭代器实现

package top.fjy8018.designpattern.pattern.behavior.iterator;

import lombok.extern.slf4j.Slf4j;

import java.util.List;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:50
 */
@Slf4j
public class CourseIteratorImpl implements CourseIterator {

 private List courseList;

 private int position;

 private Course course;

 public CourseIteratorImpl(List courseList){
 this.courseList=courseList;
 }

 @Override
 public Course nextCourse() {
 log.info("返回课程,位置是: {}",position);
 course=(Course)courseList.get(position);
 position++;
 return course;
 }

 @Override
 public boolean isLastCourse(){
 if(position< courseList.size()){
 return false;
 }
 return true;
 }
}

对课程进行处理

package top.fjy8018.designpattern.pattern.behavior.iterator;

/**
 * 对课程进行处理
 *
 * @author F嘉阳
 * @date 2020/3/4 14:48
 */
public interface CourseAggregate {

 void addCourse(Course course);

 void removeCourse(Course course);

 /**
 * 获取课程迭代器
 * @return
 */
 CourseIterator getCourseIterator();
}

实现

package top.fjy8018.designpattern.pattern.behavior.iterator;

import java.util.ArrayList;
import java.util.List;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:50
 */
public class CourseAggregateImpl implements CourseAggregate {

 private List courseList;

 public CourseAggregateImpl() {
 this.courseList = new ArrayList();
 }

 @Override
 public void addCourse(Course course) {
 courseList.add(course);
 }

 @Override
 public void removeCourse(Course course) {
 courseList.remove(course);
 }

 @Override
 public CourseIterator getCourseIterator() {
 return new CourseIteratorImpl(courseList);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.iterator;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:52
 */
@Slf4j
class CourseIteratorTest {

 @Test
 void nextCourse() {
 Course course1 = new Course("Java电商一期");
 Course course2 = new Course("Java电商二期");
 Course course3 = new Course("Java设计模式精讲");
 Course course4 = new Course("Python课程");
 Course course5 = new Course("算法课程");
 Course course6 = new Course("前端课程");


 CourseAggregate courseAggregate = new CourseAggregateImpl();

 courseAggregate.addCourse(course1);
 courseAggregate.addCourse(course2);
 courseAggregate.addCourse(course3);
 courseAggregate.addCourse(course4);
 courseAggregate.addCourse(course5);
 courseAggregate.addCourse(course6);

 log.info("-----课程列表-----");
 printCourses(courseAggregate);

 courseAggregate.removeCourse(course4);
 courseAggregate.removeCourse(course5);

 log.info("-----删除操作之后的课程列表-----");
 printCourses(courseAggregate);
 }

 public static void printCourses(CourseAggregate courseAggregate){
 CourseIterator courseIterator= courseAggregate.getCourseIterator();
 while(!courseIterator.isLastCourse()){
 Course course=courseIterator.nextCourse();
 log.info(course.getName());
 }
 }
}

输出

15:18:23.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - -----课程列表-----
15:18:23.124 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 0
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java分布式
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 1
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java系统设计
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 2
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java设计模式精讲
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 3
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Python课程
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 4
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 算法课程
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 5
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 前端课程
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - -----删除操作之后的课程列表-----
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 0
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java分布式
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 1
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java系统设计
15:18:23.127 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 2
15:18:23.127 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java设计模式精讲
15:18:23.132 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 3
15:18:23.132 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 前端课程

策略模式

介绍

定义：定义了算法家族,分别封装起来,让它们之间可以互相替换, 此模式让算法的变化不会影响到使用算法的用户。

例如有大量if…else的情况

类型：行为型

适用场景

系统有很多类，而他们的区别仅仅在于他们的行为不同
一个系统需要动态地在几种算法中选择一种

优点

开闭原则
避免使用多重条件转移语句
提高算法的保密性和安全性

缺点

客户端必须知道所有的策略类,并自行决定使用哪一个策略类。
产生很多策略类

相关设计模式

策略模式和工厂模式
策略模式和状态模式

代码样例

改造前——初级策略模式

无法避免if…else

类图

促销接口

package top.fjy8018.designpattern.pattern.behavior.strategy;

/**
 * 促销接口
 *
 * @author F嘉阳
 * @date 2020/3/4 15:37
 */
public interface PromotionStrategy {

 void doPromotion();
}

满减促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 满减促销策略
 *
 * @author F嘉阳
 * @date 2020/3/4 15:38
 */
@Slf4j
public class ManJianPromotionStrategy implements PromotionStrategy {
 @Override
 public void doPromotion() {
 log.info("满减促销，满200-20");
 }
}

立减促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 立减促销策略
 *
 * @author F嘉阳
 * @date 2020/3/4 15:39
 */
@Slf4j
public class LiJianPromotionStrategy implements PromotionStrategy{
 @Override
 public void doPromotion() {
 log.info("立减促销，课程价格直接减去配置的价格");
 }
}

返现促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 返现促销策略
 *
 * @author F嘉阳
 * @date 2020/3/4 15:38
 */
@Slf4j
public class FanXianPromotionStrategy implements PromotionStrategy{
 @Override
 public void doPromotion() {
 log.info("返现促销，返回金额存放到用户余额中");
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/4 15:46
 */
@Slf4j
class PromotionStrategyTest {

 @Test
 void existIfElse() {
 String promotionKey = "LIJIAN";
 PromotionActivity activity = null;
 if (StringUtils.equals(promotionKey,"LIJIAN")){
 // 方法和类本身线程安全
 activity= new PromotionActivity(new LiJianPromotionStrategy());
 }else if(StringUtils.equals(promotionKey,"MANJIAN")){
 activity = new PromotionActivity(new FanXianPromotionStrategy());
 }

 activity.executePromotionStrategy();
 }
}

输出

16:08:28.582 [main] INFO top.fjy8018.designpattern.pattern.behavior.strategy.LiJianPromotionStrategy - 立减促销，课程价格直接减去配置的价格

改造后——结合工厂和策略模式

类图

空促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 空策略
 *
 * @author F嘉阳
 * @date 2020/3/4 16:16
 */
@Slf4j
public class EmptyPromotionStrategy implements PromotionStrategy {
 @Override
 public void doPromotion() {
 log.info("空促销策略");
 }
}

促销策略工厂

package top.fjy8018.designpattern.pattern.behavior.strategy;

import org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory;
import org.springframework.beans.factory.support.CglibSubclassingInstantiationStrategy;
import org.springframework.beans.factory.support.SimpleInstantiationStrategy;
import org.springframework.core.io.FileSystemResource;
import org.springframework.core.io.PathResource;

import java.util.Arrays;
import java.util.Comparator;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

/**
 * 促销策略工厂
 * <p>
 * {@link java.util.Comparator} 比较器，类似于抽象策略，有大量实现类，所有具体策略都要实现定义好的接口方法
 * 例如 {@link com.google.common.collect.ExplicitOrdering}
 * 应用层可以逐级传递比较器 {@link Arrays#sort(Object[], Comparator)}
 * {@link java.util.TreeMap#compare(Object, Object)} 利用内部的比较器处理
 * <p>
 * Spring资源策略抽象 {@link org.springframework.core.io.Resource}
 * 提供大量策略实现 {@link PathResource} {@link FileSystemResource}
 * <p>
 * {@link org.springframework.beans.factory.support.InstantiationStrategy} 初始化策略接口
 * 初始化具体策略实现 {@link SimpleInstantiationStrategy} {@link CglibSubclassingInstantiationStrategy}
 * 初始化策略在 {@link AbstractAutowireCapableBeanFactory} 中被使用
 *
 * @author F嘉阳
 * @date 2020/3/4 16:09
 */
public class PromotionStrategyFactory {

 private static Map<String, PromotionStrategy> STRATEGY_MAP = new ConcurrentHashMap<>();

 /**
 * 无促销
 */
 private static final PromotionStrategy NON_PROMOTION = new EmptyPromotionStrategy();

 static {
 STRATEGY_MAP.put(PromotionKey.FANXIAN, new FanXianPromotionStrategy());
 STRATEGY_MAP.put(PromotionKey.MANJIAN, new ManJianPromotionStrategy());
 STRATEGY_MAP.put(PromotionKey.LIJIAN, new LiJianPromotionStrategy());
 }

 /**
 * 禁止实例化
 */
 private PromotionStrategyFactory() {
 }

 /**
 * 开放获取策略的方法
 *
 * @param promotionKey
 * @return
 */
 public static PromotionStrategy getPromotionStrategy(String promotionKey) {
 PromotionStrategy strategy = STRATEGY_MAP.get(promotionKey);
 return strategy == null ? NON_PROMOTION : strategy;
 }

 /**
 * 常量定义
 */
 private interface PromotionKey {
 String LIJIAN = "LIJIAN";
 String FANXIAN = "FANXIAN";
 String MANJIAN = "MANJIAN";
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/3/4 15:46
 */
@Slf4j
class PromotionStrategyTest {

 @Test
 void nonIfElse() {
 String promotionKey = "LIJIAN";
 PromotionActivity activity = new PromotionActivity(PromotionStrategyFactory.getPromotionStrategy(promotionKey));
 activity.executePromotionStrategy();
 }
}

输出

16:08:28.582 [main] INFO top.fjy8018.designpattern.pattern.behavior.strategy.LiJianPromotionStrategy - 立减促销，课程价格直接减去配置的价格

解释器模式

介绍

定义：给定一个语言，定义它的文法的一种表示，并定义一个解释器，这个解释器使用该表示来解释语言中的句子。
为了解释一种语言，而为语言创建的解释器
类型：行为型

适用场景（较少）

某个特定类型问题发生频率足够高

优点

语法由很多类表示,容易改变及扩展此语言

缺点

当语法规则数目太多时,增加了系统复杂度

相关设计模式

解释器模式和适配器模式

代码样例

类图

解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * 解释器
 * <p>
 * JDK 源码
 * {@link java.util.regex.Pattern} 正则解释器
 * {@link org.springframework.expression.spel.standard.SpelExpressionParser} Spring解释器
 *
 * @author F嘉阳
 * @date 2020/3/5 14:45
 */
public interface Interpreter {
 int interpret();
}

加法解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * 加法解释器
 *
 * @author F嘉阳
 * @date 2020/3/5 14:46
 */
public class AddInterpreter implements Interpreter {

 /**
 * 定义两个表达式
 */
 private Interpreter firstExpression, secondExpression;

 public AddInterpreter(Interpreter firstExpression, Interpreter secondExpression) {
 this.firstExpression = firstExpression;
 this.secondExpression = secondExpression;
 }

 @Override
 public int interpret() {
 // 此时传入为NumberInterpreter类型，调interpret方法转int类型
 return this.firstExpression.interpret() + this.secondExpression.interpret();
 }

 @Override
 public String toString() {
 return "+";
 }
}

乘法解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * 乘法解释器
 *
 * @author F嘉阳
 * @date 2020/3/5 14:47
 */
public class MultiInterpreter implements Interpreter {

 private Interpreter firstExpression, secondExpression;

 public MultiInterpreter(Interpreter firstExpression, Interpreter secondExpression) {
 this.firstExpression = firstExpression;
 this.secondExpression = secondExpression;
 }

 @Override
 public int interpret() {
 return this.firstExpression.interpret() * this.secondExpression.interpret();
 }

 @Override
 public String toString() {
 return "*";
 }
}

类型解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:47
 */
public class NumberInterpreter implements Interpreter {
 private int number;

 public NumberInterpreter(int number) {
 this.number = number;
 }

 public NumberInterpreter(String number) {
 this.number = Integer.parseInt(number);
 }

 @Override
 public int interpret() {
 return this.number;
 }
}

操作工具类

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:47
 */
public class OperatorUtil {
 public static boolean isOperator(String symbol) {
 return (symbol.equals("+") || symbol.equals("*"));

 }

 public static Interpreter getExpressionObject(Interpreter firstExpression, Interpreter secondExpression, String symbol) {
 if (symbol.equals("+")) {
 return new AddInterpreter(firstExpression, secondExpression);
 } else if (symbol.equals("*")) {
 return new MultiInterpreter(firstExpression, secondExpression);
 }
 return null;
 }
}

自定义解释器实现

package top.fjy8018.designpattern.pattern.behavior.interpreter;

import lombok.extern.slf4j.Slf4j;

import java.util.Stack;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:48
 */
@Slf4j
public class CustomExpressionParser {
 private Stack<Interpreter> stack = new Stack<Interpreter>();

 public int parse(String str) {
 String[] strItemArray = str.split(" ");
 for (String symbol : strItemArray) {
 if (!OperatorUtil.isOperator(symbol)) {
 Interpreter numberExpression = new NumberInterpreter(symbol);
 stack.push(numberExpression);
 log.info(String.format("入栈: %d", numberExpression.interpret()));
 } else {
 //是运算符号，可以计算
 Interpreter firstExpression = stack.pop();
 Interpreter secondExpression = stack.pop();
 log.info(String.format("出栈: %d 和 %d",
 firstExpression.interpret(), secondExpression.interpret()));
 Interpreter operator = OperatorUtil.getExpressionObject(firstExpression, secondExpression, symbol);
 log.info(String.format("应用运算符: %s", operator));
 int result = operator.interpret();
 NumberInterpreter resultExpression = new NumberInterpreter(result);
 stack.push(resultExpression);
 log.info(String.format("阶段结果入栈: %d", resultExpression.interpret()));
 }
 }
 int result = stack.pop().interpret();
 return result;

 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.interpreter;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import org.springframework.expression.Expression;
import org.springframework.expression.spel.standard.SpelExpressionParser;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:48
 */
@Slf4j
class InterpreterTest {

 @Test
 void interpret() {
 String myInterpreter = "6 100 11 + *";
 CustomExpressionParser expressionParser = new CustomExpressionParser();
 int result = expressionParser.parse(myInterpreter);
 log.info("解释器计算结果: " + result);
 }

 @Test
 void springTest() {
 org.springframework.expression.ExpressionParser parser = new SpelExpressionParser();
 Expression expression = parser.parseExpression("100 * 2 + 400 * 1 + 66");
 int result = (Integer) expression.getValue();
 log.info(String.valueOf(result));
 }
}

输出

14:52:28.086 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 6
14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 100
14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 11
14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 出栈: 11 和 100
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 应用运算符: +
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 阶段结果入栈: 111
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 出栈: 111 和 6
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 应用运算符: *
14:52:28.091 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 阶段结果入栈: 666
14:52:28.091 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.InterpreterTest - 解释器计算结果: 666


14:52:28.550 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.InterpreterTest - 666

观察者模式

介绍

定义：定义了对象之间的一对多依赖，让多个观察者对象同时监听某一个主题对象，当主题对象发生变化时，它的所有依赖者（观察者）都会收到通知并更新

类型：行为型

适用场景

关联行为场景，建立一套触发机制

优点

观察者模式支持广播通信
观察者和被观察者之间建立一个抽象的耦合

缺点

使用要得当,要避免循环调用
观察者之间有过多的细节依赖、提高时间消耗及程序复杂度

代码样例——JDK观察者

类图

课程问题

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.AllArgsConstructor;
import lombok.Getter;

/**
 * 课程问题
 *
 * @author F嘉阳
 * @date 2020/3/5 15:23
 */
@Getter
@AllArgsConstructor
public class Question {

 private String username;

 private String content;

}

课程（被观察者）

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;

import java.util.EventListener;
import java.util.Observable;
import java.util.Vector;

/**
 * 课程
 * 被观察者类
 *
 * {@link Observable} 使用 {@link Vector} 保证线程安全
 *
 * JDK源码
 * {@link java.awt.Event} 事件监听器
 * {@link javax.servlet.ServletRequestListener} HTTP请求事件监听器
 * 继承 {@link EventListener}，有大量实现，例如{@link org.springframework.beans.factory.parsing.ReaderEventListener}
 *
 * @author F嘉阳
 * @date 2020/3/5 15:23
 */
@Slf4j
@Getter
@AllArgsConstructor
public class Course extends Observable {

 private String courseName;

 /**
 * 产生问题
 * @param course
 * @param question
 */
 public void produceQuestion(Course course,Question question){
 log.info("{}在{}提交了一个问题",question.getUsername(),course.getCourseName());
 // 状态变化
 setChanged();
 // 通知观察者
 notifyObservers(question);
 }
}

老师实体（观察者）

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;

import java.util.Observable;
import java.util.Observer;

/**
 * 老师实体
 *
 * 老师观察课程是否产生了新的问题
 * @author F嘉阳
 * @date 2020/3/5 15:24
 */
@Slf4j
@Getter
@AllArgsConstructor
public class Teacher implements Observer {

 private String teacherName;

 /**
 * This method is called whenever the observed object is changed. An
 * application calls an <tt>Observable</tt> object's
 * <code>notifyObservers</code> method to have all the object's
 * observers notified of the change.
 *
 * @param o the observable object.
 * @param arg an argument passed to the <code>notifyObservers</code>
 */
 @Override
 public void update(Observable o, Object arg) {
 Course course = (Course) o;
 Question question = (Question) arg;
 log.info("{}老师的{}课程接收到一个{}提交的问题，内容为：{}",
 teacherName,course.getCourseName(),question.getUsername(),question.getContent());
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/5 15:36
 */
@Slf4j
class TeacherTest {

 @Test
 void update() {
 Course course = new Course("Java设计模式");
 Teacher teacher = new Teacher("fjy8018");

 course.addObserver(teacher);

 // 业务逻辑
 Question question = new Question("fjy","问题内容XXX");

 course.produceQuestion(course,question);
 }
}

输出

15:38:34.437 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.Course - fjy在Java设计模式提交了一个问题
15:38:34.444 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.Teacher - fjy8018老师的Java设计模式课程接收到一个fjy提交的问题，内容为：问题内容XXX

代码样例——Guava监听器

package top.fjy8018.designpattern.pattern.behavior.observer.guava;

import com.google.common.eventbus.Subscribe;
import lombok.extern.slf4j.Slf4j;

import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CopyOnWriteArraySet;

/**
 * Guava事件监听器
 * {@link EventBus} 使用内部对象 {@link SubscriberRegistry} 内部使用 {@link ConcurrentMap} 存储订阅者
 * 同时使用 {@link CopyOnWriteArraySet} 保证线程安全和订阅者唯一
 * 故{@link com.google.common.eventbus.Subscriber}
 * 重写了 {@link com.google.common.eventbus.Subscriber#equals(Object)}
 * 和 {@link com.google.common.eventbus.Subscriber#hashCode()} 方法
 *
 * @author F嘉阳
 * @date 2020/3/5 15:56
 */
@Slf4j
public class GuavaEvent {

 @Subscribe
 public void subscribe(String str) {
 // 业务逻辑
 log.info("执行订阅方法，传入参数是：{}", str);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.observer;

import com.google.common.eventbus.EventBus;
import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import top.fjy8018.designpattern.pattern.behavior.observer.guava.Course;
import top.fjy8018.designpattern.pattern.behavior.observer.guava.GuavaEvent;

/**
 * @author F嘉阳
 * @date 2020/3/5 15:58
 */
@Slf4j
class GuavaEventTest {

 @Test
 void subscribe() {
 EventBus eventBus = new EventBus();
 GuavaEvent event = new GuavaEvent();
 // 注册
 eventBus.register(event);
 eventBus.post("提交事件的内容");
 }
}

输出

16:24:27.716 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.GuavaEvent - 执行订阅方法，传入参数是：提交事件的内容

备忘录模式

介绍

定义：保存一个对象的某个状态，以便在适当的时候恢复对象。（类似历史记录存储）

“后悔药”

类型：行为型

适用场景

保存及恢复数据相关业务场景
后悔的时候，即想恢复到之前的状态

优点

为用户提供一种可恢复机制
存档信息的封装

缺点

资源占用

相关设计模式

备忘录模式和状态模式

代码样例

类图

文章

package top.fjy8018.designpattern.pattern.behavior.memento;

import lombok.AllArgsConstructor;
import lombok.Data;

/**
 * 文章，字段不宜过多，否则导致过大内存占用
 *
 * @author F嘉阳
 * @date 2020/3/6 17:00
 */
@Data
@AllArgsConstructor
public class Article {

 private String title;

 private String content;

 private String imgs;

 /**
 * 保存
 * @return
 */
 public ArticleMemento saveToMemento() {
 ArticleMemento articleMemento = new ArticleMemento(this.title,this.content,this.imgs);
 return articleMemento;
 }

 /**
 * 撤销
 * @param articleMemento
 */
 public void undoFromMemento(ArticleMemento articleMemento) {
 this.title = articleMemento.getTitle();
 this.content = articleMemento.getContent();
 this.imgs = articleMemento.getImgs();
 }
}

文章备忘录（快照）

package top.fjy8018.designpattern.pattern.behavior.memento;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.ToString;
import org.springframework.binding.message.StateManageableMessageContext;

/**
 * 文章备忘录（快照）
 * 快照不需要set方法，只能通过构造器注入
 *
 * Spring工作流中源码 {@link StateManageableMessageContext#createMessagesMemento()}
 *
 * @author F嘉阳
 * @date 2020/3/6 17:03
 */
@Getter
@ToString
@AllArgsConstructor
public class ArticleMemento {

 private String title;

 private String content;

 private String imgs;
}

备忘录核心，栈实现管理者

package top.fjy8018.designpattern.pattern.behavior.memento;

import java.util.Stack;

/**
 * 保存一个对象的某个状态，以便在适当的时候恢复对象
 *
 * @author F嘉阳
 * @date 2020/3/6 17:04
 */
public class ArticleMementoManager {

 /**
 * 使用栈特性实现动作撤销
 */
 private final Stack<ArticleMemento> ARTICLE_MEMENTO_STACK = new Stack<ArticleMemento>();

 public ArticleMemento getMemento() {
 ArticleMemento articleMemento= ARTICLE_MEMENTO_STACK.pop();
 return articleMemento;
 }

 public void addMemento(ArticleMemento articleMemento)
 {
 ARTICLE_MEMENTO_STACK.push(articleMemento);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.memento;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/3/6 17:07
 */
@Slf4j
class ArticleMementoTest {

 @Test
 void getTitle() {
 ArticleMementoManager articleMementoManager = new ArticleMementoManager();

 Article article= new Article("如影随行的设计模式A","手记内容A","手记图片A");
 // 存档
 ArticleMemento articleMemento = article.saveToMemento();

 articleMementoManager.addMemento(articleMemento);
 log.info("标题:"+article.getTitle()+" 内容:"+article.getContent()+" 图片:"+article.getImgs()+" 暂存成功");

 log.info("手记完整信息:"+article);
 
 log.info("修改手记start");

 article.setTitle("如影随行的设计模式B");
 article.setContent("手记内容B");
 article.setImgs("手记图片B");

 log.info("修改手记end");

 log.info("手记完整信息:"+article);

 articleMemento = article.saveToMemento();
 articleMementoManager.addMemento(articleMemento);
 
 article.setTitle("如影随行的设计模式C");
 article.setContent("手记内容C");
 article.setImgs("手记图片C");

 log.info("暂存回退start");
 // 取档
 log.info("回退出栈1次");
 articleMemento = articleMementoManager.getMemento();
 article.undoFromMemento(articleMemento);

 log.info("回退出栈2次");
 articleMemento = articleMementoManager.getMemento();
 article.undoFromMemento(articleMemento);

 log.info("暂存回退end");
 log.info("手记完整信息:"+article);
 }
}

输出

17:33:51.154 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 标题:如影随行的设计模式A 内容:手记内容A 图片:手记图片A 暂存成功
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式A, content=手记内容A, imgs=手记图片A)
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 修改手记start
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 修改手记end
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式B, content=手记内容B, imgs=手记图片B)
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 暂存回退start
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 回退出栈1次
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 回退出栈2次
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 暂存回退end
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式A, content=手记内容A, imgs=手记图片A)

命令模式

介绍

定义：将请求封装成对象,以便使用不同的请求

特点：命令模式解决了应用程序中对象的职责以及它们之间的通信方式

类型：行为型

适用场景

请求调用者和请求接收者需要解耦,使得调用者和接收者不直接交互
需要抽象出等待执行的行为

优点

降低耦合
容易扩展新命令或者一组命令

缺点

命令的无限扩展会增加类的数量，提高系统实现复杂度

相关设计模式

命令模式和备忘录模式

代码样例

类图

命令抽象

package top.fjy8018.designpattern.pattern.behavior.command;

/**
 * 命令抽象
 * <p>
 * Java相关源码 {@link Runnable} 可理解为命令抽象
 * 其实现类为具体的命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
public interface Command {
 void execute();
}

操作视频

package top.fjy8018.designpattern.pattern.behavior.command;

import lombok.AllArgsConstructor;

/**
 * 关闭视频命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
@AllArgsConstructor
public class CloseCourseVideoCommand implements Command {
 private final CourseVideo courseVideo;

 @Override
 public void execute() {
 courseVideo.close();
 }
}

打开视频命令

package top.fjy8018.designpattern.pattern.behavior.command;

import lombok.AllArgsConstructor;

/**
 * 打开视频命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
@AllArgsConstructor
public class OpenCourseVideoCommand implements Command {
 private final CourseVideo courseVideo;

 @Override
 public void execute() {
 courseVideo.open();
 }
}

关闭视频命令

package top.fjy8018.designpattern.pattern.behavior.command;

import lombok.AllArgsConstructor;

/**
 * 关闭视频命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
@AllArgsConstructor
public class CloseCourseVideoCommand implements Command {
 private final CourseVideo courseVideo;

 @Override
 public void execute() {
 courseVideo.close();
 }
}

命令执行人

package top.fjy8018.designpattern.pattern.behavior.command;

import java.util.ArrayList;
import java.util.List;

/**
 * 命令执行人
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
public class Staff {
 /**
 * 保证执行顺序
 */
 private final List<Command> commandList = new ArrayList<Command>();

 public void addCommand(Command command) {
 commandList.add(command);
 }

 public void executeCommands() {
 for (Command command : commandList) {
 command.execute();
 }
 commandList.clear();
 }

}

测试

package top.fjy8018.designpattern.pattern.behavior.command;

import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/7/21 16:02
 */
class CommandTest {

 @Test
 void doTest() {
 CourseVideo courseVideo = new CourseVideo("Java设计模式精讲");
 OpenCourseVideoCommand openCourseVideoCommand = new OpenCourseVideoCommand(courseVideo);
 CloseCourseVideoCommand closeCourseVideoCommand = new CloseCourseVideoCommand(courseVideo);

 Staff staff = new Staff();

 staff.addCommand(openCourseVideoCommand);
 staff.addCommand(closeCourseVideoCommand);

 staff.executeCommands();
 }
}

输出

Java设计模式精讲课程视频开放
Java设计模式精讲课程视频关闭

中介者模式

介绍

定义：定义一个封装一组对象如何交互的对象

特点：通过使对象明确地相互引用来促进松散耦合,并允许独立地改变它们的交互

适用场景

系统中对象之间存在复杂的引用关系，产生的相互依赖关系结构混乱且难以理解
交互的公共行为，如果需要改变行为则可以增加新的中介者类

优点

将一对多转化成了一对一、降低程序复杂度
类之间解耦

缺点

中介者过多,导致系统复杂

相关设计模式

中介者模式和观察者模式

代码样例

类图

学习群组（中介者）

package top.fjy8018.designpattern.pattern.behavior.mediator;

import java.util.Date;
import java.util.Timer;

/**
 * 学习群组（中介者）
 * <p>
 * jdk源码 {@link Timer#schedule(java.util.TimerTask, Date)} 多个方法重载
 * 都是调用{@link Timer#sched(java.util.TimerTask, long, long)} 实现的，由底层方法进行整体协调
 *
 * @author F嘉阳
 * @date 2020/7/21 16:20
 */
public class StudyGroup {

 public static void showMessage(User user, String message) {
 System.out.println(new Date().toString() + " [" + user.getName() + "] : " + message);
 }
}

用户

package top.fjy8018.designpattern.pattern.behavior.mediator;

import lombok.AllArgsConstructor;
import lombok.Data;

/**
 * @author F嘉阳
 * @date 2020/7/21 16:20
 */
@Data
@AllArgsConstructor
public class User {
 private String name;

 public void sendMessage(String message) {
 // 只和中介者交互
 StudyGroup.showMessage(this, message);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.mediator;

import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/7/21 16:25
 */
class MediatorTest {

 @Test
 void doTest() {
 User fjy = new User("F嘉阳");
 User tom = new User("Tom");

 fjy.sendMessage(" Hey! Tom! Let's learn Design Pattern");
 tom.sendMessage("OK! F嘉阳");
 }
}

输出

Tue Jul 21 16:47:03 CST 2020 [F嘉阳] : Hey! Tom! Let's learn Design Pattern
Tue Jul 21 16:47:03 CST 2020 [Tom] : OK! F嘉阳

责任链模式

介绍

定义：为请求创建一个接收此次请求对象的链

类型：行为型

适用场景

一个请求的处理需要多个对象当中的一个或几个协作处理

优点

请求的发送者和接收者(请求的处理)解耦
责任链可以动态组合

缺点

责任链太长或者处理时间过长,影响性能
责任链有可能过多

相关设计模式

责任链模式和状态模式

代码样例

类图

模拟二级审批场景

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import javax.servlet.FilterChain;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;

/**
 * 模拟二级审批场景
 * <p>
 * 审批抽象
 * <p>
 * java 过滤器实现 {@link javax.servlet.Filter#doFilter(ServletRequest, ServletResponse, FilterChain)}
 * 实现 {@link FilterChain#doFilter(ServletRequest, ServletResponse)} 接口的组成一条过滤器链
 * 所有子类通过调用 chain.doFilter(request, response) 实现链式传递
 * <p>
 * SpringSecurity过滤器链实现也类似
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
public abstract class Approver {
 /**
 * 责任链核心：包含与自身相同的类型对象
 */
 protected Approver approver;

 /**
 * 设置下一个节点，组成链条
 *
 * @param approver
 */
 public void setNextApprover(Approver approver) {
 this.approver = approver;
 }

 /**
 * 交由子类实现
 *
 * @param course
 */
 public abstract void deploy(Course course);
}

手记审批

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import org.apache.commons.lang3.StringUtils;

/**
 * 手记审批
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
public class ArticleApprover extends Approver {
 @Override
 public void deploy(Course course) {
 if (StringUtils.isNotEmpty(course.getArticle())) {
 System.out.println(course.getName() + "含有手记,批准");
 // 是否为末尾
 if (approver != null) {
 // 执行下一个处理链
 approver.deploy(course);
 }
 } else {
 System.out.println(course.getName() + "不含有手记,不批准,流程结束");
 return;
 }
 }
}

视频审批

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import org.apache.commons.lang3.StringUtils;

/**
 * 视频审批
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
public class VideoApprover extends Approver {
 @Override
 public void deploy(Course course) {
 if (StringUtils.isNotEmpty(course.getVideo())) {
 System.out.println(course.getName() + "含有视频,批准");
 if (approver != null) {
 approver.deploy(course);
 }
 } else {
 System.out.println(course.getName() + "不含有视频,不批准,流程结束");
 return;
 }
 }
}

课程

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import lombok.Data;

/**
 * 课程
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
@Data
public class Course {
 private String name;

 private String article;

 private String video;
}

测试

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/7/21 15:06
 */
class ChainofResponsibilityTest {

 @Test
 void doTest() {
 Approver articleApprover = new ArticleApprover();
 Approver videoApprover = new VideoApprover();

 Course course = new Course();
 course.setName("Java设计模式");
 course.setArticle("Java设计模式的手记");
 course.setVideo("Java设计模式的视频");

 articleApprover.setNextApprover(videoApprover);

 articleApprover.deploy(course);
 }
}

输出

Java设计模式含有手记,批准
Java设计模式含有视频,批准

访问者模式

介绍

定义：封装作用于某数据结构(如List/set/Map等)中的各元素的操作

特性：可以在不改变各元素的类的前提下定义作用于这些元素的操作

类型：行为型

适用场景

一个数据结构如(List/set/Map等)包含很多类型对象
数据结构与数据操作分离

优点

增加新的操作很容易即增加一个新的访问者

缺点

增加新的数据结构困难
具体元素变更比较麻烦

相关设计模式

访问者模式和迭代器模式

代码样例

类图

课程抽象

package top.fjy8018.designpattern.pattern.behavior.visitor;

import lombok.Data;

/**
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
@Data
public abstract class Course {
 private String name;

 public abstract void accept(IVisitor visitor);

}

实战课

package top.fjy8018.designpattern.pattern.behavior.visitor;

import lombok.Data;
import lombok.EqualsAndHashCode;

/**
 * 实战课，包含价格
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
@Data
@EqualsAndHashCode(callSuper = true)
public class CodingCourse extends Course {
 private int price;

 @Override
 public void accept(IVisitor visitor) {
 visitor.visit(this);
 }
}

免费课

package top.fjy8018.designpattern.pattern.behavior.visitor;

/**
 * 免费课
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
public class FreeCourse extends Course {

 /**
 * 接受策略
 *
 * @param visitor
 */
 @Override
 public void accept(IVisitor visitor) {
 visitor.visit(this);
 }
}

访问者

package top.fjy8018.designpattern.pattern.behavior.visitor;

/**
 * 封装操作数据结构的方法
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
public interface IVisitor {

 void visit(FreeCourse freeCourse);

 void visit(CodingCourse codingCourse);
}

访问者核心实现

package top.fjy8018.designpattern.pattern.behavior.visitor;

/**
 * 访问者
 * 核心：对不同类型产生不同的操作行为
 * <p>
 * jdk源码 {@link java.nio.file.FileVisitor} 实现对文件的访问
 * spring源码 {@link org.springframework.beans.factory.config.BeanDefinitionVisitor} 遍历Bean具体属性并填充
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
public class Visitor implements IVisitor {

 /**
 * 访问免费课程，打印所有免费课程名称
 *
 * @param freeCourse
 */
 @Override
 public void visit(FreeCourse freeCourse) {
 System.out.println("免费课程:" + freeCourse.getName());
 }

 /**
 * 访问实战课程，打印所有实战课程名称及价格
 *
 * @param codingCourse
 */
 @Override
 public void visit(CodingCourse codingCourse) {
 System.out.println("实战课程:" + codingCourse.getName() + " 价格:" + codingCourse.getPrice() + "元");
 }

}

测试

package top.fjy8018.designpattern.pattern.behavior.visitor;

import org.junit.jupiter.api.Test;

import java.util.ArrayList;
import java.util.List;

/**
 * @author F嘉阳
 * @date 2020/7/22 16:19
 */
class VisitorTest {

 @Test
 void visit() {
 List<Course> courseList = new ArrayList<>();

 FreeCourse freeCourse = new FreeCourse();
 freeCourse.setName("SpringMVC数据绑定");

 CodingCourse codingCourse = new CodingCourse();
 codingCourse.setName("Java设计模式精讲");
 codingCourse.setPrice(299);
 // 数组组装
 courseList.add(freeCourse);
 courseList.add(codingCourse);

 for (Course course : courseList) {
 course.accept(new Visitor());
 }
 }
}

输出

免费课程:SpringMVC数据绑定
实战课程:Java设计模式精讲 价格:299元

状态模式

介绍

定义：允许一个对象在其内部状态改变时改变它的行为

类型：行为型

适用场景

一个对象存在多个状态(不同状态下行为不同),且状态可相互转换

优点

将不同的状态隔离
把各种状态的转换逻辑,分布到 State的子类中,减少相互间依赖
增加新的状态非常简单

缺点

状态多的业务场景导致类数目增加,系统变复杂

相关设计模式

状态模式和享元模式

代码样例

类图

视频播放状态转换场景

package top.fjy8018.designpattern.pattern.behavior.state;

import javax.faces.lifecycle.Lifecycle;

/**
 * 视频播放状态转换场景
 * <p>
 * {@link Lifecycle#execute(javax.faces.context.FacesContext)} 该方法通过外部状态控制改变行为
 * {@link javax.faces.webapp.FacesServlet} 状态控制
 *
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public abstract class CourseVideoState {
 /**
 * 视频上下文
 */
 protected CourseVideoContext courseVideoContext;

 public void setCourseVideoContext(CourseVideoContext courseVideoContext) {
 this.courseVideoContext = courseVideoContext;
 }

 public abstract void play();

 public abstract void speed();

 public abstract void pause();

 public abstract void stop();

}

视频上下文

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * 视频上下文
 *
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class CourseVideoContext {

 /**
 * 存储当前状态
 */
 private CourseVideoState courseVideoState;

 /**
 * 结合享元模式设置为内存共享
 */
 public final static PlayState PLAY_STATE = new PlayState();
 public final static StopState STOP_STATE = new StopState();
 public final static PauseState PAUSE_STATE = new PauseState();
 public final static SpeedState SPEED_STATE = new SpeedState();

 public CourseVideoState getCourseVideoState() {
 return courseVideoState;
 }

 public void setCourseVideoState(CourseVideoState courseVideoState) {
 this.courseVideoState = courseVideoState;
 // 把自己本身设置到实例上下文中
 this.courseVideoState.setCourseVideoContext(this);
 }
 public void play(){
 this.courseVideoState.play();
 }

 public void speed(){
 this.courseVideoState.speed();
 }

 public void stop(){
 this.courseVideoState.stop();
 }

 public void pause(){
 this.courseVideoState.pause();
 }
}

各状态实现

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class PlayState extends CourseVideoState {

 @Override
 public void play() {
 System.out.println("正常播放课程视频状态");
 }

 @Override
 public void speed() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.SPEED_STATE);
 }

 @Override
 public void pause() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PAUSE_STATE);
 }

 @Override
 public void stop() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE);
 }
}

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class PauseState extends CourseVideoState {

 @Override
 public void play() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE);
 }

 @Override
 public void speed() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.SPEED_STATE);
 }

 @Override
 public void pause() {
 System.out.println("暂停播放课程视频状态");
 }

 @Override
 public void stop() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE);
 }
}

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class SpeedState extends CourseVideoState {
 @Override
 public void play() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE);
 }

 @Override
 public void speed() {
 System.out.println("快进播放课程视频状态");
 }

 @Override
 public void pause() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PAUSE_STATE);
 }

 @Override
 public void stop() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE);
 }
}

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class StopState extends CourseVideoState {
 @Override
 public void play() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE);
 }

 @Override
 public void speed() {
 System.out.println("ERROR 停止状态不能快进!!");
 }

 @Override
 public void pause() {
 System.out.println("ERROR 停止状态不能暂停!!");
 }

 @Override
 public void stop() {
 System.out.println("停止播放课程视频状态");
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.state;

import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/7/22 16:19
 */
class PlayStateTest {

 @Test
 void play() {
 CourseVideoContext courseVideoContext = new CourseVideoContext();
 courseVideoContext.setCourseVideoState(new PlayState());

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());
 courseVideoContext.pause();

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());

 courseVideoContext.speed();

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());

 courseVideoContext.stop();

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());

 courseVideoContext.speed();
 }
}

输出

当前状态:PlayState
当前状态:PauseState
当前状态:SpeedState
当前状态:StopState
ERROR 停止状态不能快进!!