SpringCloud on F嘉阳

SpringCloudStream函数式编程模型

Sun, 21 Feb 2021 12:22:00 +0800

背景

SpringCloud 3.1版本之后，@EnableBinding、@Output等StreamApi注解都标记为废弃

/**
 * Indicates that an output binding target will be created by the framework.
 *
 * @author Dave Syer
 * @author Marius Bogoevici
 * @author Artem Bilan
 *
 * @deprecated as of 3.1 in favor of functional programming model
 */

@Qualifier
@Target({ ElementType.FIELD, ElementType.METHOD, ElementType.ANNOTATION_TYPE,
		ElementType.PARAMETER })
@Retention(RetentionPolicy.RUNTIME)
@Inherited
@Documented
@Deprecated
public @interface Output {

	/**
	 * Specify the binding target name; used as a bean name for binding target and as a
	 * destination name by default.
	 * @return the binding target name
	 */
	String value() default "";

}

/**
 * Enables the binding of targets annotated with {@link Input} and {@link Output} to a
 * broker, according to the list of interfaces passed as value to the annotation.
 *
 * @author Dave Syer
 * @author Marius Bogoevici
 * @author David Turanski
 * @author Soby Chacko
 *
 * @deprecated as of 3.1 in favor of functional programming model
 */
@Target({ ElementType.TYPE, ElementType.ANNOTATION_TYPE })
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@Configuration
@Import({ BindingBeansRegistrar.class, BinderFactoryAutoConfiguration.class })
@EnableIntegration
@Deprecated
public @interface EnableBinding {

	/**
	 * A list of interfaces having methods annotated with {@link Input} and/or
	 * {@link Output} to indicate binding targets.
	 * @return list of interfaces
	 */
	Class<?>[] value() default {};

}

Producing and Consuming Messages — SpringCloud 官方文档

@EnableBinding @deprecated as of 3.1 in favor of functional programming model — stackoverflow

The type EnableBinding is deprecated — stackoverflow

因此，所有采用命令式编程的API，在升级到新版本时，官方推荐使用更加简单的函数式编程API

完整代码样例

FJiayang/spring-cloud-stream-rabbit-example — GitHub项目地址

升级方式

命令式编程模型

以简单多Topic消息为例，原本的编程方式如下

生产者

@SpringBootApplication
public class ProducerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ProducerApplication.class, args);
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 17:57
 */
@RestController
@EnableBinding(MySource.class)
public class Producer {

 @Autowired
 private MySource channel;

 @RequestMapping("/send")
 public String send() {
 channel.output().send(MessageBuilder.withPayload(new Date()).build());
 return "success";
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 18:01
 */
public interface MySource {
 String OUTPUT = "output";

 @Output(MySource.OUTPUT)
 MessageChannel output();
}

配置文件

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin
 cloud:
 stream:
 bindings:
 output:
 destination: my-test-channel
server:
 port: 8082

消费者

@SpringBootApplication
public class ConsumerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ConsumerApplication.class, args);
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 18:10
 */
@EnableBinding(MySink.class)
public class Consumer {
 @StreamListener(MySink.INPUT)
 public void receive(Message<String> message) {
 System.out.println("接收到MQ消息:" + message.getPayload());
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 18:07
 */
public interface MySink {
 String INPUT = "input";

 @Input(MySink.INPUT)
 SubscribableChannel input();
}

配置文件

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin
 cloud:
 stream:
 bindings:
 input:
 destination: my-test-channel
server:
 port: 8081

函数式编程模型

从命令式升级到函数式编程，首先需要升级SpringCloud版本至3.1以上，此处升级到最新版2020.0.1

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>

 <groupId>top.fjy8018</groupId>
 <artifactId>cloud-stream</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>pom</packaging>

 <name>cloud-stream</name>
 <description>Demo project for Spring Boot</description>

 <modules>
 <module>producer</module>
 <module>consumer</module>
 </modules>

 <parent>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-parent</artifactId>
 <version>2.4.2</version>
 <relativePath/> <!-- lookup parent from repository -->
 </parent>

 <properties>
 <spring-cloud.version>2020.0.1</spring-cloud.version>
 </properties>

 <dependencyManagement>
 <dependencies>
 <dependency>
 <groupId>org.springframework.cloud</groupId>
 <artifactId>spring-cloud-dependencies</artifactId>
 <version>${spring-cloud.version}</version>
 <type>pom</type>
 <scope>import</scope>
 </dependency>
 </dependencies>
 </dependencyManagement>
</project>

目前国内有清晰的升级指引的还比较少，目前只能参考官方文档材料

Programming Model — SpringCloud官方文档

模型1——系统定时驱动的消息队列

官方文档描述

Suppliers (Sources)
Function and Consumer are pretty straightforward when it comes to how their invocation is triggered. They are triggered based on data (events) sent to the destination they are bound to. In other words, they are classic event-driven components.

However, Supplier is in its own category when it comes to triggering. Since it is, by definition, the source (the origin) of the data, it does not subscribe to any in-bound destination and, therefore, has to be triggered by some other mechanism(s). There is also a question of Supplier implementation, which could be imperative or reactive and which directly relates to the triggering of such suppliers.

Consider the following sample:
@SpringBootApplication
public static class SupplierConfiguration {

	@Bean
	public Supplier<String> stringSupplier() {
		return () -> "Hello from Supplier";
	}
}
The preceding Supplier bean produces a string whenever its get() method is invoked. However, who invokes this method and how often? The framework provides a default polling mechanism (answering the question of “Who?”) that will trigger the invocation of the supplier and by default it will do so every second (answering the question of “How often?”). In other words, the above configuration produces a single message every second and each message is sent to an output destination that is exposed by the binder.

其大意为，若采用默认的函数式编程模型，会每秒钟发送一条消息到消费者，定时发送参数可以配置

按照文档升级后如下

生产者

@SpringBootApplication
public class ProducerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ProducerApplication.class, args);
 }

 @Bean
 public Supplier<Date> source1() {
 return () -> new Date();
 }
}

函数式编程不需要在代码中定义监听队列，只需要按照约定写在配置文件中即可

官方文档描述

Functional binding names
Unlike the explicit naming required by annotation-based support (legacy) used in the previous versions of spring-cloud-stream, the functional programming model defaults to a simple convention when it comes to binding names, thus greatly simplifying application configuration. Let’s look at the first example:
@SpringBootApplication
public class SampleApplication {

	@Bean
	public Function<String, String> uppercase() {
	 return value -> value.toUpperCase();
	}
}
In the preceding example we have an application with a single function which acts as message handler. As a Function it has an input and output. The naming convention used to name input and output bindings is as follows:

input - <functionName> + -in- + <index>

output - <functionName> + -out- + <index>

The in and out corresponds to the type of binding (such as input or output). The index is the index of the input or output binding. It is always 0 for typical single input/output function, so it’s only relevant for Functions with multiple input and output arguments.

So if for example you would want to map the input of this function to a remote destination (e.g., topic, queue etc) called “my-topic” you would do so with the following property:
--spring.cloud.stream.bindings.uppercase-in-0.destination=my-topic
Note how uppercase-in-0 is used as a segment in property name. The same goes for uppercase-out-0.

配置文件

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin

 cloud:
 stream:
 bindings:
 source1-out-0:
 destination: test1
 function:
 definition: source1

server:
 port: 8083

由此可见，整体配置简化了很多，多为规约大于配置的方式进行定义，特别注意function配置的位置，是在stream配置的同级

消费者

@SpringBootApplication
public class ConsumerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ConsumerApplication.class, args);
 }

 @Bean
 public Consumer<Date> sink1() {
 return System.out::println;
 }
}

配置文件同理

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin

 cloud:
 stream:
 bindings:
 sink1-in-0:
 destination: test1
 function:
 definition: sink1


server:
 port: 8081

运行结果

生产者每隔一秒发送一次消息

模型2——`StreamBridge`

但实际生产中，更多是由业务场景触发，故不能使用该模式进行配置，SpringCloud还提供了另外一种StreamBridge模式

官方文档介绍

Sending arbitrary data to an output (e.g. Foreign event-driven sources)
There are cases where the actual source of data may be coming from the external (foreign) system that is not a binder. For example, the source of the data may be a classic REST endpoint. How do we bridge such source with the functional mechanism used by spring-cloud-stream?

Spring Cloud Stream provides two mechanisms, so let’s look at them in more details

Here, for both samples we’ll use a standard MVC endpoint method called delegateToSupplier bound to the root web context, delegating incoming requests to stream via two different mechanisms - imperative (via StreamBridge) and reactive (via EmitterProcessor).

大意是，实际的数据源可以是外部事件驱动，例如通过Rest接口

官方样例

Using StreamBridge
@SpringBootApplication
@Controller
public class WebSourceApplication {

	public static void main(String[] args) {
		SpringApplication.run(WebSourceApplication.class, "--spring.cloud.stream.source=toStream");
	}

	@Autowired
	private StreamBridge streamBridge;

	@RequestMapping
	@ResponseStatus(HttpStatus.ACCEPTED)
	public void delegateToSupplier(@RequestBody String body) {
		System.out.println("Sending " + body);
		streamBridge.send("toStream-out-0", body);
	}
}
Here we autowire a StreamBridge bean which allows us to send data to an output binding effectively bridging non-stream application with spring-cloud-stream. Note that preceding example does not have any source functions defined (e.g., Supplier bean) leaving the framework with no trigger to create source bindings, which would be typical for cases where configuration contains function beans. So to trigger the creation of source binding we use spring.cloud.stream.source property where you can declare the name of your sources. The provided name will be used as a trigger to create a source binding. So in the preceding example the name of the output binding will be toStream-out-0 which is consistent with the binding naming convention used by functions (see Binding and Binding names). You can use ; to signify multiple sources (e.g., --spring.cloud.stream.source=foo;bar)

参考官方文档改造后

生产者

这里直接贴出多主题的消费模式，实际生产中多数也是订阅多个主题

@SpringBootApplication
public class ProducerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ProducerApplication.class, args);
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 17:57
 */
@RestController
public class Producer {

 @Autowired
 private StreamBridge streamBridge;

 @RequestMapping("/send1")
 public String send1() {
 streamBridge.send("source1-out-0", new Date());
 return "success1";
 }

 @RequestMapping("/send2")
 public String send2() {
 streamBridge.send("source2-out-0", LocalDateTime.now().format(DateTimeFormatter.ISO_LOCAL_DATE_TIME));
 return "success2";
 }

}

配置文件

spring:
 rabbitmq:
 host: 192.168.133.128
 username: dev-user
 password: devpassword

 cloud:
 stream:
 bindings:
 source1-out-0:
 destination: test2
 source2-out-0:
 destination: test3
 function:
 definition: source1;source2


server:
 port: 8083

消费者

@SpringBootApplication
public class ConsumerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ConsumerApplication.class, args);
 }

 @Bean
 public Consumer<Date> sink1() {
 return System.out::println;
 }

 @Bean
 public Consumer<String> sink2() {
 return System.out::println;
 }
}

配置文件

spring:
 rabbitmq:
 host: 192.168.133.128
 username: dev-user
 password: devpassword

 cloud:
 stream:
 bindings:
 sink1-in-0:
 destination: test2
 sink2-in-0:
 destination: test3
 function:
 definition: sink1;sink2

server:
 port: 8081

运行结果

触发/send1

触发/send2

可见消息被正确的路由

总结

整体来看，函数式编程比命令式编程更加简洁，而且加上SpringCloud规约大于配置的特点，可以明显减少配置的代码量，是未来发展的趋势。

微服务平台开发总结（3）——第二代密码管理平台

Thu, 07 May 2020 21:38:00 +0800

背景

吸取前两代系统的失败经验，对第二代密码管理微服务进行重新规划，并且在探索过程中见识到了cloud-platform、SpringBlade、pigx等优秀的微服务项目，考虑基于开源项目二次开发

规划

为了尽快落地上线，第一期以实现最小功能为主，其他非必须功能尽可能往后放

第一期

SpringCloud微服务集成
Nacos、Sentinel、graylog、zipkin、 Turbine集成
前后端密码管理功能对接
操作日志记录
邮件通知
Excel导入导出
v1迁移到v2工具包

第二期

dble读写分离
支持忘记密码、密码锁定等
集成微信推送、微信注册功能
支持上传前端日志
页面排序、检索优化

第三期

增加用户反馈功能
接入Seata分布式事务
支持配置导入导出权限
支持录入标签
…

技术体系

前端

前后端分离-采用前后端分离模式，前端基于Vue
采用Vuex进行状态管理，采用ES6作为语法支持
前后端基于开源项目SpringBlade二次开发
前后端代码生成-定制针对两套前端与后端的代码生成模板，轻松生成整个模块的前后端代码，减少重复工作量
多租户顶部菜单配置-提供给每个租户独立的顶部菜单配置模块，可以自定义顶部菜单切换

后端

基于SpringCloud的分布式架构
采用Mybatis与MybatisPlus作为ORM持久层框架
多租户系统-完整的SaaS多租户架构
借鉴OAuth2，通过Jwt令牌实现了多终端认证系统，可控制子系统的token权限互相隔离
权限管理-精心设计的权限管理方案，角色权限精确到按钮
集成邮件服务，通过事件传播实现不依赖消息队列通知发送服务
集成微信推送服务，实现微信注册、敏感操作消息推送
动态聚合文档-实现基于Nacos的Swagger SpringCloud聚合文档
实现动态网关，不重启就能设置网关配置项
集成Turbine聚合熔断监控

数据架构

采用MySQL进行数据存储，底层配置为一主两从的高可用集群
采用ElasticSearch三节点集群进行日志数据存储

分布式支持

Nacos-集成阿里巴巴的Nacos完成统一的服务注册与配置
Sentinel-集成Sentinel从流量控制、熔断降级、系统负载等多个维度保护服务的稳定性
采用Docker打包，部署于Kubernetes集群上
Seata分布式事务-定制集成Seata，支持分布式事务，无代码侵入，不失灵活与简洁
使用graylog进行分布式日志采集，集中监控
通过zipkin进行分布式链路追踪，耗时统计和服务调用情况

DevOps

配置管理采用单仓库模式，集成docker打包插件，自动推送到harbor仓库
采用GitLab Runner CICD流水线自动部署到测试环境和生产环境
结合Maven私服和Harbor私服为内网自动化部署提供支持

官方架构图

截图

管理员

操作日志记录

错误日志系统内采集

Turbine集成熔断监控

通知发布

普通用户

主题切换

微信扫码注册

补全信息

主页资源权限控制

菜单资源权限控制

Excel导入导出

密码管理

运维

Nacos注册中心

Sentinel限流管控

SpringBoot Admin应用内指标监控

Harbor内网私服

Nexus内网私服，提供maven和npm包

graylog日志采集

三节点ES集群支持日志存储

耗时统计

新UI风格

动态服务依赖关系图

新UI依赖图

Prometheus指标采集

Grafana资源指标监控

Kuberneties服务部署监控

开发过程工具

Gitlab CICD流水线

看板

功能开发周期分析

gitlab集成k8s部署环境

Sonarqube代码质量检查

分支图

工程模块图

结果——成功

经过两个月的开发和调试，第一期工程符合预定计划，按期投产上线运行，并经过2周试运行和测试后，进入生产环境，迁移老版本数据，正式替换了运行一年半的第一代密码管理系统

时间线

2020年3月开始基于SpringBlade 2.6.2编码
2020年4月20日第一个测试版本进入k8s测试环境
2020年4月28日第一个稳定版进行试运行阶段
2020年5月4日试运行阶段结束，第一个稳定版1.1.2发布，正式进行数据迁移并投入使用

SpringCloud中遇到的坑

Tue, 24 Jul 2018 21:34:00 +0800

SpringCloud Config

配置无法读取

问题描述

配置中心的配置文件如下

eureka:
 client:
 service-url:
 defaultZone: http://localhost:8761/eureka/
spring:
 application:
 name: config
 cloud:
 config:
 server:
 git:
 uri: http://XXX/fjy8018/config-repo
 username: 1186032234@qq.com
 password: XXX
 basedir: spring-cloud\config\basedir
server:
 port: 8084

已经通过网页测试该账号和密码可以访问该仓库，并有权限，启动配置中心后通过页面访问提示Cannot clone or checkout repository

解决方法

SpringCloud正式版的配置中心路径必须是带.git结尾的全路径，将配置改为全路径即可

eureka:
 client:
 service-url:
 defaultZone: http://localhost:8761/eureka/
spring:
 application:
 name: config
 cloud:
 config:
 server:
 git:
 uri: http://XXX/fjy8018/config-repo.git
 username: 1186032234@qq.com
 password: XXX
 basedir: spring-cloud\config\basedir
server:
server:
 port: 8084

SpringCloud on F嘉阳

SpringCloudStream函数式编程模型

背景

升级方式

命令式编程模型

生产者

消费者

函数式编程模型

模型1——系统定时驱动的消息队列

Suppliers (Sources)

生产者

Functional binding names

消费者

运行结果

模型2——StreamBridge

Sending arbitrary data to an output (e.g. Foreign event-driven sources)

Using StreamBridge

生产者

消费者

运行结果

总结

微服务平台开发总结（3）——第二代密码管理平台

背景

规划

第一期

第二期

第三期

技术体系

前端

后端

数据架构

分布式支持

DevOps

官方架构图

截图

管理员

普通用户

运维

开发过程工具

结果——成功

时间线

SpringCloud中遇到的坑

SpringCloud Config

配置无法读取

问题描述

解决方法

模型2——`StreamBridge`