Posts on F嘉阳

ArgoCD落地GitOps

Sat, 23 Aug 2025 22:49:50 +0800

背景

在过去的DevOps实践中，跨环境的基础设施部署和迁移经常存在遗漏，导致环境差异，部署一个新环境往往需要几天。传统CICD流水线采用“推”模式，将变更推送到不同的环境，推送后也不对目标环境状态进行管理和监控，容易导致推送后部署失败。随着云原生容器化已经成为行业事实标准，GitOps成为一种更可靠的应用部署和基础设置运维的范式。

技术选型

ArgoCD遵循GitOps的核心原则，已经成为落地GitOps的默认技术选型，因此此处不探讨其他选型方案。

原理

Git 作为唯一事实来源（GitOps思想）
自动化和持续同步：Argo CD会持续监控 Git 仓库的状态，并自动将集群的实际状态与 Git 中声明的目标状态进行同步。
- 利用K8s的Kustomize能力完成对集群apply操作

实践

环境

安装ArgoCD

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

检查安装结果

查看argo cd登录密码

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

设计代码库结构

落地ArgoCD通常有以下几种仓库设计范式

单仓库模式
多仓库模式
App of Apps模式

本次采用App of Apps的单仓库模式，仓库结构如下

.
├── apps
│ ├── app-one
│ ├── app-two
├── cluster-infra
│ ├── argocd
│ │ ├── base
│ │ └── overlays
│ ├── ingress
│ │ ├── base
│ │ └── overlays
│ │ └── dev
│ └── prometheus-stack
│ ├── base
│ └── overlays
│ └── dev
└── environments
 └── dev
 └── root.yaml

该结构特点如下

基础设施和应用部署分离（apps和cluster-infra）
环境基础配置和环境定制配置分离（base和overlays）
环境根节点配置（environments控制git仓库配置、应用apps注册）

本次将该代码库存放到阿里云代码仓库中。

环境配置迁移

目前的k3s环境中已经手动运维安装了argocd，现在需要将已安装的配置迁移到git库中

将官方安装配置https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml写入到cluster-infra/argocd/base/argocd.yaml中

配置``cluster-infra/argocd/base/kustomization.yaml`文件，注册同目录的安装文件

resources:
 - argocd.yaml

本次依赖阿里云代码仓库，通过ssh连接，需要覆盖默认的argocd配置，编写cluster-infra/argocd/overlays/with-known-hosts/ssh-known-hosts.patch.yaml

apiVersion: v1 
kind: ConfigMap 
metadata: 
 name: argocd-ssh-known-hosts-cm 
data: 
 ssh_known_hosts: |- 
 # BEGIN >>> 自定义条目（全局生效） 
 codeup.aliyun.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6WbValczSDKC4ktjRzDNC1qINDAViw/oGA8ud7qH5F9XFZ4QAWhZmFBsp+cppBIOM63exjc4O9vbzMrd7AUItXigymGfTqrGt5VykIiaNhjttxSwbZf3N9eDcAvX4gGm3fcm6D1ACvPxvxMsCwOpdz5tDHqDxGOLPAXN+FSAvjdcN4uMz0MXghGh9Q3zSpy8z1TLkAKT2QBnr4q2tsdZKwWON8bBHIPwP3RSAdSUL6bAyotyncMDAvG53kwRKJ3wJ5ZezXqZNNV7EzHds+pvwThtuWtWsBuuRpL/oZiviWlXy6hILoWbkRfgONPwT0faaOn1GLzjHLuqHHfALGOen 
 # END <<< 自定义条目 
 
 # This file was automatically generated by hack/update-ssh-known-hosts.sh. DO NOT EDIT 
 [ssh.github.com]:443 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg= 
 [ssh.github.com]:443 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl 
 [ssh.github.com]:443 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk= 
 bitbucket.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPIQmuzMBuKdWeF4+a2sjSSpBK0iqitSQ+5BM9KhpexuGt20JpTVM7u5BDZngncgrqDMbWdxMWWOGtZ9UgbqgZE= 
 bitbucket.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO 
 bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQeJzhupRu0u0cdegZIa8e86EG2qOCsIsD1Xw0xSeiPDlCr7kq97NLmMbpKTX6Esc30NuoqEEHCuc7yWtwp8dI76EEEB1VqY9QJq6vk+aySyboD5QF61I/1WeTwu+deCbgKMGbUijeXhtfbxSxm6JwGrXrhBdofTsbKRUsrN1WoNgUa8uqN1Vx6WAJw1JHPhglEGGHea6QICwJOAr/6mrui/oB7pkaWKHj3z7d1IC4KWLtY47elvjbaTlkN04Kc/5LFEirorGYVbt15kAUlqGM65pk6ZBxtaO3+30LVlORZkxOh+LKL/BvbZ/iRNhItLqNyieoQj/uh/7Iv4uyH/cV/0b4WDSd3DptigWq84lJubb9t/DnZlrJazxyDCulTmKdOR7vs9gMTo+uoIrPSb8ScTtvw65+odKAlBj59dhnVp9zd7QUojOpXlL62Aw56U4oO+FALuevvMjiWeavKhJqlR7i5n9srYcrNV7ttmDw7kf/97P5zauIhxcjX+xHv4M= 
 github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg= 
 github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl 
 github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk= 
 gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY= 
 gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf 
 gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9 
 ssh.dev.azure.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H 
 vs-ssh.visualstudio.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

配置``cluster-infra/argocd/overlays/with-known-hosts/kustomization.yaml`文件，注册hosts patch文件

resources: 
 - ../../base 
 
patchesStrategicMerge: 
 - ssh-known-hosts.patch.yaml

按这样配置后，argocd会先执行base的安装文件apply基础环境，再使用ssh-known-hosts.patch.yaml覆盖ssh的配置，保证环境与git配置一致。

接下来，继续在apps根目录注册应用，配置environments/dev/kustomization.yaml

resources:
 - ../../cluster-infra/argocd/overlays/with-known-hosts

编写根目录git配置

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: dev-root 
 namespace: argocd 
spec: 
 project: default 
 source: 
 repoURL: git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git 
 targetRevision: master 
 path: environments/dev 
 kustomize: 
 enableHelm: true 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: argocd 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

到此，所有配置迁移完成，将git仓库提交到远程后，在master节点添加私钥

argocd repo add git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git --ssh-private-key-path /root/.ssh/gitops_ed25519

同步dev根应用

kubectl apply -f dev-root.yaml

本地使用kubectl暴露临时端口，再使用argocd cli连接集群查看状态

kubectl port-forward svc/argocd-server -n argocd --address 0.0.0.0 30443:443
argocd login xxxx:30443 --username admin
argocd app list

部署ingress

一直使用临时端口+命令行查看应用状态也不方便，官方提供webui界面查看部署状态，考虑使用ingress暴露固定域名访问

ingress的部署也遵循相同步骤

配置官方安装文件
配置环境个性化
1. 端口暴露
根应用添加配置

将官方https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml安装文件配置到cluster-infra/ingress/base/ingress-nginx.yaml

配置端口暴露cluster-infra/ingress/overlays/dev/ingress-nginx-controller-nodeport.yaml

apiVersion: v1 
kind: Service 
metadata: 
 name: ingress-nginx-controller 
 namespace: ingress-nginx 
spec: 
 type: NodePort 
 ports: 
 - name: http 
 port: 80 
 targetPort: http 
 protocol: TCP 
 nodePort: 30080 
 - name: https 
 port: 443 
 targetPort: https 
 protocol: TCP 
 nodePort: 30443

根应用添加配置

resources: 
 - ../../cluster-infra/ingress/overlays/dev 
 - ../../cluster-infra/argocd/overlays/with-known-hosts

在argocd环境目录下添加ingress域名配置

apiVersion: networking.k8s.io/v1 
kind: Ingress 
metadata: 
 name: argocd-ingress 
 namespace: argocd 
 annotations: 
 nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" 
spec: 
 ingressClassName: nginx 
 rules: 
 - host: xxxx 
 http: 
 paths: 
 - path: / 
 pathType: Prefix 
 backend: 
 service: 
 name: argocd-server 
 port: 
 number: 443

git提交后即可自动同步并创建ingress

访问域名查看argocd webui

部署Prometheus+Grafana

为了完善集群监控，argocd 支持通过helm charts部署应用

配置cluster-infra/prometheus-stack/base/prometheus-stack.yaml

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: prometheus-stack 
 namespace: argocd 
 annotations: 
 argocd.argoproj.io/sync-options: ServerSideApply=true 
spec: 
 project: default 
 source: 
 repoURL: https://prometheus-community.github.io/helm-charts 
 chart: kube-prometheus-stack 
 targetRevision: 75.16.1 
 helm: 
 valueFiles: 
 - values.yaml 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: monitoring 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

注意，由于Helm仓库部署prometheus时伴随大量CRD资源文件apply，需要开启ServerSideApply模式避免通过接口apply导致字符超长报错。参考：https://argocd.devops.gold/user-guide/sync-options/#_9

使用helm模式安装的应用无法使用patchesStrategicMerge策略覆盖，目前覆盖个性化配置只能将所有内容写到一个资源文件内，因此dev环境的个性化配置values如下

prometheus: 
 prometheusSpec: 
 retention: 3d 
 retentionSize: "2Gi" 
 resources: 
 requests: 
 memory: 128Mi 
 limits: 
 memory: 1Gi 
 storageSpec: 
 volumeClaimTemplate: 
 spec: 
 resources: 
 requests: 
 storage: 5Gi 
 ingress: 
 enabled: true 
 ingressClassName: nginx 
 annotations: {} 
 hosts: 
 - prometheus.fjy8018.top 
 paths: 
 - / 
 tls: 
 - hosts: 
 - xxx.xxx
 secretName: prometheus-tls 
 prometheusSpec: 
 externalUrl: https://xxx.xxx 
 routePrefix: / 
 
grafana: 
 adminPassword: "2EvwJJ0NOLzhYa" 
 resources: 
 requests: 
 memory: 64Mi 
 limits: 
 memory: 512Mi 
 ingress: 
 enabled: true 
 ingressClassName: nginx 
 annotations: {} 
 hosts: 
 - xxx.xxx 
 tls: 
 - hosts: 
 - xxx.xxx
 secretName: grafana-tls

需要在dev中覆盖base的资源文件，通过multi repo模式合并helm配置

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: prometheus-stack 
 namespace: argocd 
 annotations: 
 argocd.argoproj.io/sync-options: ServerSideApply=true 
spec: 
 project: default 
 sources: 
 - repoURL: https://prometheus-community.github.io/helm-charts 
 chart: kube-prometheus-stack 
 targetRevision: 75.16.1 
 helm: 
 valueFiles: 
 - $values/cluster-infra/prometheus-stack/overlays/dev/values-dev.yaml 
 - repoURL: 'git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git' 
 targetRevision: master 
 ref: values 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: monitoring 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

prometheus依赖CRD提前部署的namespace需要注册为独立应用提前配置好

apiVersion: v1 
kind: Namespace 
metadata: 
 name: monitoring

命名空间的配置和普通应用一致

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: common-ns 
 namespace: argocd 
spec: 
 project: default 
 source: 
 repoURL: git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git 
 targetRevision: master 
 path: cluster-infra/common-ns 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: argocd 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

最终根目录应用配置如下

resources: 
 - ../../cluster-infra/ingress/overlays/dev 
 - ../../cluster-infra/argocd/overlays/with-known-hosts 
 - ../../cluster-infra/prometheus-stack/overlays/dev 
 - common-ns-app.yaml

同步后效果如下

查看监控部署拓扑

通过域名访问grafana

本地LLM训练

Sun, 30 Mar 2025 11:44:00 +0800

参考资料

jingyaogong/minimind

环境准备

本地环境配置

CPU 12700
内存 32G DDR4 3200
GPU RTX 2060Super
系统 Win11
CUDA 12.6
Python 3.12

由于需要本地编译，windows环境需要安装Microsoft Visual C++ 编译器，tiktoken编译需要rust环境，安装地址：

克隆并安装环境

git clone https://github.com/jingyaogong/minimind.git
cd minimind
pip install -r requirements.txt

注意：此处不建议使用清华源、也不建议使用conda虚拟环境，会导致pytorch包版本匹配失败

依赖安装检查，执行以下代码

import torch 
print(torch.__version__) 
print(torch.version.cuda) 
print(torch.cuda.is_available())

输入如下即成功

(base) PS F:\java_workspace\minimind> python .\test_env.py
2.6.0+cu126
12.6
True

按照文档指引下载默认的预训练集和监督训练集

minimind_dataset · 数据集

训练过程

预训练

执行预训练

python train_pretrain.py

在我的环境下需要3小时才能完成预训练

执行预训练，得到pretrain_*.pth作为预训练的输出权重（其中为模型的dimension，默认为512）

监督微调

python train_full_sft.py

执行监督全参数微调，在我的环境下需要2.5小时

效果测试

测试全参微调

python eval_model.py --model_mode 1

从结果来看，能进行对话，但受限于参数量，很多知识不完整，对话回答质量比较一般。

模型优化

受本地环境限制，只能完成监督微调步骤，后续步骤执行会爆显存，耗时显著增加。后续学习改为使用阿里云PAI-DSW环境

DeepSeek 本地部署开启联网搜索

Sun, 09 Feb 2025 17:28:00 +0800

背景

由于官方DeepSeek频繁不可用，第三方API站点部署的全量模型无法联网，因此有需要联网推理的场景只能使用本地部署解决。

环境

CPU：12700 内存：32G DDR4 3200 显卡：RTX 2060 Super

模型准备

受显存限制，使用ollama只能下载8b和14b模型。

8b模型可以完全加载到显存中，使用cuda推理。
14b模型需要利用CPU+GPU混合推理。

本地下载开源docker镜像open-webui

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

该镜像下载困难可以手动传到阿里云镜像仓库做一次代理，本次使用我同步的代理镜像启动

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --rm registry.cn-guangzhou.aliyuncs.com/fjy8018_proxy/open-webui:main

首次启动耗时较长，耐心等待，显示如下画面表示启动完成

联网API

进入http://localhost:3000/admin/settings 点击联网搜索配置搜索引擎

目前有3种免费的联网搜索API，各有优劣

Duckduckgo

优点：免费，无需API配置缺点：网络无法直接访问。

Bing Search

优点：免费，国内网络可直连。缺点：需要微软账户+信用卡认证。

Google Search

优点：免费，搜索内容可靠。缺点：需要配置，网络无法直连。

此处为保障最好的搜索质量，使用Google搜索API接入。

申请API和密钥

进入谷歌可编程搜索引擎官网：Programmable Search Engine | Google for Developers 创建搜索引擎

点击控制面板

创建搜索引擎，名称任意

创建后得到搜索引擎ID

选择程序化访问

获取密钥

配置联网搜索

选择google pse，填写密钥和搜索ID

验证

8b + 联网搜索

使用8b+联网搜索验证

容器日志可以看到搜索到的文本内容

8b联网+推理显存沾满，显卡满载，但不需要CPU参与推理，32token/s，速度较快。

14b+联网搜索

改成14b后，token速度下降到7token/s，勉强可用。

由于显存不足，需要CPU+GPU混合推理，显卡cuda性能无法完全利用，CPU占用率高。

ClassLoader类隔离

Wed, 13 Nov 2024 21:56:49 +0800

基本概念

参考oracle jdk 21中对ClassLoader定义

A class loader is an object that is responsible for loading classes. The class ClassLoader is an abstract class. Given the binary name of a class, a class loader should attempt to locate or generate data that constitutes a definition for the class. A typical strategy is to transform the name into a file name and then read a “class file” of that name from a file system.

在Java中，ClassLoader负责加载类，当给定一个类的二进制名称，ClassLoader会尝试定位或生成构成该类定义的数据。

层次结构

ClassLoader遵循双亲委派模型，当请求查找类或资源时， ClassLoader 实例通常会将类或资源的搜索委托给其父类加载器，然后再尝试查找类或资源本身

由于JDK9中模块系统（Jigsaw）的引入，jdk8和jdk9类加载器层次结构有区别，ExtClassLoader变成PlatformClassLoader负责加载--module-path（或MODPATH环境变量）指定的模块

graph TD
 Bootstrap[Bootstrap ClassLoader]
 Extension[Extension ClassLoader]
 Application[Application ClassLoader]

 Bootstrap --> Extension
 Extension --> Application

 subgraph "JDK 8"
 Bootstrap:::primary
 Extension:::secondary
 Application:::tertiary
 end

graph TD
 Bootstrap[Bootstrap ClassLoader]
 Platform[Platform ClassLoader]
 App[Application ClassLoader]

 Bootstrap --> Platform
 Platform --> App

 subgraph "JDK 9"
 Bootstrap:::primary
 Platform:::secondary
 App:::tertiary
 end

生命周期

类加载的过程可以分为几个阶段及常见异常

加载（Loading）
ClassNotFoundException（找不到目标类）
验证（Verification）
ClassFormatError（类文件格式不正确）
NoClassDefFoundError（已加载的类无法初始化）
准备（Preparation）
解析（Resolution）
LinkageError（类加载器无法找到或者无法链接到其他类或接口）
IncompatibleClassChangeError
AbstractMethodError
NoSuchMethodError
…
初始化（Initialization）

核心方法

ClassLoader类提供了几个关键的方法来支持类的加载过程，其中loadClass和findClass最常使用

loadClass
protected Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException
Loads the class with the specified binary name. The default implementation of this method searches for classes in the following order:

Invoke findLoadedClass(String) to check if the class has already been loaded.

Invoke the loadClass method on the parent class loader. If the parent is null the class loader built into the virtual machine is used, instead.

Invoke the findClass(String) method to find the class.

If the class was found using the above steps, and the resolve flag is true, this method will then invoke the resolveClass(Class) method on the resulting Class object.

Subclasses of ClassLoader are encouraged to override findClass(String), rather than this method.

Unless overridden, this method synchronizes on the result of getClassLoadingLock method during the entire class loading process.

Parameters:

name - The binary name of the class

resolve - If true then resolve the class

Returns:

The resulting Class object

Throws:

ClassNotFoundException - If the class could not be found

官方文档给出了方法设计介绍，loadClass用于实现双亲委派职责，当父类无法加载目标class时，调用子类的findClass加载。

当需要打破双亲委派模型时，则直接覆盖loadClass

findClass
protected Class<?> findClass(String name) throws ClassNotFoundException
Finds the class with the specified binary name. This method should be overridden by class loader implementations that follow the delegation model for loading classes, and will be invoked by the loadClass method after checking the parent class loader for the requested class.

Implementation Requirements:

The default implementation throws ClassNotFoundException.

Parameters:

name - The binary name of the class

Returns:

The resulting Class object

Throws:

ClassNotFoundException - If the class could not be found

Since:

1.2

在findClass用于实现自定义类加载逻辑

以常用的URLClassLoader为例，findClass将传入的类名转换成路径格式匹配在URLClassPath类路径下实际存在的文件结构，如果找到了对应的资源，则继续处理；否则返回null。

public class URLClassLoader extends SecureClassLoader implements Closeable {
 protected Class<?> findClass(final String name)
 throws ClassNotFoundException
 {
 final Class<?> result;
 try {
 result = AccessController.doPrivileged(
 new PrivilegedExceptionAction<Class<?>>() {
 public Class<?> run() throws ClassNotFoundException {
 String path = name.replace('.', '/').concat(".class");
 Resource res = ucp.getResource(path, false);
 if (res != null) {
 try {
 return defineClass(name, res);
 } catch (IOException e) {
 throw new ClassNotFoundException(name, e);
 }
 } else {
 return null;
 }
 }
 }, acc);
 } catch (java.security.PrivilegedActionException pae) {
 throw (ClassNotFoundException) pae.getException();
 }
 if (result == null) {
 throw new ClassNotFoundException(name);
 }
 return result;
 }
}

应用

tomcat

为了支持应用隔离和模块化部署，tomcat扩展了已有的双亲委派模型

以tomcat9为例，其层次结构如下

graph TD
 Bootstrap[Bootstrap ClassLoader]
 System[System ClassLoader]
 Common[Common ClassLoader]
 Webapp1[Webapp1 ClassLoader]
 Webapp2[Webapp2 ClassLoader]
 Webapp3[Webapp3 ClassLoader]

 Bootstrap --> System
 System --> Common
 Common --> Webapp1
 Common --> Webapp2
 Common --> Webapp3

Bootstrap（所有应用可见，JVM核心依赖）
$JAVA_HOME/jre/lib/ext
System（所有应用可见，tomcat核心依赖）
$CATALINA_HOME/bin
Common（需要对所有应用可见的扩展类）
Webapp（每个应用隔离，打破双亲委派，本地优先）

As mentioned above, the web application class loader diverges from the default Java delegation model (in accordance with the recommendations in the Servlet Specification, version 2.4, section 9.7.2 Web Application Classloader). When a request to load a class from the web application’s WebappX class loader is processed, this class loader will look in the local repositories first, instead of delegating before looking.

重点在于webapp classloader，其打破双亲委派机制，当从 Web 应用程序的 WebappX 类加载器加载类的请求被处理时，这个类加载器会首先查看本地存储库，而不是先进行委托。

在应用程序视角，类查找顺序如下

Bootstrap ClassLoader
WebApp ClassLoader
/WEB-INF/classes
/WEB-INF/lib/*.jar
System ClassLoader
Common ClassLoader

当应用配置<Loader delegate="true"/>时，可切换为标准双亲委派模型

OSGi

OSGi是一个 Java 模块化系统和服务平台，它提供了一种动态管理模块（Bundle）的方法，支持不停止应用程序的情况下动态更新、添加或删除模块。

以Apache Felix实现的OSGi为例，其查找逻辑如下

循环检测：防止在查找过程中出现无限递归。
委托给父类加载器：如果目标类或资源属于需要委托给父类加载器处理的包，则先尝试使用父类加载器进行查找。
从导入列表中查找：在当前修订版本的导入列表中搜索类或资源。
从本地类路径查找：如果前面步骤未能找到，则尝试从当前修订版本自己的类路径中查找。
从动态导入中查找：最后，如果还没有找到，则会尝试从动态导入中查找。

private Object findClassOrResourceByDelegation(String name, boolean isClass)
 throws ClassNotFoundException, ResourceNotFoundException
 {
 // 初始化结果为null，表示尚未找到类或资源
 Object result = null;

 // 用于循环检测的集合，避免在查找过程中出现无限递归的情况
 Set requestSet = (Set) m_cycleCheck.get();
 if (requestSet == null)
 {
 // 如果当前线程中没有设置循环检测集合，则创建一个新的并设置到当前线程
 requestSet = new HashSet();
 m_cycleCheck.set(requestSet);
 }
 // 将要查找的名字添加到循环检测集合中，如果已经存在说明出现了循环依赖
 if (requestSet.add(name))
 {
 try
 {
 // 根据是查找类还是资源来获取目标的包名
 String pkgName = (isClass)
 ? Util.getClassPackage(name)
 : Util.getResourcePackage(name);

 // 检查是否应该将该包委托给父类加载器
 if (shouldBootDelegate(pkgName))
 {
 try
 {
 // 获取适当的类加载器进行委托
 ClassLoader bdcl = getBootDelegationClassLoader();
 // 根据是要查找类还是资源，调用相应的方法
 result = (isClass)
 ? (Object) bdcl.loadClass(name)
 : (Object) bdcl.getResource(name);
 // 如果是java.*包下的内容或者已经找到了结果，则直接返回
 if (pkgName.startsWith("java.") || (result != null))
 {
 return result;
 }
 }
 catch (ClassNotFoundException ex)
 {
 // 如果是java.*包下的类且未找到，则抛出异常
 if (pkgName.startsWith("java."))
 {
 throw ex;
 }
 }
 }

 // 在导入列表中查找
 result = searchImports(pkgName, name, isClass);

 // 如果在导入列表中未找到，则尝试从修订版本自己的类路径中查找
 if (result == null)
 {
 if (isClass)
 {
 // 获取内部类加载器，并尝试加载类
 ClassLoader cl = getClassLoaderInternal();
 if (cl == null)
 {
 // 如果类加载器无效，则抛出异常
 throw new ClassNotFoundException(
 "Unable to load class '"
 + name
 + "' because the bundle wiring for "
 + m_revision.getSymbolicName()
 + " is no longer valid.");
 }
 result = ((BundleClassLoader) cl).findClass(name);
 }
 else
 {
 // 如果不是查找类而是资源，则使用修订版本的方法来查找资源
 result = m_revision.getResourceLocal(name);
 }

 // 如果还是没找到，则尝试从动态导入中查找
 if (result == null)
 {
 result = searchDynamicImports(pkgName, name, isClass);
 }
 }
 }
 finally
 {
 // 清理：从循环检测集合中移除当前查找的名字
 requestSet.remove(name);
 }
 }
 else
 {
 // 如果检测到循环依赖，则返回null以打破循环
 return null;
 }

 // 如果最终结果仍为null，根据查找类型抛出相应的异常
 if (result == null)
 {
 if (isClass)
 {
 throw new ClassNotFoundException(
 name + " not found by " + this.getBundle());
 }
 else
 {
 throw new ResourceNotFoundException(
 name + " not found by " + this.getBundle());
 }
 }

 // 返回最终找到的结果
 return result;
 }

SofaArk

SofaArk是蚂蚁集团开源轻量类隔离框架

其定义3个概念：Ark Container，Ark Plugin，Ark Biz。

sofaark通过maven插件在打包时将插件启动器、导出类、导入类、导出资源、导入资源、优先级等元信息保存在 Ark Plugin 中的 META-INF/MANIFEST.MF 中，每个ark plugin都有独立的类加载器读取插件元信息和类资源导入导出关系。插件之间可以互相委托加载类。
ark biz只能加载自己模块的类信息，也可以委托plugin加载所需的类。

graph TD
 Bootstrap[BootstrapClassLoader]
 Extension[ExtensionClassLoader]
 Application[ApplicationClassLoader]
 URLClassLoader[URLClassLoader]
 Container[ContainerClassLoader]
 Agent[AgentClassLoader]
		Biz[BizClassLoader]
		Plugin[PluginClassLoader]

 Bootstrap --> Extension
 Extension --> Application
 Application --> URLClassLoader
	 URLClassLoader --> Container
		URLClassLoader --> Biz
		URLClassLoader --> Plugin
		URLClassLoader --> Agent

源码

AbstractClasspathClassLoader提供了类加载查找模式的抽象实现，loadClass会从以下顺序查找

本地类加载缓存
子类实现的loadClassInternal方法

BizClassLoader实现如下

package com.alipay.sofa.ark.container.service.classloader;

public class BizClassLoader extends AbstractClasspathClassLoader {

 
 @Override
 protected Class<?> loadClassInternal(String name, boolean resolve) throws ArkLoaderException {
 Class<?> clazz = null;

 // 0. sun reflect related class throw exception directly
 if (classloaderService.isSunReflectClass(name)) {
 throw new ArkLoaderException(
 String
 .format(
 "[ArkBiz Loader] %s : can not load class: %s, this class can only be loaded by sun.reflect.DelegatingClassLoader",
 bizIdentity, name));
 }

 // 1. findLoadedClass
 if (clazz == null) {
 clazz = findLoadedClass(name);
 }

 // 2. JDK related class
 if (clazz == null) {
 clazz = resolveJDKClass(name);
 }

 // 3. Ark Spi class
 if (clazz == null) {
 clazz = resolveArkClass(name);
 }

 // 4. pre find class
 if (clazz == null) {
 clazz = preLoadClass(name);
 }

 // 5. Plugin Export class
 if (clazz == null) {
 clazz = resolveExportClass(name);
 }

 // 6. Biz classpath class
 if (clazz == null) {
 clazz = resolveLocalClass(name);
 }

 // 7. Java Agent ClassLoader for agent problem
 if (clazz == null) {
 clazz = resolveJavaAgentClass(name);
 }

 // 8. post find class
 if (clazz == null) {
 clazz = postLoadClass(name);
 }

 if (clazz != null) {
 if (resolve) {
 super.resolveClass(clazz);
 }
 return clazz;
 }

 throw new ArkLoaderException(String.format("[ArkBiz Loader] %s : can not load class: %s",
 bizIdentity, name));
 }
}

PluginClassLoader实现如下

package com.alipay.sofa.ark.container.service.classloader;

public class PluginClassLoader extends AbstractClasspathClassLoader {

	@Override
 protected Class<?> loadClassInternal(String name, boolean resolve) throws ArkLoaderException {
 Class<?> clazz = null;

 // 0. sun reflect related class throw exception directly
 if (classloaderService.isSunReflectClass(name)) {
 throw new ArkLoaderException(
 String
 .format(
 "[ArkPlugin Loader] %s : can not load class: %s, this class can only be loaded by sun.reflect.DelegatingClassLoader",
 pluginName, name));
 }

 // 1. findLoadedClass
 if (clazz == null) {
 clazz = findLoadedClass(name);
 }

 // 2. JDK related class
 if (clazz == null) {
 clazz = resolveJDKClass(name);
 }

 // 3. Ark Spi class
 if (clazz == null) {
 clazz = resolveArkClass(name);
 }

 // 4. pre find class
 if (clazz == null) {
 clazz = preLoadClass(name);
 }

 // 5. Import class export by other plugins
 if (clazz == null) {
 clazz = resolveExportClass(name);
 }

 // 6. Plugin classpath class
 if (clazz == null) {
 clazz = resolveLocalClass(name);
 }

 // 7. Java Agent ClassLoader for agent problem
 if (clazz == null) {
 clazz = resolveJavaAgentClass(name);
 }

 // 8. Post find class
 if (clazz == null) {
 clazz = postLoadClass(name);
 }

 if (clazz != null) {
 if (resolve) {
 super.resolveClass(clazz);
 }
 return clazz;
 }

 throw new ArkLoaderException(String.format(
 "[ArkPlugin Loader] %s : can not load class: %s", pluginName, name));
 }
}

流程基本一致，主要在于plugin和biz读取到的import类元信息不同

参考资料

基于 ChatGLM2-6B 搭建个人知识库

Wed, 06 Sep 2023 22:43:00 +0800

背景

ChatGLM2-6B 是开源中英双语对话模型 ChatGLM-6B的第二代版本，提升了性能和推理能力，可结合Langchain-Chatchat和Embedding 可实现基于本地知识库的推理能力

准备环境

由于需要构建强大的知识库而非通过prompt组装问答方式，故需要更大的显存和内存，此处直接使用阿里云提供的A10显卡，可免费试用3个月

申请机器

创建工作空间

创建实例

启动机器后进入工作台

部署ChatGLM2-6B

下载仓库

参考官方仓库的部署指南

$ git clone https://github.com/THUDM/ChatGLM2-6B
$ cd ChatGLM2-6B

然后使用 pip 安装依赖：

$ pip install -r requirements.txt

安装git lfs

阿里云的git源直接安装lfs会存在问题，需要安装前置依赖

$ apt -y install build-essential nghttp2 libnghttp2-dev libssl-dev
$ git lfs install --skip-smudge

下载模型

$ git clone https://huggingface.co/THUDM/chatglm2-6b /mnt/workspace/model/chatglm2-6b
$ git lfs fetch
$ git lfs checkout

修改启动配置

此处只使用webUI模式验证，修改 web_demo.py

tokenizer= AutoTokenizer.from_pretrained("/mnt/workspace/model/chatglm2-6b", trust_remote_code=True)
model= AutoModel.from_pretrained("/mnt/workspace/model/chatglm2-6b", trust_remote_code=True).half().cuda()

# 要暴露在公网上
demo.queue().launch(share=True, inbrowser=True, server_name='0.0.0.0', server_port=7860)

安装启动依赖

$ pip install gradio

启动WebUI

$ python web_demo.py

启动成功

使用非量化模型，显存占用12G

问答验证

基于A10显卡推理，运算速度非常快，对问题的理解能力也比之前体验的chatglm-6b更加准确，但依然有不少文字错误

部署Langchain知识库应用

介绍

Langchain-Chatchat是一种利用 langchain 思想实现的基于本地知识库的问答应用，目标期望建立一套对中文场景与开源模型支持友好、可离线运行的知识库问答解决方案。

原理示意图

更多介绍参考官方仓库

下载仓库

Langchain依赖的python包有些许差别，需要先升级python依赖版本

$ pip3 install --upgrade pip
$ pip uninstall accelerate
$ pip install accelerate>=0.20.3

# 拉取仓库
$ git clone https://github.com/chatchat-space/Langchain-Chatchat.git

# 进入目录
$ cd Langchain-Chatchat

# 安装全部依赖
$ pip install -r requirements.txt

下载Embedding模型

此处使用官方推荐的m3e-base模型为例

$ git clone https://huggingface.co/moka-ai/m3e-base
$ git lfs fetch
$ git lfs checkout

修改配置

修改Langchain模型引用配置

$ cd Langchain-Chatchat

复制样例文件，创建新配置configs/model_config.py 和 configs/server_config.py

本地的 LLM 模型本地存储路径写在 llm_model_dict 对应模型的 local_model_path 属性中

llm_model_dict={
 "chatglm2-6b": {
 "local_model_path": "/mnt/workspace/model/chatglm2-6b",
 "api_base_url": "http://127.0.0.1:9090/v1", # "name"修改为 FastChat 服务中的"api_base_url"
 "api_key": "EMPTY"
 },
 }

本地的 Embedding 模型本地存储路径写在 embedding_model_dict 对应模型位置

embedding_model_dict = {
 "m3e-base": "/mnt/workspace/model/m3e-base",
 }

重建知识库

首次使用模型需要重建知识库

$ python init_database.py --recreate-vs

启动知识库

一键启动所有 Fastchat 服务、API 服务、WebUI 服务

$ python startup.py -a

问答验证

LLM模型验证，连接本地ChatGLM2-6B成功

工程自带样例知识库，可以使用内置知识库验证，此处跳过，直接使用个人知识库验证

构建知识库

基金研报知识库

从天天基金拉取研报：中信证券：把握宏观政策接连发力下电商板块估值修复机会_天天基金网 (eastmoney.com)

将内容存储为txt上传到知识库

基于基金研报知识库问答，总结准确，速度非常快

推理过程日志，显存占用14G

个人面试知识库

将Java并发知识点转为txt类型知识库上传并创建新的知识库空间

不使用知识库情况下回答

使用知识库情况下回答

显然使用知识库回答更加准确

ChatGLM-6B 本地化部署体验

Sun, 28 May 2023 19:50:00 +0800

背景

在ChatGPT引爆大语言模型热潮下，国内清华大学开源了包含62亿参数的GLM-6B，由于使用的参数较少，因此能支持在消费级显卡上私有化部署体验。

硬件

CPU：12700

内存：32G

GPU：RTX 2060 SUPER 8G

实践-ChatGLM-6B官方版本

介绍

GitHub - THUDM/ChatGLM-6B: ChatGLM-6B: An Open Bilingual Dialogue Language Model | 开源双语对话语言模型

ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于 General Language Model (GLM) 架构，具有 62 亿参数。结合模型量化技术，用户可以在消费级的显卡上进行本地部署（INT4 量化级别下最低只需 6GB 显存）。 ChatGLM-6B 使用了和 ChatGPT 相似的技术，针对中文问答和对话进行了优化。经过约 1T 标识符的中英双语训练，辅以监督微调、反馈自助、人类反馈强化学习等技术的加持，62 亿参数的 ChatGLM-6B 已经能生成相当符合人类偏好的回答，更多信息请参考我们的博客。

部署

采用秋叶一键部署包，使用8G显存版本

启动成功，访问web-ui

对话

对话流畅度与chatgpt类似，精准度不如gpt4

由于显存限制，咨询过大的问题会直接导致爆显存无法回答

实践-闻达版本

介绍

GitHub - wenda-LLM/wenda: 闻达：一个LLM调用平台。为小模型外挂知识库查找和设计自动执行动作，实现不亚于于大模型的生成能力

闻达基于GLM-6B扩展，支持了本地知识库和联网知识库，提供了近似new bing的体验

部署

使用官方一键包，默认使用6g显存模型

启动成功

对话-离线

只使用已有模型数据，询问

回答内容文不对题

主动提供正确数据内容后再询问，此时回答正确

对话-使用在线知识库

打开联网知识库，清理历史后重新询问，不需要提供本地知识也可以自行通过网络检索后回答正确

系统设计问题

文档解析

直接读取文档

直接使用读取文档并对话的方式很容易脱离文档内容，导致答非所问

尽管使用的是6g显存的版本，读取较大的文档时也会爆显存

手动模拟文档读取

GLM文档分析实际上是将文档识别后转换为特定prompt进行问答对话

参考内嵌prompt

手动复制文档内容进行模拟对话，内容回答正确

类AutoGPT功能

使用wenda提供的“根据标题写论文功能”可自动触发AutoGPT模式

其会自己列提纲，并逐个扩展，无需手工操作

总结

在消费级甜品卡上已经能实现接近chatgpt3的能力，ChatGLM-130B更是拥有1300亿参数的中英双语稠密模型，在能力上更进一步。

未来GLM企业级私有化部署可能会被包装为简单易用的产品向政府、企业推广，满足数据保密要求。

GraphQL实践10——Netflix Dgs Graphql异步订阅

Fri, 28 Oct 2022 21:59:23 +0800

介绍

GraphQL订阅模式支持服务端主动向客户端推送数据通知，避免客户端轮训

DGS Subscriptions

新增依赖

此处使用的是SpringBoot Web组件，故引入websockets依赖

如果使用的是WebFlux则要引入对应的dgs-webflux

		 <dependency>
 <groupId>com.netflix.graphql.dgs</groupId>
 <artifactId>graphql-dgs-subscriptions-websockets-autoconfigure</artifactId>
 </dependency>

异步通知解析

此处定义为当新增一个Actor就给所有订阅者发异步通知

@Slf4j
@DgsComponent
@RequiredArgsConstructor
public class ActorDataFetcher {

	private final ActorAssembler actorAssembler;

	private FluxSink<Actor> actorStream;

	private ConnectableFlux<Actor> actorPublisher;

	@PostConstruct
	private void createActorPublisher() {
		Flux<Actor> publisher = Flux.create(emitter -> {
			actorStream = emitter;
		});

		actorPublisher = publisher.publish();
		actorPublisher.connect();
	}

	@DgsMutation
	public Actor addActor(@InputArgument SubmitActor actor) {
		Actor actorEntity = actorAssembler.convert(actor);
		actorEntity.setActorId(10);
		actorEntity.setLastUpdate(DateUtil.format(new Date(), DatePattern.NORM_DATETIME_PATTERN));
		actorStream.next(actorEntity);
		log.info("服务端创建演员：{}", actor);
		return actorEntity;
	}

	/**
	 * 异步通知
	 *
	 * @return
	 */
	@DgsSubscription
	public Publisher<Actor> actorAdded() {
		return actorPublisher;
	}
}

单元测试

由于异步通知无法用浏览器自带的控制台测试，只能通过GraphQL客户端订阅后接收异步通知，此处参考官方最佳实践，使用单元测试模拟

@Slf4j
@SpringBootTest
class ActorDataFetcherTest {

	@Autowired
	private DgsQueryExecutor dgsQueryExecutor;

	@Test
	void addActor() {

		GraphQLQueryRequest graphQLQueryRequest = new GraphQLQueryRequest(
				AddActorGraphQLQuery.newRequest()
						.actor(
								SubmitActor.newBuilder()
										.firstName("fu")
										.lastName("jiayang")
										.build())
						.build(),
				new AddActorProjectionRoot()
						.actorId()
						.firstName()
						.lastUpdate());
		dgsQueryExecutor.execute(graphQLQueryRequest.serialize());
	}

	@Test
	void actorAdded() {
		ExecutionResult executionResult = dgsQueryExecutor.execute("""
				subscription {
				 actorAdded {
				 actorId
				 firstName
				 lastName
				 lastUpdate
				 }
				}
				""");
		Publisher<ExecutionResult> reviewPublisher = executionResult.getData();
		List<Actor> actors = new CopyOnWriteArrayList<>();

		reviewPublisher.subscribe(new Subscriber<>() {
			@Override
			public void onSubscribe(Subscription s) {
				s.request(2);
			}

			@Override
			public void onNext(ExecutionResult executionResult) {
				if (executionResult.getErrors().size() > 0) {
					executionResult.getErrors().forEach(error -> log.error(error.toString()));
				}
				Map<String, Object> actorResult = executionResult.getData();
				Actor actor = new ObjectMapper().convertValue(actorResult.get("actorAdded"), Actor.class);
 log.info("客户端监听到演员新增：{}",actor);
				actors.add(actor);
			}

			@Override
			public void onError(Throwable t) {
				t.printStackTrace();
			}

			@Override
			public void onComplete() {
			}
		});

		addActor();
		addActor();

		assertThat(actors.size()).isEqualTo(2);
	}
}

从日志输出可以看到客户端成功接收异步通知

总结

订阅模式属于GraphQL特色功能，类似消息队列实现方式，减少客户端对普通查询接口轮训性能开销

GraphQL实践9——Netflix Dgs Graphql数据更新

Fri, 28 Oct 2022 21:58:54 +0800

功能介绍

graphql-java提供了Mutations类型支持数据更新

NGS Mutations

类型定义

type Mutation {
 addActor(actor: SubmitActor!): Actor
}

type Subscription {
 actorAdded: Actor
}

input SubmitActor {
 firstName: String!
 lastName: String!
}

SubmitActor类型可以交给代码生成器自动生成

类型转换器

@Mapper(componentModel = SPRING)
public interface ActorAssembler {

	Actor convert(SubmitActor submit);
}

定义解析器

	@DgsMutation
	public Actor addActor(@InputArgument SubmitActor actor) {
		Actor actorEntity = actorAssembler.convert(actor);
		actorEntity.setActorId(10);
		actorEntity.setLastUpdate(new Date());
		return actorEntity;
	}

测试

访问http://localhost:8080/graphiql即可看到在线查询页面

GraphQL实践8——Netflix Dgs Graphql代码生成

Fri, 28 Oct 2022 21:57:00 +0800

功能介绍

DGS官方开发了识别schema配置生成对应类型、测试类的代码生成器，而Maven插件则由社区维护

Maven代码生成插件

新增依赖

<dependency> 
 <groupId>com.netflix.graphql.dgs.codegen</groupId> 
 <artifactId>graphql-dgs-codegen-client-core</artifactId> 
 <version>5.1.17</version> 
</dependency>

还需要新增构建插件

	 <plugin>
 <groupId>io.github.deweyjose</groupId>
 <artifactId>graphqlcodegen-maven-plugin</artifactId>
 <version>1.24</version>
 <executions>
 <execution>
 <goals>
 <goal>generate</goal>
 </goals>
 </execution>
 </executions>
 <configuration>
 <addGeneratedAnnotation>true</addGeneratedAnnotation>
 <packageName>top.fjy8018.graphsqldemo.generated</packageName>
 <schemaPaths>
 <param>src/main/resources/graphql/actor.graphqls</param>
 </schemaPaths>
 <!-- 生成测试客户端 -->
 <generateClient>true</generateClient>
 </configuration>
 </plugin>
 <plugin>
 <groupId>org.codehaus.mojo</groupId>
 <artifactId>build-helper-maven-plugin</artifactId>
 <executions>
 <execution>
 <phase>generate-sources</phase>
 <goals>
 <goal>add-source</goal>
 </goals>
 <configuration>
 <sources>
 <source>${project.build.directory}/generated-sources</source>
 </sources>
 </configuration>
 </execution>
 </executions>
 </plugin>

插件配置指引：deweyjose/graphqlcodegen: Maven port of the Netflix Gradle code generation plugin for graphql.

编译

执行

mvn clean package

即可生成对应的类

存在问题

IDEA默认不识别target目录下的资源文件，生成后会导致无法引用

解决方法

需要手动将对应的生成目录配置为Source Root即可

总结

DGS官方推荐开发者优先使用代码生成器生成标准类库后再开发，可以减少开发量

GraphQL实践7——Netflix Dgs Graphql分页查询

Fri, 28 Oct 2022 16:06:31 +0800

问题背景

上一篇文章提到，使用List查询每次都返回全量数据，而实际场景更多使用分页查询，graphql-java提供Connection实现游标分页，在Dgs也有对应功能扩展Relay Pagination

集成Relay Pagination

新增依赖

 <dependency>
 <groupId>com.netflix.graphql.dgs</groupId>
 <artifactId>graphql-dgs-pagination</artifactId>
 </dependency>

Mybatis Plus 配置分页插件

@Configuration
@MapperScan("top.fjy8018.graphsqldemo.mapper")
public class MybatisPlusConfig {

 /**
 * 分页插件配置
 *
 * @return
 */
 @Bean
 public MybatisPlusInterceptor mybatisPlusInterceptor() {
 MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor();
 PaginationInnerInterceptor pageConfig = new PaginationInnerInterceptor(DbType.MYSQL);
 interceptor.addInnerInterceptor(pageConfig);
 return interceptor;
 }
}

Relay Pagination配置

type Query {
 actorList: [Actor]
 findOneActor(id : Int!): Actor
 filmList(current: Int!,size: Int!): FilmConnection
 findOneFilm(id : Int!): Film
}

type Actor {
 actorId: Int!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

type Film @connection {
 filmId: Int!
 title: String!
 description: String
 releaseYear: Int
 languageId: Int!
 originalLanguageId: Int
 rentalDuration: Int
 rentalRate: Float
 length: Int
 replacementCost: Float
 rating: String
 specialFeatures: String
 lastUpdate: String
 actors: [Actor]
}

@connection和XXXConection已经由graphql-java底层框架自动生成，无需手动定义

类型转换器

public interface ConnectionAssembler {

 /**
 * mybatis plus分页对象转为Connection
 *
 * @param page 分页对象
 * @param <T> 泛型
 * @return Connection
 */
 static <T> Connection<T> convert(IPage<T> page) {
 AtomicInteger index = new AtomicInteger();
 List<Edge<T>> defaultEdges = Stream.ofNullable(page.getRecords())
 .flatMap(Collection::stream)
 .map(e -> (Edge<T>) new DefaultEdge<>(e, new DefaultConnectionCursor(String.valueOf(index.incrementAndGet()))))
 .toList();
 return new DefaultConnection<>(defaultEdges,
 new DefaultPageInfo(new DefaultConnectionCursor(String.valueOf(page.getCurrent())),
 new DefaultConnectionCursor(String.valueOf(page.getPages())),
 page.getCurrent() > 1,
 page.getCurrent() < page.getPages()));
 }
}

Graphql解析配置

	@DgsQuery
	public Connection<Film> filmList(@InputArgument("current") Integer current,
									 @InputArgument("size") Integer size,
									 DgsDataFetchingEnvironment dfe) {
		IPage<Film> page = new PageDTO<>(current, size);
		IPage<Film> pageResult = filmRepository.page(page);
		return ConnectionAssembler.convert(pageResult);
	}

将List查询改为返回Connection即可

测试

访问http://localhost:8080/graphiql即可看到在线查询页面

查看查询数量，确认是按照分页查询

总结

到此，DSG核心功能尝试基本完成，基本实现常用的Rest功能，同时集成方便，而且官方文档清晰，但目前开源社区还没有太多的基于DSG的系统集成样例，还需要在实践中评估摸索。

thoughtworks也从2016年开始关注graphql，目前给出的建议也是“评估”，可以进行小规模尝试并在合适的时候推广

We’ve seen many successful GraphQL implementations on our projects. We’ve seen some interesting patterns of use too, including GraphQL for server-side resource aggregation. That said, we’ve concerns about misuse of this framework and some of the problems that can occur. Examples include performance gotchas around N+1 queries and lots of boilerplate code needed when adding new models, leading to complexity. There are workarounds to these gotchas such as query caching. Even though it’s not a silver bullet, we still think it’s worth assessing as part of your architecture.

GraphQL实践6——Netflix Dgs Graphql N+1问题

Fri, 28 Oct 2022 16:05:51 +0800

N+1问题介绍

对于上一篇文章样例，如果要获取每个电影的演员名单，要执行如下动作

查询所有电影清单
遍历N个电影，查询对应电影的演员名单

总查询开销为N+1次查询，代价非常大，效率低

优化方案 DataLoader

对于一对一表关联的情况且每个关联对象只有一个值，可以直接使用BatchLoader

但实际上，对于大部分表关联情况，通常为一对多或者多对多，不保证每个关联都有值，此时需要使用MappedBatchLoader维护关联关系

维护多对多实体关系Map

public interface ActorService {
	Map<Integer, List<Actor>> listByFilmId(Collection<Integer> filmIds);
}

@Slf4j
@Service
@RequiredArgsConstructor
public class ActorServiceImpl implements ActorService {

	private final ActorRepository actorRepository;

	private final FilmActorRepository filmActorRepository;

	@Override
	public Map<Integer, List<Actor>> listByFilmId(Collection<Integer> filmIds) {
		List<FilmActor> filmActors = filmActorRepository.listByFilmId(filmIds);
		if (CollectionUtils.isEmpty(filmActors)) {
			return Collections.emptyMap();
		}
		Map<Integer, List<FilmActor>> filmActorMap = filmActors.stream().collect(Collectors.groupingBy(FilmActor::getFilmId));
		List<Integer> actorIds = filmActors.stream().map(FilmActor::getActorId).distinct().toList();
		List<Actor> actors = actorRepository.listByIds(actorIds);
		Map<Integer, Actor> actorsMap = Stream.ofNullable(actors).flatMap(Collection::stream).collect(Collectors.toMap(Actor::getActorId, Function.identity(), (e1, e2) -> e1));
		Map<Integer, List<Actor>> result = Maps.newHashMapWithExpectedSize(filmIds.size());
		filmActorMap.forEach((k, v) -> {
			List<Actor> actorList = Stream.ofNullable(filmActorMap.get(k)).flatMap(Collection::stream).map(e -> actorsMap.get(e.getActorId())).toList();
			result.put(k, actorList);
		});
		return result;
	}
}

Dgs MappedBatchLoader 配置

@DgsDataLoader(name = "actors")
@RequiredArgsConstructor
public class FilmActorsDataLoader implements MappedBatchLoader<Integer, List<Actor>> {

	private final ActorService actorService;

	@Override
	public CompletionStage<Map<Integer, List<Actor>>> load(Set<Integer> keys) {
		return CompletableFuture.supplyAsync(() -> actorService.listByFilmId(keys));
	}
}

测试

访问http://localhost:8080/graphiql即可看到在线查询页面

此时执行嵌套查询只会查询2次，一次查电影列表，一次查所有电影的Actor列表，非常快速

总结

使用BatchLoader实现对N+1问题优化，但还有一个潜在问题，即大数据分页，该样例中，有1000个电影，对应5000+演员，查询一次获取全量数据，对数据库压力较大，也不符合实际场景，还需要进行分页优化

GraphQL实践5——Netflix Dgs Graphql懒加载

Fri, 28 Oct 2022 16:04:43 +0800

问题背景

在graphql中，需要查询的字段由调用方传递，如果此时存在关联表才能获取的字段，但前端不需要该字段的时候，默认查询所有字段会带来较大开销

例如

type Actor {
 actorId: Int!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

type Film {
 filmId: Int!
 title: String!
 description: String
 actors: [Actor]
}

type Query {
 filmList: [Film]
}

解决方案——懒加载

DGS框架中支持配置调用方传递了指定字段才读取对应的数据

配置懒加载

增加实体定义，模拟嵌套实体

Mybatis Plus配置

扫描配置

@Configuration
@MapperScan("top.fjy8018.graphsqldemo.mapper")
public class MybatisPlusConfig {
}

实体

表结构还是使用sakila样例数据库的表

@Data
public class Actor implements Serializable {

 @Serial
 private static final long serialVersionUID = 1L;

 /**
 * 联系人ID
 */
 @TableId(type = IdType.AUTO)
 private Integer actorId;

 private String firstName;

 private String lastName;

 private Date lastUpdate;

}

@Data
public class Film implements Serializable {

	@Serial
	private static final long serialVersionUID = 1L;

	/**
	 * 联系人ID
	 */
	@TableId(type = IdType.AUTO)
	private Integer filmId;

	private String title;

	private String description;

	private Integer releaseYear;

	private Integer languageId;

	private Integer originalLanguageId;

	private Integer rentalDuration;

	private BigDecimal rentalRate;

	private Integer length;

	private BigDecimal replacementCost;

	private String rating;

	private String specialFeatures;

	private Date lastUpdate;
}

@Data
public class FilmActor implements Serializable {

	@Serial
	private static final long serialVersionUID = 1L;

	/**
	 * 联系人ID
	 */
	@TableId(type = IdType.AUTO)
	private Integer actorId;

	private Integer filmId;

	private Date lastUpdate;
}

Mapper

public interface ActorMapper extends BaseMapper<Actor> {}
public interface FilmActorMapper extends BaseMapper<FilmActor> {}
public interface FilmMapper extends BaseMapper<Film> {}

DAO

public interface ActorRepository extends IService<Actor> {
}
public interface FilmActorRepository extends IService<FilmActor> {
	List<FilmActor> listByFilmId(@Size.List({@Size(min = 1, message = "至少要有一个元素")}) List<Integer> filmIds);
}
public interface FilmRepository extends IService<Film> {
}

@Validated
@Repository
@RequiredArgsConstructor
public class ActorRepositoryImpl extends ServiceImpl<ActorMapper, Actor> implements ActorRepository {
}
@Validated
@Repository
@RequiredArgsConstructor
public class FilmActorRepositoryImpl extends ServiceImpl<FilmActorMapper, FilmActor> implements FilmActorRepository {
	@Override
	public List<FilmActor> listByFilmId(List<Integer> filmIds) {
		return lambdaQuery().in(FilmActor::getFilmId, filmIds).list();
	}
}
@Validated
@Repository
@RequiredArgsConstructor
public class FilmRepositoryImpl extends ServiceImpl<FilmMapper, Film> implements FilmRepository {
}

Service

多表查询聚合逻辑

public interface ActorService {
	List<Actor> listByFilmId(Integer filmId);
}

@Slf4j
@Service
@RequiredArgsConstructor
public class ActorServiceImpl implements ActorService {

	private final ActorRepository actorRepository;

	private final FilmActorRepository filmActorRepository;

	@Override
	public List<Actor> listByFilmId(Integer filmId) {
		List<FilmActor> filmActors = filmActorRepository.listByFilmId(List.of(filmId));
		if (CollectionUtils.isEmpty(filmActors)) {
			return Collections.emptyList();
		}
		List<Integer> actorIds = filmActors.stream().map(FilmActor::getActorId).toList();
		return actorRepository.listByIds(actorIds);
	}
}

Graphql 查询解析器

@DgsComponent
@RequiredArgsConstructor
public class FilmDataFetcher {

	private final FilmRepository filmRepository;

	private final ActorService actorService;

	@DgsQuery
	public Collection<Film> filmList() {
		return filmRepository.list();
	}

	@DgsData(parentType = "Film")
	public List<Actor> actors(DgsDataFetchingEnvironment dfe) {
		Film film = dfe.getSource();
		return actorService.listByFilmId(film.getFilmId());
	}

	@DgsQuery
	public Film findOneFilm(@InputArgument Integer id) {
		return filmRepository.getById(id);
	}
}

增加DgsDataFetchingEnvironment入参，DGS会自动获取请求上下文参数

此处定义即为传入了actors参数才执行该方法获取对应的Actor集合，实现懒加载

测试

访问http://localhost:8080/graphiql即可看到在线查询页面

不访问Actor

此时可以看到日志输出，没有查询Actor表

访问Actor

此时DSG会执行N+1查询Actor表，性能损耗非常大

总结

对于表关联，开销较大字段，都可以使用DSG懒加载实现性能优化，但此时还是存在N+1查询性能问题，将在后面使用DataLoader优化

GraphQL实践4——Netflix Dgs Graphql集成Mybatis Plus

Fri, 28 Oct 2022 16:03:00 +0800

介绍

通过JPA或者Mybatis Plus集成Graphql框架已经很简便，但其实使用graphql-java集成过于底层，很多扩展内容实现并不方便，例如：

分页查询
资源懒加载
N+1问题

Netflix 基于长期的Graphql实践，2020年开源了DGS Framework，让解决上述问题更加简单，大量开源系统集成Graphql也是通过集成DGS组件

The DGS framework project started at Netflix in 2019 as internal teams began developing multiple GraphQL services. As 2020 wrapped up, Netflix decided to open source the framework and build a community around it.

集成Netflix DGS

官网：Home - DGS Framework (netflix.github.io)

引入依赖

技术栈：

SpringBoot 2.7.1
Mybatis Plus
JDK 17
Dgs SpringBoot

本次和上一篇文章一样，选择原因不在赘述

	<dependencyManagement>
 <dependencies>
 <dependency>
 <groupId>com.netflix.graphql.dgs</groupId>
 <artifactId>graphql-dgs-platform-dependencies</artifactId>
 <!-- The DGS BOM/platform dependency. This is the only place you set version of DGS -->
 <version>4.9.16</version>
 <type>pom</type>
 <scope>import</scope>
 </dependency>
 </dependencies>
 </dependencyManagement>

 <dependencies>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-validation</artifactId>
 </dependency>

 <dependency>
 <groupId>com.netflix.graphql.dgs</groupId>
 <artifactId>graphql-dgs-spring-boot-starter</artifactId>
 </dependency>

 <dependency>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 <optional>true</optional>
 </dependency>

 <dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <scope>runtime</scope>
 </dependency>
 <!-- mybatis-plus-boot-starter -->
 <dependency>
 <groupId>com.baomidou</groupId>
 <artifactId>mybatis-plus-boot-starter</artifactId>
 <version>3.5.2</version>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-test</artifactId>
 <scope>test</scope>
 </dependency>
 </dependencies>

配置文件

spring:
 datasource:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: root
 password: ****
 url: jdbc:mysql://localhost:3306/sakila?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
server:
 port: 8080
logging:
 level:
 top.fjy8018.graphsqldemo.mapper: DEBUG

dgs:
 graphql:
 schema-locations:
 - classpath*:graphql/*.graphqls

由于DGS默认扫描路径为classpath*:schema/**/*.graphql*，本次基于上一篇文章的工程配置，需要手动指定一下扫描文件目录

Mybatis Plus配置

扫描配置

@Configuration
@MapperScan("top.fjy8018.graphsqldemo.mapper")
public class MybatisPlusConfig {
}

实体

表结构还是使用sakila样例数据库的表

@Data
public class Actor implements Serializable {

 @Serial
 private static final long serialVersionUID = 1L;

 /**
 * 联系人ID
 */
 @TableId(type = IdType.AUTO)
 private Integer actorId;

 private String firstName;

 private String lastName;

 private Date lastUpdate;

}

Mapper

public interface ActorMapper extends BaseMapper<Actor> {
}

DAO

从分层模型定义上，MP自带的CRUD接口本质上还是一个DAO，故定义为Repository更加合适，不与业务逻辑Serice冲突

同时也建议QueryWrapper仅在对应的Repository使用，防止查询条件散落在各处

public interface ActorRepository extends IService<Actor> {
}

@Validated
@Repository
@RequiredArgsConstructor
public class ActorRepositoryImpl extends ServiceImpl<ActorMapper, Actor> implements ActorRepository {
}

Graphql 查询解析器

@DgsComponent
@RequiredArgsConstructor
public class ActorDataFetcher {

	private final ActorRepository actorRepository;

	@DgsQuery
	public Collection<Actor> actorList() {
		return actorRepository.list();
	}

	@DgsQuery
	public Actor findOneActor(@InputArgument Integer id) {
		return actorRepository.getById(id);
	}
}

type Query {
 actorList: [Actor]
 findOneActor(id : Int!): Actor
}

type Actor {
 actorId: Int!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

此处集成和graphql-java-kickstart类似，也是十分简便，不指定别名默认将方法名映射到文件方法名

测试

访问http://localhost:8080/graphiql即可看到在线查询页面

总结

此处还看不出DGS与其他graphql-java组件的区别，都是集成简便，可以和任何数据源、ORM框架集成，后续将会逐步实践DGS特性，包括懒加载、N+1问题、分页组件等

GraphQL实践3——Spring-Graphql集成Mybatis Plus

Thu, 27 Oct 2022 15:37:00 +0800

介绍

考虑到国内大部分场景对接的是Mybatis或者Mybatis Plus生态，而不是JPA，同时Graphql版本也进一步迭代，故计划使用Mybatis Plus对接Graphql

集成过程

引入依赖

由于JDK已经发展到19，目前最新的LTS为17，我所在的公司也将生产升级到了JDK 17，未来更多新应用会直接基于17构建，故此处直接基于JDK17对接
SpringBoot版本为2.7.1，目前已知3.0.0 RC1还不兼容第三方graphql
此处不使用spring官方的starter，改为使用graphql-java-kickstart的starter，编写更加简单


 <dependencies>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 </dependency>

 <dependency>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 <optional>true</optional>
 </dependency>

 <!-- graphql-spring-boot-starter -->
 <dependency>
 <groupId>com.graphql-java-kickstart</groupId>
 <artifactId>graphql-spring-boot-starter</artifactId>
 <version>12.0.0</version>
 </dependency>

 <dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <scope>runtime</scope>
 </dependency>
 <!-- mybatis-plus-boot-starter -->
 <dependency>
 <groupId>com.baomidou</groupId>
 <artifactId>mybatis-plus-boot-starter</artifactId>
 <version>3.5.2</version>
 </dependency>
 <!-- querydsl-core -->
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-core</artifactId>
 </dependency>
 
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-test</artifactId>
 <scope>test</scope>
 </dependency>
 </dependencies>

配置文件

spring:
 datasource:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: root
 password: ****
 url: jdbc:mysql://localhost:3306/sakila?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
server:
 port: 8080
graphql:
 tools:
 schema-location-pattern: graphql/*.graphqls
 graphiql:
 # 开启可交互页面，用于调试
 enabled: true
logging:
 level:
 top.fjy8018.graphsqldemo.mapper: DEBUG

Mybatis Plus配置

扫描配置

@Configuration
@MapperScan("top.fjy8018.graphsqldemo.mapper")
public class MybatisPlusConfig {
}

实体

@Data
public class Actor implements Serializable {

 @Serial
 private static final long serialVersionUID = 1L;

 /**
 * 联系人ID
 */
 @TableId(type = IdType.AUTO)
 private Integer actorId;

 private String firstName;

 private String lastName;

 private Date lastUpdate;

}

Mapper

public interface ActorMapper extends BaseMapper<Actor> {
}

DAO

public interface ActorRepository extends IService<Actor> {
}

@Validated
@Repository
@RequiredArgsConstructor
public class ActorRepositoryImpl extends ServiceImpl<ActorMapper, Actor> implements ActorRepository {
}

Graphql 查询解析器

@Component
@AllArgsConstructor
public class ActorGraphQLQueryResolver implements GraphQLQueryResolver {

 private final ActorRepository actorRepository;

 public Collection<Actor> actorList() {
 return actorRepository.list();
 }

 public Actor findOneActor(Integer id) {
 return actorRepository.getById(id);
 }
}

查询文件定义在resources/graphql/actor.graphqls

type Query {
 actorList: [ActorEntity]
 findOneActor(id : ID!): ActorEntity
}

type ActorEntity {
 actorId: ID!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

测试

访问http://localhost:8080/graphiql即可看到在线查询页面

总结

依托Mybatis Plus强大的自带CRUD方法，可以实现和JPA类似的快速对接

Java 17 生产实践

Fri, 06 May 2022 11:04:00 +0800

背景

版本发布

2021年3月16日 Java 17 作为继Java 11的下一个LTS版本正式发布，JDK 17 作为下一代 LTS 将提供至少到 2026 年的支持。

同业动态

2022年ThoughtWorks 26期技术雷达也提到，建议各个组织将Java 17纳入评估。

评估 82. Java 17

我们通常不会专门介绍某一语言的新版本，但我们还是想关注一下 Java 新的长期支持（LTS）版本——Java 17。尽管出现了前景不错的新特性，比如模式匹配等预览特性，但其实向新 LTS 版本升级的方案更应该吸引各个组织的兴趣。我们建议各个组织在 Java 有新的发布时对其进行评估，确保能够恰当地适配这些新特性和版本。尽管定期更新有利于简化开发并且方便管理，但许多组织却意外地并不经常更新语言的版本。希望通过 LTS 版本的升级以及开发团队对语言的定期更新，能够使生产软件免于因为“更新成本太高”而一直困在 Java 的过时版本上。

框架发展情况

Spring Framework 6和SpringBoot 3.0 将在2022年底正式发布，将基于Java 17构建。

Spring Boot 3.0.0 M1 Release Notes · spring-projects/spring-boot Wiki (github.com)

Minimum Requirements Changes
Spring Boot 3.0 makes the following changes to its minimum supported versions:

Gradle 7.3

Jakarta EE 9

Java 17

Kotlin 1.6

Spring Framework 6

其他团队落地情况

在此之前，我所在的团队使用的Java版本一直停留在Java8，今年经过其他开发团队的尝试，我们团队在4月份也正式决定在生产环境使用Java 17。

实践情况

开发组件版本

本次计划在内部运营管理系统重构中使用Java 17作为基础版本，其他核心组件版本如下

SpringBoot 2.6.6
mapstruct 1.4.2.Final
mybatis-plus 3.5.1
jdk 17.0.1

开发时SpringBoot 3.0还处于M2里程碑阶段，暂时不适合带入生产环境，等未来3.0 GA后再考虑升级

基础镜像

原本使用的基于jdk8 的Docker image也需要替换成基于jdk 17的image

带来的问题：

原本基于jdk8构建并注入到镜像中的opentelemetry-javaagent暂时不可用，等待后续基于jdk17 构建后才能重新启用

JVM参数

由于从Java11开始G1替代CMS作为默认的垃圾收集器，Java14开始移除了CMS GC。

故团队内定义的系统框架配置的GC调优参数基本废弃，并且可以使用Java 14新引入的JVM参数，例如-XX:+ShowCodeDetailsInExceptionMessages

开启前

Exception in thread "main" java.lang.NullPointerException
 at Book.main(Book.java:5)

开启后

Exception in thread "main" java.lang.NullPointerException: 
 Cannot assign field "book" because "shoopingcart.buy" is null
 at Book.main(Book.java:5)

语法特性

从Java8到Java17期间新增的语法特性很多，此处提及几个比较常用的

快速创建不可变集合

不可变对象具有许多优点，包括：

可安全使用不受信任的库。
线程安全。
不需要支持变化判断，所有不可变的集合实现都比其可变的同级实现更节省内存。

Java9开始新增List、Map、Set等of方法快速创建不可变集合

List.of(prodStrategy, instStrategy)

实现源码

 static <E> List<E> of(E e1, E e2) {
 return new ImmutableCollections.List12<>(e1, e2);
 }

Java16开始，Stream类中也新增toList()方法，快速将流转成不可变集合

List<Long> ids = idList.stream().map(ContactId::id).toList();

实现源码

 default List<T> toList() {
 return (List<T>) Collections.unmodifiableList(new ArrayList<>(Arrays.asList(this.toArray())));
 }

注意：如果需要将流转成可变集合，还是需要使用collect(Collectors.toList())方法

record类型

Java14开始新增record类型，用于表示不可变class，实践中可以方便的表示值对象，无需使用@Value注解

public record ContactId(@NotNull(message = "ID不能为空") Long id) {
}

instanceof模式匹配

Java14开始支持模式匹配，无需做强制类型转换

	private void configMessageConverters(List<HttpMessageConverter<?>> converters) {
		converters.removeIf(x -> x instanceof StringHttpMessageConverter || x instanceof MappingJackson2HttpMessageConverter);
		converters.add(new StringHttpMessageConverter(Charsets.UTF_8));
		converters.add(new MappingJackson2HttpMessageConverter(objectMapper));
	}

创建类型安全的流

Java9 开始，新增ofNullable方法创建类型安全的流对象，避免在调用stream方法时报NPE

// query.getDescColumns()可能为null
List<OrderItem> descItems = Stream.ofNullable(query.getDescColumns())
				.flatMap(Collection::stream)
				.map(OrderItem::desc)
				.toList();

性能和安全提升

从同一段功能逻辑中测试发现，Java8和Java17在并发场景下性能差距达到2倍以上

并且同样的逻辑（使用了并发安全的工具）在并发量达到50000的时候在Java8中依然有线程安全问题，而Java17中不会有线程安全问题

外部测评结果

从规划调度引擎 OptaPlanner 项目（原文 (opens new window)）对 JDK 17和 JDK 11 的性能基准测试进行了对比来看：

对于 G1GC（默认），Java 17 比 Java 11 快 8.66%；

对于 ParallelGC，Java 17 比 Java 11 快 6.54%；

Parallel GC 整体比 G1 GC 快 16.39%

简而言之，JDK17 更快，高吞吐量垃圾回收器比低延迟垃圾回收器更快。

GraphQL实践2——Spring-GraphQL集成JPA与MySQL

Mon, 15 Nov 2021 15:38:00 +0800

介绍

上一篇介绍到用第三方库集成GraphQL，目前spring-graphql项目已经出到1.0.0-M3版本，属于内部预览版，此处尝鲜验证

引入依赖

由于SpringBoot 2.6.0还未发布，因此需要引入较多依赖

<dependencies>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 </dependency>
 <dependency>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 <optional>true</optional>
 </dependency>
 <dependency>
 <groupId>org.springframework.experimental</groupId>
 <artifactId>graphql-spring-boot-starter</artifactId>
 <version>1.0.0-M3</version>
 </dependency>
 <!--升级到Spring Boot 2.6后可以移除-->
 <dependency>
 <groupId>org.springframework.data</groupId>
 <artifactId>spring-data-commons</artifactId>
 <version>2.6.0-RC1</version>
 </dependency>
 <dependency>
 <groupId>org.springframework.data</groupId>
 <artifactId>spring-data-jpa</artifactId>
 <version>2.6.0-RC1</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/org.hibernate.javax.persistence/hibernate-jpa-2.1-api -->
 <dependency>
 <groupId>org.hibernate.javax.persistence</groupId>
 <artifactId>hibernate-jpa-2.1-api</artifactId>
 <version>1.0.2.Final</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/com.querydsl/querydsl-jpa -->
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-jpa</artifactId>
 <version>4.4.0</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/com.querydsl/querydsl-apt -->
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-apt</artifactId>
 <version>4.4.0</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/javax.annotation/javax.annotation-api -->
 <dependency>
 <groupId>javax.annotation</groupId>
 <artifactId>javax.annotation-api</artifactId>
 </dependency>
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-core</artifactId>
 </dependency>
 <dependency>
 <groupId>com.querydsl</groupId>
 <artifactId>querydsl-jpa</artifactId>
 </dependency>
 <dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <scope>runtime</scope>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-data-jpa</artifactId>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-test</artifactId>
 <scope>test</scope>
 </dependency>
</dependencies>

引入里程碑仓库

由于还没有GA，所以要引入里程碑仓库

<repositories>
 <repository>
 <id>spring-milestones</id>
 <name>Spring Milestones</name>
 <url>https://repo.spring.io/milestone</url>
 </repository>
 <repository>
 <id>spring-snapshots</id>
 <name>Spring Snapshots</name>
 <url>https://repo.spring.io/snapshot</url>
 <snapshots>
 <enabled>true</enabled>
 </snapshots>
 </repository>
</repositories>

引入插件

由于使用了dsl动态生成，还要引入注解处理器

<build>
 <plugins>
 <plugin>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-maven-plugin</artifactId>
 <configuration>
 <excludes>
 <exclude>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 </exclude>
 </excludes>
 </configuration>
 </plugin>
 <!-- added for QueryDsl-->
 <plugin>
 <groupId>com.mysema.maven</groupId>
 <artifactId>apt-maven-plugin</artifactId>
 <version>1.1.3</version>
 <executions>
 <execution>
 <goals>
 <goal>process</goal>
 </goals>
 <configuration>
 <outputDirectory>target/generated-sources/java</outputDirectory>
 <processor>com.querydsl.apt.jpa.JPAAnnotationProcessor</processor>
 </configuration>
 </execution>
 </executions>
 </plugin>
 </plugins>
</build>

SpringBoot 配置

编写配置文件

spring:
 datasource:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: root
 password: "****"
 url: jdbc:mysql://localhost:3306/sakila?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
 jpa:
 hibernate:
 ddl-auto: none
 show-sql: true
 database-platform: org.hibernate.dialect.MySQL55Dialect
 properties:
 hibernate:
 format_sql: true
 use_sql_comments: true
 graphql:
 path: /graphql
 schema:
 locations: classpath:graphql/
 fileExtensions: .graphqls, .gqls
 printer:
 enabled: true
server:
 port: 8080

实体类

package top.fjy8018.graphsqldemo.entity;

import lombok.Data;

import javax.persistence.*;
import java.sql.Timestamp;
import java.util.Objects;

/**
 * 演员表实体类
 *
 * @author F嘉阳
 * @date 2021/11/5 10:34
 */
@Data
@Entity
@Table(name = "actor", schema = "sakila")
public class ActorEntity {
 @Id
 @Column(name = "actor_id", nullable = false)
 private Integer actorId;

 @Column(name = "first_name", nullable = false, length = 45)
 private String firstName;

 @Column(name = "last_name", nullable = false, length = 45)
 private String lastName;

 @Column(name = "last_update", nullable = false)
 private Date lastUpdate;

}

DAO

DAO直接继承dsl相关处理器，可自动配对增删改查方法，相比第三方库更加简便

package top.fjy8018.graphsqldemo.repository;

import org.springframework.data.querydsl.QuerydslPredicateExecutor;
import org.springframework.data.repository.CrudRepository;
import org.springframework.graphql.data.GraphQlRepository;
import top.fjy8018.graphsqldemo.entity.ActorEntity;

/**
 * 演员表DAO
 *
 * @author F嘉阳
 * @date 2021/11/5 10:35
 */
@GraphQlRepository
public interface ActorEntityRepository extends
 CrudRepository<ActorEntity, Integer>, QuerydslPredicateExecutor<ActorEntity> {
}

Graphql资源定义

在resources/graphql下定义接口和实体文件

接口声明和实体定义schema.graphqls

type Query {
 findOneActor(actorId : ID!): ActorEntity
 actorList: [ActorEntity]
}

type ActorEntity {
 actorId: ID!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

接口定义acotrEntity.graphql

query findOneActor($id: ID!) {
 findOneActor(actorId: $id) {
 actorId
 firstName
 lastName
 lastUpdate
 }
}

query actorList {
 actorList{
 actorId
 }
}

启动测试

Spring-Graphql默认不包含可视化界面，此处使用postman进行测试

唯一查询

列表查询

总结

使用Spring官方组件好处在于和Spring生态集成度很高，如果本身就采用Jpa方式进行业务开发，迁移更加方便，需要开发的代码也很少。

样例源码地址

FJiayang/graphql-demo

GraphQL实践1——集成JPA与MySQL

Fri, 05 Nov 2021 17:41:00 +0800

介绍

官方定义：

GraphQL 既是一种用于 API 的查询语言也是一个满足你数据查询的运行时。 GraphQL 对你的 API 中的数据提供了一套易于理解的完整描述，使得客户端能够准确地获得它需要的数据，而且没有任何冗余，也让 API 更容易地随着时间推移而演进，还能用于构建强大的开发者工具。

中文官网：

GraphQL — A query language for your API

集成方案选择

GraphQL经过近几年的发展，SpringBoot官方、第三方都提供了starter

SpringBoot官方提供的starter

Spring GraphQL

第三方提供的starter

graphql-spring-boot

两者活跃度都还不错，但考虑到SpringBoot官方的依赖库还未GA，而且使用起来与Spring家族耦合过大，最终决定采用第三方的starter

集成过程

数据库配置

此处采用MySQL数据库，数据内容采用MySQL官方样例sakila数据，表结构和数据地址如下

sakila database下载地址

导入后如下

工程配置

引入最新的SpringBoot框架，依赖如下

<dependencies>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 </dependency>

 <dependency>
 <groupId>org.projectlombok</groupId>
 <artifactId>lombok</artifactId>
 <optional>true</optional>
 </dependency>

 <!-- https://mvnrepository.com/artifact/com.graphql-java-kickstart/graphql-spring-boot-starter -->
 <dependency>
 <groupId>com.graphql-java-kickstart</groupId>
 <artifactId>graphql-spring-boot-starter</artifactId>
 <version>12.0.0</version>
 </dependency>

 <dependency>
 <groupId>mysql</groupId>
 <artifactId>mysql-connector-java</artifactId>
 <scope>runtime</scope>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-data-jpa</artifactId>
 </dependency>
 <dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-test</artifactId>
 <scope>test</scope>
 </dependency>
</dependencies>

增加数据库配置

spring:
 datasource:
 driver-class-name: com.mysql.cj.jdbc.Driver
 username: root
 password: "****"
 url: jdbc:mysql://localhost:3306/sakila?useSSL=false&useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
 jpa:
 hibernate:
 ddl-auto: update
 show-sql: true
 database-platform: org.hibernate.dialect.MySQL55Dialect
 properties:
 hibernate:
 format_sql: true
 use_sql_comments: true
server:
 port: 8080
graphql:
 tools:
 schema-location-pattern: graphql/*.graphqls
 graphiql:
 # 开启可交互页面，用于调试
 enabled: true

实体映射

新增实体映射，此处只映射Actor表作为样例

package top.fjy8018.graphsqldemo.entity;

import lombok.Data;

import javax.persistence.*;
import java.sql.Timestamp;
import java.util.Objects;

/**
 * 演员表实体映射
 *
 * @author F嘉阳
 * @date 2021/11/5 10:34
 */
@Data
@Entity
@Table(name = "actor", schema = "sakila")
public class ActorEntity {
 @Id
 @Column(name = "actor_id", nullable = false)
 private Integer actorId;

 @Column(name = "first_name", nullable = false, length = 45)
 private String firstName;

 @Column(name = "last_name", nullable = false, length = 45)
 private String lastName;

 @Column(name = "last_update", nullable = false)
 private Timestamp lastUpdate;
}

新增DAO

package top.fjy8018.graphsqldemo.repository;

import org.springframework.data.jpa.repository.JpaRepository;
import top.fjy8018.graphsqldemo.entity.ActorEntity;

/**
 * 演员表DAO
 *
 * @author F嘉阳
 * @date 2021/11/5 10:35
 */
public interface ActorEntityRepository extends JpaRepository<ActorEntity, Integer> {
}

新增查询接口配置

在resources/graphql下新增文件actor.graphqls

type Query {
 actorList: [ActorEntity]
 findOneActor(id : ID!): ActorEntity
}

type ActorEntity {
 actorId: ID!
 firstName: String!
 lastName: String!
 lastUpdate: String
}

新增查询解析器

package top.fjy8018.graphsqldemo.resolver;

import graphql.kickstart.tools.GraphQLQueryResolver;
import lombok.AllArgsConstructor;
import org.springframework.stereotype.Component;
import top.fjy8018.graphsqldemo.entity.ActorEntity;
import top.fjy8018.graphsqldemo.repository.ActorEntityRepository;

import java.util.Collection;

/**
 * 演员表GraphQL查询解析器
 *
 * @author F嘉阳
 * @date 2021/11/5 16:27
 */
@Component
@AllArgsConstructor
public class ActorGraphQLQueryResolver implements GraphQLQueryResolver {

 private final ActorEntityRepository actorRepository;

 public Collection<ActorEntity> actorList() {
 return actorRepository.findAll();
 }

 public ActorEntity findOneActor(Integer id) {
 return actorRepository.findById(id).orElse(null);
 }
}

启动验证

启动工程后，浏览器打开http://localhost:8080/graphiql

列表查询

由于采用交互式页面，输入也会有语法提示，比较方便

唯一查询

优缺点分析

优点

接口定义清晰，配置即接口
前端按需获取字段，减少数据传输量
接口整合、演进更加便利

缺点

减少字段个数对后端实现没有变化，并不能提升数据库性能

样例源码地址

FJiayang/graphql-demo

Kubesphere落地DevOps

Sat, 19 Jun 2021 22:16:00 +0800

创建企业空间

创建一个企业空间和一个帐户 (project-admin)，必须邀请该帐户至该企业空间并赋予 workspace-self-provisioner 角色。

进入企业空间，切换到DevOps工程，点击创建

创建工程

安装helm

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

找不到环境变量报错

安装helm遇到报错

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
 % Total % Received % Xferd Average Speed Time Time Time Current
unset -f pathmunge
 Dload Upload Total Spent Left Speed
100 11248 100 11248 0 0 61543 0 --:--:-- --:--:-- --:--:-- 61802
Helm v3.6.0 is available. Changing from version v3.2.1.
Downloading https://get.helm.sh/helm-v3.6.0-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
helm not found. Is /usr/local/bin on your $PATH?
Failed to install helm
 For support, go to https://github.com/helm/helm.

则需要配置环境变量

$ vi /etc/profile
# 添加配置
export PATH=/usr/local/bin:$PATH

$ source /etc/profile

验证

$ helm version
version.BuildInfo{Version:"v3.6.0", GitCommit:"7f2df6467771a75f5646b7f12afb408590ed1755", GitTreeState:"clean", GoVersion:"go1.16.3"}

安装SonarQube

执行以下命令安装 SonarQube 服务器

$ helm upgrade --install sonarqube sonarqube --repo https://charts.kubesphere.io/main -n kubesphere-devops-system --create-namespace --set service.type=NodePort

输出

执行以下命令以获取 SonarQube NodePort

$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
$ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo http://$NODE_IP:$NODE_PORT

输出

此时可在kubesphere控制台中看到sonar服务

配置 SonarQube 服务器

访问地址，点击右上角的 Log in，然后使用默认帐户 admin/admin 登录。

创建 SonarQube 管理员令牌 (Token)

点击右上角字母 A，然后从菜单中选择 My Account 以转到 Profile 页面。

生成token

创建 Webhook 服务器

执行以下命令获取 SonarQube Webhook 的地址。

$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services ks-jenkins)
$ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo http://$NODE_IP:$NODE_PORT/sonarqube-webhook/

输出

依次点击 Administration、Configuration 和 Webhooks 创建一个 Webhook。

点击 Create

由于webhook都通过内部网络调用，故可以使用内部网络地址

将 SonarQube 配置添加到 ks-installer

执行以下命令编辑 ks-installer

$ kubectl edit cc -n kubesphere-system ks-installer

搜寻至 devops。添加字段 sonarqube 并在其下方指定 externalSonarUrl 和 externalSonarToken。

 devops:
 enabled: true
 jenkinsJavaOpts_MaxRAM: 2g
 jenkinsJavaOpts_Xms: 512m
 jenkinsJavaOpts_Xmx: 512m
 jenkinsMemoryLim: 2Gi
 jenkinsMemoryReq: 1500Mi
 jenkinsVolumeSize: 8Gi
 sonarqube:
 externalSonarUrl: http://10.170.0.5:32108
 externalSonarToken: c6a6e2e15a9d52e30b744df0eef4c639bbea88a1

完成操作后保存此文件。

将 SonarQube 服务器添加至 Jenkins

执行以下命令获取 Jenkins 的地址。

$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services ks-jenkins)
$ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo http://$NODE_IP:$NODE_PORT

输出

使用地址 http://{$Public IP}:30180 访问 Jenkins。安装 KubeSphere 时，默认情况下也会安装 Jenkins 仪表板。此外，Jenkins 还配置有 KubeSphere LDAP，这意味着您可以直接使用 KubeSphere 帐户（例如 admin/P@88w0rd）登录 Jenkins。

使用kubesphere admin账户登录

前往系统管理下的 Manage Credentials 并点击 Stores scoped to Jenkins 下的 Jenkins，再点击全局凭据 (unrestricted)

点击左侧导航栏的添加凭据，用 SonarQube 管理员令牌添加凭证

点击左侧的系统管理，向下翻页找到并点击系统配置。

搜寻到 SonarQube servers，然后点击 Add SonarQube。

添加sonar服务器地址和名称，点击应用完成操作。

将 sonarqubeURL 添加到 KubeSphere 控制台

指定 sonarqubeURL，以便可以直接从 KubeSphere 控制台访问 SonarQube。

执行以下命令：

$ kubectl edit cm -n kubesphere-system ks-console-config

搜寻到 client，添加 devops 字段并指定 sonarqubeURL。

 client:
 version:
 kubesphere: v3.1.0
 kubernetes: v1.20.4
 openpitrix: v0.3.5
 enableKubeConfig: true
 devops:
 sonarqubeURL: http://34.92.90.181:32108

结果

重启服务

执行以下命令。

$ kubectl -n kubesphere-system rollout restart deploy ks-apiserver
$ kubectl -n kubesphere-system rollout restart deploy ks-console

为新工程创建 SonarQube Token

创建一个 SonarQube 令牌，以便您的流水线可以在运行时与 SonarQube 通信。

在 SonarQube 控制台上，点击 Create new project。

设置工程名称

生成token

创建令牌后，点击 Continue。

分别选择 Java 和 Maven。复制下图所示绿色框中的序列号，如果要在流水线中使用，则需要在凭证中添加此序列号。

创建流水线工程

创建凭证

凭证 ID	类型	用途
dockerhub-id	帐户凭证	Docker Hub
demo-kubeconfig	kubeconfig	Kubernetes

demo-kubeconfig

dockerhub-id

创建一个凭证 ID (sonar-token)

列表中看到已创建的三个凭证。

创建流水线

在流水线页面点击创建，为了使用图形化流水线搭建，不在此处引用代码仓库，直接点击下一步

保持默认配置，点击创建

创建成功

进入流水线，点击编辑

此时选择CICD模板

可视化编辑流水线

编辑后运行

流水线支持交互式操作

在 KubeSphere 控制台查看结果

流水线执行成功后，即可查看到代码质量标签

sonar控制台也可以查看对应结果

jenkins视图

blue ocean

Kubesphere安装——以GCP为例

Sat, 19 Jun 2021 22:15:00 +0800

平台准备

创建工作空间

启用计算引擎API

准备虚拟机

作为演示环境，此处使用Google Cloud Platform创建3台虚拟机，配置如下

CPU	内存	磁盘
4	16G	40G
4	16G	40G
4	16G	40G

创建实例模板

创建需要的实例模板

配置操作系统、资源等，注意放行防火墙

创建实例

基于实例模板创建3台虚拟机，选择地区为香港

此处用于演示用途，可选择无状态机器，机器删除后不会保留磁盘内容

注意关闭自动扩缩容

等待实例创建

由于资源调度问题，最终只成功分配两台虚拟机，不过对于演示环境安装过程一样。

配置连接秘钥

添加本地生成好的rsa秘钥即可

本地SSH工具配置私钥即可连接

前置环境准备

kubesphere建议前置项

节点要求

所有节点必须都能通过 SSH 访问。

所有节点时间同步。

所有节点都应使用 sudo/curl/openssl。

依赖项要求
KubeKey 可以一同安装 Kubernetes 和 KubeSphere。根据要安装的 Kubernetes 版本，需要安装的依赖项可能会不同。您可以参考下表，查看是否需要提前在节点上安装相关依赖项。

依赖项 Kubernetes 版本 ≥ 1.18 Kubernetes 版本 < 1.18

socat 必须可选，但建议安装

conntrack 必须可选，但建议安装

ebtables 可选，但建议安装可选，但建议安装

ipset 可选，但建议安装可选，但建议安装

依赖项	Kubernetes 版本 ≥ 1.18	Kubernetes 版本 < 1.18
`socat`	必须	可选，但建议安装
`conntrack`	必须	可选，但建议安装
`ebtables`	可选，但建议安装	可选，但建议安装
`ipset`	可选，但建议安装	可选，但建议安装

以下内容所有节点都需要执行

$ yum install -y socat curl openssl conntrack ebtables ipset

主节点下载kubekey，由于使用google虚拟机，能正常访问google和github，故直接下载即可

$ curl -sfL https://get-kk.kubesphere.io | VERSION=v1.1.0 sh -
$ chmod +x kk

放行VPC防火墙

创建集群

服务器免密登陆

主节点生成免密登陆配置

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:yxln4RFEXhsvIMcL0/EzF8CLUpo/hMlkM8KBDcNEjbU root@kubesphere-instance-group-1-bx5d
The key's randomart image is:
+---[RSA 2048]----+
| +=Oo..=Bo+.. |
| +.=.B++=.+ . |
| E= @+o*.o |
| B.+o.= |
| S++ |
| . *o |
| + . |
| |
| |
+----[SHA256]-----+

所有主机允许公钥免密登陆

$ vi /etc/ssh/sshd_config
# 将下面配置改为yes并去掉注释
PubkeyAuthentication yes
PermitRootLogin yes

$ service sshd restart

控制台添加主节点公钥，则所有新建主机都能被主节点访问

创建示例配置文件

$./kk create config --with-kubernetes v1.20.4 --with-kubesphere v3.1.0 -f gcp-config

按默认配置只安装控制台和基础服务

apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
 name: sample
spec:
 hosts:
 - {name: node1, address: 34.92.90.181, internalAddress: 10.170.0.5, privateKeyPath: "/root/.ssh/id_rsa"}
 - {name: node2, address: 35.220.138.230, internalAddress: 10.170.0.6, privateKeyPath: "/root/.ssh/id_rsa"}
 roleGroups:
 etcd:
 - node1
 master: 
 - node1
 worker:
 - node1
 - node2
 controlPlaneEndpoint:
 domain: lb.kubesphere.local
 address: ""
 port: 6443
 kubernetes:
 version: v1.20.4
 imageRepo: kubesphere
 clusterName: cluster.local
 network:
 plugin: calico
 kubePodsCIDR: 10.233.64.0/18
 kubeServiceCIDR: 10.233.0.0/18
 registry:
 registryMirrors: []
 insecureRegistries: []
 addons: []


---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
 name: ks-installer
 namespace: kubesphere-system
 labels:
 version: v3.1.0
spec:
 persistence:
 storageClass: "" 
 authentication:
 jwtSecret: ""
 zone: ""
 local_registry: "" 
 etcd:
 monitoring: false 
 endpointIps: localhost 
 port: 2379 
 tlsEnable: true
 common:
 redis:
 enabled: false
 redisVolumSize: 2Gi 
 openldap:
 enabled: false
 openldapVolumeSize: 2Gi 
 minioVolumeSize: 20Gi
 monitoring:
 endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
 es: 
 elasticsearchMasterVolumeSize: 4Gi 
 elasticsearchDataVolumeSize: 20Gi 
 logMaxAge: 7 
 elkPrefix: logstash
 basicAuth:
 enabled: false
 username: ""
 password: ""
 externalElasticsearchUrl: ""
 externalElasticsearchPort: "" 
 console:
 enableMultiLogin: true 
 port: 30880
 alerting: 
 enabled: false
 # thanosruler:
 # replicas: 1
 # resources: {}
 auditing: 
 enabled: false
 devops: 
 enabled: false
 jenkinsMemoryLim: 2Gi 
 jenkinsMemoryReq: 1500Mi 
 jenkinsVolumeSize: 8Gi 
 jenkinsJavaOpts_Xms: 512m 
 jenkinsJavaOpts_Xmx: 512m
 jenkinsJavaOpts_MaxRAM: 2g
 events: 
 enabled: false
 ruler:
 enabled: true
 replicas: 2
 logging: 
 enabled: false
 logsidecar:
 enabled: true
 replicas: 2
 metrics_server: 
 enabled: false
 monitoring:
 storageClass: ""
 prometheusMemoryRequest: 400Mi 
 prometheusVolumeSize: 20Gi 
 multicluster:
 clusterRole: none 
 network:
 networkpolicy:
 enabled: false
 ippool:
 type: none
 topology:
 type: none
 notification: 
 enabled: false
 openpitrix:
 store:
 enabled: false
 servicemesh: 
 enabled: false 
 kubeedge:
 enabled: false
 cloudCore:
 nodeSelector: {"node-role.kubernetes.io/worker": ""}
 tolerations: []
 cloudhubPort: "10000"
 cloudhubQuicPort: "10001"
 cloudhubHttpsPort: "10002"
 cloudstreamPort: "10003"
 tunnelPort: "10004"
 cloudHub:
 advertiseAddress: 
 - "" 
 nodeLimit: "100"
 service:
 cloudhubNodePort: "30000"
 cloudhubQuicNodePort: "30001"
 cloudhubHttpsNodePort: "30002"
 cloudstreamNodePort: "30003"
 tunnelNodePort: "30004"
 edgeWatcher:
 nodeSelector: {"node-role.kubernetes.io/worker": ""}
 tolerations: []
 edgeWatcherAgent:
 nodeSelector: {"node-role.kubernetes.io/worker": ""}
 tolerations: []

安装集群

此时，在虚拟机安装代理即可查看虚拟机监控

创建完成

访问成功

集群正常，不安装组件时内存占用率并不高

安装kubectl

$ yum install -y kubectl

安装可插拔组件

点击自定义资源 CRD，然后在搜索栏中输入 clusterconfiguration，点击搜索结果进入其详情页面。

按照自身需要启用可插拔组件，此处打开所有组件，即修改为如下配置

apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
 labels:
 version: v3.1.0
 name: ks-installer
 namespace: kubesphere-system
spec:
 alerting:
 enabled: true
 auditing:
 enabled: true
 authentication:
 jwtSecret: ''
 common:
 es:
 basicAuth:
 enabled: false
 password: ''
 username: ''
 elasticsearchDataVolumeSize: 20Gi
 elasticsearchMasterVolumeSize: 4Gi
 elkPrefix: logstash
 externalElasticsearchPort: ''
 externalElasticsearchUrl: ''
 logMaxAge: 7
 minioVolumeSize: 2Gi
 monitoring:
 endpoint: 'http://prometheus-operated.kubesphere-monitoring-system.svc:9090'
 openldap:
 enabled: true
 openldapVolumeSize: 2Gi
 redis:
 enabled: true
 redisVolumSize: 2Gi
 console:
 enableMultiLogin: true
 port: 30880
 devops:
 enabled: true
 jenkinsJavaOpts_MaxRAM: 2g
 jenkinsJavaOpts_Xms: 512m
 jenkinsJavaOpts_Xmx: 512m
 jenkinsMemoryLim: 2Gi
 jenkinsMemoryReq: 1500Mi
 jenkinsVolumeSize: 8Gi
 etcd:
 endpointIps: 10.170.0.5
 monitoring: true
 port: 2379
 tlsEnable: true
 events:
 enabled: true
 ruler:
 enabled: true
 replicas: 2
 kubeedge:
 cloudCore:
 cloudHub:
 advertiseAddress:
 - ''
 nodeLimit: '100'
 cloudhubHttpsPort: '10002'
 cloudhubPort: '10000'
 cloudhubQuicPort: '10001'
 cloudstreamPort: '10003'
 nodeSelector:
 node-role.kubernetes.io/worker: ''
 service:
 cloudhubHttpsNodePort: '30002'
 cloudhubNodePort: '30000'
 cloudhubQuicNodePort: '30001'
 cloudstreamNodePort: '30003'
 tunnelNodePort: '30004'
 tolerations: []
 tunnelPort: '10004'
 edgeWatcher:
 edgeWatcherAgent:
 nodeSelector:
 node-role.kubernetes.io/worker: ''
 tolerations: []
 nodeSelector:
 node-role.kubernetes.io/worker: ''
 tolerations: []
 enabled: true
 logging:
 enabled: true
 logsidecar:
 enabled: true
 replicas: 2
 metrics_server:
 enabled: true
 monitoring:
 prometheusMemoryRequest: 400Mi
 prometheusVolumeSize: 5Gi
 storageClass: ''
 multicluster:
 clusterRole: none
 network:
 ippool:
 type: calico
 networkpolicy:
 enabled: true
 topology:
 type: weave-scope
 openpitrix:
 store:
 enabled: true
 persistence:
 storageClass: ''
 servicemesh:
 enabled: true

通过在线控制台，查看安装进度

执行以下命令

$ kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

安装组件过程中，对CPU负载较大

所有组件安装完成

资源占用情况

Kubesphere运维Kubernetes集群

Sun, 23 May 2021 22:58:00 +0800

背景

使用原生Kubernetes-dashboard和Grafana+Prometheus管理较大规模集群比较困难，有如下缺陷

每次执行命令都要登陆服务器
容器日志采集方案、监控方案搭建比较繁琐
添加节点不方便
管理存储插件、秘钥、配置不方便

解决方案

目前国内流行的k8s管理平台主要有rancher、kubesphere、kubeboard

rancher主打原生集群管理，侧重运维人员
kubesphere主打开发运维工作流，侧重研发人员
kubeboard与kubesphere类似，侧重微服务管理，但功能上较少，对多节点管理需要收费

经过评估，最终决定采用kubesphere进行集群管理

Rancher — 多云混合云多集群Kubernetes管理平台

[post url=“https://kubesphere.io/zh/” title=“Kubesphere” intro=“面向云原生应用的容器混合云” cover=“https://kubesphere.io/images/logo.svg” /]

Kuboard — 快速在 Kubernetes 上落地微服务

安装步骤

目标是面对小规模集群管理，节点数最多为5个，故不考虑高可用集群配置，采用单主节点+4个worker节点配置

采用纯净CentOS 7 Linux系统，需要做以下前置准备

依赖包安装

安装官方要求的依赖包，由于计划安装NFS存储插件，还需要安装nfs工具包

$ yum install -y socat conntrack ebtables ipset nfs-utils

配置DNS解析

将各节点DNS配置为私有DNS服务器

$ tee /etc/resolv.conf <<-'EOF'
# Generated by NetworkManager
nameserver 192.168.1.13
EOF

私有DNS搭建可以参考文章

[[利用Docker自建Dns服务器]]

按照官方教程安装多节点kubernetes

强烈建议先用KubeKey安装纯净k8s集群+控制面板，关闭所有插件。等安装完毕再一个个插件打开，防止因为内存不够而无法打开kubesphere

安装步骤参考官方文档一键安装方式

多节点安装 — Kubesphere

启动成功

启动成功后按照官方指南打开插件

启用可插拔组件 — Kubesphere

所有插件打开后如下

使用

节点状态

存储卷管理

监控告警

日志采集

日志查询

操作审计

工具箱

企业空间权限控制

内置helm应用商店，支持上线审批，在线编辑等

支持关联外部仓库

应用流量治理

链路追踪

灰度发布

费用统计

更多的使用场景包括：DevOps、边缘节点、多集群管理还在探索中

利用Docker自建DNS服务器

Sun, 09 May 2021 18:14:12 +0800

背景

当Linux集群中机器不断增多，集群内通过配置/hosts实现域名内网访问。

目标

相同域名在集群之外通过公有DNS访问外网地址
相同域名在集群内通过DNS解析访问内网地址

步骤

搭建DNS服务器

DNS服务器可以通过jpillora/dnsmasq容器实现

jpillora/dnsmasq — DockerHub

新增配置文件 dnsmasq.conf

# 完整配置文件参考以下地址
# http://oss.segetech.com/intra/srv/dnsmasq.conf
# 记录所有DNS查询请求
log-queries
# 不使用本地配置的域名解析器
no-resolv
# 不使用本地hosts
no-hosts
# 配置上游DNS服务器地址，用于解析国内域名
server=114.114.114.114
# 配置国际DNS服务器地址
server=8.8.8.8
strict-order
# 配置内网解析
address=/gitlab.fjy8018.top/192.168.1.11

配置docker-compose文件，DNS服务器默认通过53 udp端口解析，不建议修改53端口的外部地址

version: "3.5"

services:

 dnsmasq:
 image: jpillora/dnsmasq:1.1.0
 networks:
 - dns-net
 ports:
 - 53:53/udp
 - 5380:8080
 volumes:
 - ./dnsmasq.conf:/etc/dnsmasq.conf
 environment:
 - "HTTP_USER=admin"
 - "HTTP_PASS=admin"
 restart: always

networks:
 dns-net:

启动服务即可

后续需要修改DNS解析只需要访问5380的web UI界面即可动态调整

服务器指向新DNS服务器

将所有服务器/etc/resolv.conf文件指向新DNS服务器地址

$ tee /etc/resolv.conf <<-'EOF'
# Generated by NetworkManager
nameserver 192.168.1.13
EOF

补充

对于kubernetes集群，由于集群内使用了coredns和nodelocaldns缓存服务，还需要重启两个dns服务组件

$ kubectl -n kube-system rollout restart daemonset nodelocaldns
$ kubectl -n kube-system rollout restart deploy coredns

SpringCloudStream函数式编程模型

Sun, 21 Feb 2021 12:22:00 +0800

背景

SpringCloud 3.1版本之后，@EnableBinding、@Output等StreamApi注解都标记为废弃

/**
 * Indicates that an output binding target will be created by the framework.
 *
 * @author Dave Syer
 * @author Marius Bogoevici
 * @author Artem Bilan
 *
 * @deprecated as of 3.1 in favor of functional programming model
 */

@Qualifier
@Target({ ElementType.FIELD, ElementType.METHOD, ElementType.ANNOTATION_TYPE,
		ElementType.PARAMETER })
@Retention(RetentionPolicy.RUNTIME)
@Inherited
@Documented
@Deprecated
public @interface Output {

	/**
	 * Specify the binding target name; used as a bean name for binding target and as a
	 * destination name by default.
	 * @return the binding target name
	 */
	String value() default "";

}

/**
 * Enables the binding of targets annotated with {@link Input} and {@link Output} to a
 * broker, according to the list of interfaces passed as value to the annotation.
 *
 * @author Dave Syer
 * @author Marius Bogoevici
 * @author David Turanski
 * @author Soby Chacko
 *
 * @deprecated as of 3.1 in favor of functional programming model
 */
@Target({ ElementType.TYPE, ElementType.ANNOTATION_TYPE })
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@Configuration
@Import({ BindingBeansRegistrar.class, BinderFactoryAutoConfiguration.class })
@EnableIntegration
@Deprecated
public @interface EnableBinding {

	/**
	 * A list of interfaces having methods annotated with {@link Input} and/or
	 * {@link Output} to indicate binding targets.
	 * @return list of interfaces
	 */
	Class<?>[] value() default {};

}

Producing and Consuming Messages — SpringCloud 官方文档

@EnableBinding @deprecated as of 3.1 in favor of functional programming model — stackoverflow

The type EnableBinding is deprecated — stackoverflow

因此，所有采用命令式编程的API，在升级到新版本时，官方推荐使用更加简单的函数式编程API

完整代码样例

FJiayang/spring-cloud-stream-rabbit-example — GitHub项目地址

升级方式

命令式编程模型

以简单多Topic消息为例，原本的编程方式如下

生产者

@SpringBootApplication
public class ProducerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ProducerApplication.class, args);
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 17:57
 */
@RestController
@EnableBinding(MySource.class)
public class Producer {

 @Autowired
 private MySource channel;

 @RequestMapping("/send")
 public String send() {
 channel.output().send(MessageBuilder.withPayload(new Date()).build());
 return "success";
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 18:01
 */
public interface MySource {
 String OUTPUT = "output";

 @Output(MySource.OUTPUT)
 MessageChannel output();
}

配置文件

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin
 cloud:
 stream:
 bindings:
 output:
 destination: my-test-channel
server:
 port: 8082

消费者

@SpringBootApplication
public class ConsumerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ConsumerApplication.class, args);
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 18:10
 */
@EnableBinding(MySink.class)
public class Consumer {
 @StreamListener(MySink.INPUT)
 public void receive(Message<String> message) {
 System.out.println("接收到MQ消息:" + message.getPayload());
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 18:07
 */
public interface MySink {
 String INPUT = "input";

 @Input(MySink.INPUT)
 SubscribableChannel input();
}

配置文件

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin
 cloud:
 stream:
 bindings:
 input:
 destination: my-test-channel
server:
 port: 8081

函数式编程模型

从命令式升级到函数式编程，首先需要升级SpringCloud版本至3.1以上，此处升级到最新版2020.0.1

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>

 <groupId>top.fjy8018</groupId>
 <artifactId>cloud-stream</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>pom</packaging>

 <name>cloud-stream</name>
 <description>Demo project for Spring Boot</description>

 <modules>
 <module>producer</module>
 <module>consumer</module>
 </modules>

 <parent>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-parent</artifactId>
 <version>2.4.2</version>
 <relativePath/> <!-- lookup parent from repository -->
 </parent>

 <properties>
 <spring-cloud.version>2020.0.1</spring-cloud.version>
 </properties>

 <dependencyManagement>
 <dependencies>
 <dependency>
 <groupId>org.springframework.cloud</groupId>
 <artifactId>spring-cloud-dependencies</artifactId>
 <version>${spring-cloud.version}</version>
 <type>pom</type>
 <scope>import</scope>
 </dependency>
 </dependencies>
 </dependencyManagement>
</project>

目前国内有清晰的升级指引的还比较少，目前只能参考官方文档材料

Programming Model — SpringCloud官方文档

模型1——系统定时驱动的消息队列

官方文档描述

Suppliers (Sources)
Function and Consumer are pretty straightforward when it comes to how their invocation is triggered. They are triggered based on data (events) sent to the destination they are bound to. In other words, they are classic event-driven components.

However, Supplier is in its own category when it comes to triggering. Since it is, by definition, the source (the origin) of the data, it does not subscribe to any in-bound destination and, therefore, has to be triggered by some other mechanism(s). There is also a question of Supplier implementation, which could be imperative or reactive and which directly relates to the triggering of such suppliers.

Consider the following sample:
@SpringBootApplication
public static class SupplierConfiguration {

	@Bean
	public Supplier<String> stringSupplier() {
		return () -> "Hello from Supplier";
	}
}
The preceding Supplier bean produces a string whenever its get() method is invoked. However, who invokes this method and how often? The framework provides a default polling mechanism (answering the question of “Who?”) that will trigger the invocation of the supplier and by default it will do so every second (answering the question of “How often?”). In other words, the above configuration produces a single message every second and each message is sent to an output destination that is exposed by the binder.

其大意为，若采用默认的函数式编程模型，会每秒钟发送一条消息到消费者，定时发送参数可以配置

按照文档升级后如下

生产者

@SpringBootApplication
public class ProducerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ProducerApplication.class, args);
 }

 @Bean
 public Supplier<Date> source1() {
 return () -> new Date();
 }
}

函数式编程不需要在代码中定义监听队列，只需要按照约定写在配置文件中即可

官方文档描述

Functional binding names
Unlike the explicit naming required by annotation-based support (legacy) used in the previous versions of spring-cloud-stream, the functional programming model defaults to a simple convention when it comes to binding names, thus greatly simplifying application configuration. Let’s look at the first example:
@SpringBootApplication
public class SampleApplication {

	@Bean
	public Function<String, String> uppercase() {
	 return value -> value.toUpperCase();
	}
}
In the preceding example we have an application with a single function which acts as message handler. As a Function it has an input and output. The naming convention used to name input and output bindings is as follows:

input - <functionName> + -in- + <index>

output - <functionName> + -out- + <index>

The in and out corresponds to the type of binding (such as input or output). The index is the index of the input or output binding. It is always 0 for typical single input/output function, so it’s only relevant for Functions with multiple input and output arguments.

So if for example you would want to map the input of this function to a remote destination (e.g., topic, queue etc) called “my-topic” you would do so with the following property:
--spring.cloud.stream.bindings.uppercase-in-0.destination=my-topic
Note how uppercase-in-0 is used as a segment in property name. The same goes for uppercase-out-0.

配置文件

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin

 cloud:
 stream:
 bindings:
 source1-out-0:
 destination: test1
 function:
 definition: source1

server:
 port: 8083

由此可见，整体配置简化了很多，多为规约大于配置的方式进行定义，特别注意function配置的位置，是在stream配置的同级

消费者

@SpringBootApplication
public class ConsumerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ConsumerApplication.class, args);
 }

 @Bean
 public Consumer<Date> sink1() {
 return System.out::println;
 }
}

配置文件同理

spring:
 rabbitmq:
 host: 192.168.163.128
 username: cms
 password: cms-mq-admin

 cloud:
 stream:
 bindings:
 sink1-in-0:
 destination: test1
 function:
 definition: sink1


server:
 port: 8081

运行结果

生产者每隔一秒发送一次消息

模型2——`StreamBridge`

但实际生产中，更多是由业务场景触发，故不能使用该模式进行配置，SpringCloud还提供了另外一种StreamBridge模式

官方文档介绍

Sending arbitrary data to an output (e.g. Foreign event-driven sources)
There are cases where the actual source of data may be coming from the external (foreign) system that is not a binder. For example, the source of the data may be a classic REST endpoint. How do we bridge such source with the functional mechanism used by spring-cloud-stream?

Spring Cloud Stream provides two mechanisms, so let’s look at them in more details

Here, for both samples we’ll use a standard MVC endpoint method called delegateToSupplier bound to the root web context, delegating incoming requests to stream via two different mechanisms - imperative (via StreamBridge) and reactive (via EmitterProcessor).

大意是，实际的数据源可以是外部事件驱动，例如通过Rest接口

官方样例

Using StreamBridge
@SpringBootApplication
@Controller
public class WebSourceApplication {

	public static void main(String[] args) {
		SpringApplication.run(WebSourceApplication.class, "--spring.cloud.stream.source=toStream");
	}

	@Autowired
	private StreamBridge streamBridge;

	@RequestMapping
	@ResponseStatus(HttpStatus.ACCEPTED)
	public void delegateToSupplier(@RequestBody String body) {
		System.out.println("Sending " + body);
		streamBridge.send("toStream-out-0", body);
	}
}
Here we autowire a StreamBridge bean which allows us to send data to an output binding effectively bridging non-stream application with spring-cloud-stream. Note that preceding example does not have any source functions defined (e.g., Supplier bean) leaving the framework with no trigger to create source bindings, which would be typical for cases where configuration contains function beans. So to trigger the creation of source binding we use spring.cloud.stream.source property where you can declare the name of your sources. The provided name will be used as a trigger to create a source binding. So in the preceding example the name of the output binding will be toStream-out-0 which is consistent with the binding naming convention used by functions (see Binding and Binding names). You can use ; to signify multiple sources (e.g., --spring.cloud.stream.source=foo;bar)

参考官方文档改造后

生产者

这里直接贴出多主题的消费模式，实际生产中多数也是订阅多个主题

@SpringBootApplication
public class ProducerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ProducerApplication.class, args);
 }
}

/**
 * @author F嘉阳
 * @date 2018-10-08 17:57
 */
@RestController
public class Producer {

 @Autowired
 private StreamBridge streamBridge;

 @RequestMapping("/send1")
 public String send1() {
 streamBridge.send("source1-out-0", new Date());
 return "success1";
 }

 @RequestMapping("/send2")
 public String send2() {
 streamBridge.send("source2-out-0", LocalDateTime.now().format(DateTimeFormatter.ISO_LOCAL_DATE_TIME));
 return "success2";
 }

}

配置文件

spring:
 rabbitmq:
 host: 192.168.133.128
 username: dev-user
 password: devpassword

 cloud:
 stream:
 bindings:
 source1-out-0:
 destination: test2
 source2-out-0:
 destination: test3
 function:
 definition: source1;source2


server:
 port: 8083

消费者

@SpringBootApplication
public class ConsumerApplication {

 public static void main(String[] args) {
 SpringApplication.run(ConsumerApplication.class, args);
 }

 @Bean
 public Consumer<Date> sink1() {
 return System.out::println;
 }

 @Bean
 public Consumer<String> sink2() {
 return System.out::println;
 }
}

配置文件

spring:
 rabbitmq:
 host: 192.168.133.128
 username: dev-user
 password: devpassword

 cloud:
 stream:
 bindings:
 sink1-in-0:
 destination: test2
 sink2-in-0:
 destination: test3
 function:
 definition: sink1;sink2

server:
 port: 8081

运行结果

触发/send1

触发/send2

可见消息被正确的路由

总结

整体来看，函数式编程比命令式编程更加简洁，而且加上SpringCloud规约大于配置的特点，可以明显减少配置的代码量，是未来发展的趋势。

Mapstruct爬坑记录

Sat, 20 Feb 2021 22:45:11 +0800

背景

众所周知，Spring框架提供了著名的类拷贝工具BeanUtils，其使用简单、方便。

但是其内部实现基于运行时反射进行同名属性拷贝，对于需要进行字段映射和高性能运行时场景都无法满足要求，因此，开源社区衍生出了以下的类拷贝框架

使用

目前第二代密码管理平台采用的是mica重写后的BeanUtil类拷贝方法，整体来说性能可以满足要求，但为了进一步提升查询时类转换性能，同时考虑到应用中本身就包含了swagger依赖，而swagger中本身就引入了mapstruct依赖包，故无需引入新依赖。

因此，最终考虑将业务场景下的类拷贝改成Mapstruct实现，由此开始mapstruct爬坑之路。。

软件包版本

SpringBoot:2.4.2

SpringCloud:2020.0.1

lombok: 1.18.16

mapstruct: 1.3.1.Final

新增依赖

由于需要由mapstruct根据定义的接口自动生成类转换实现，故需要引入mapstruct接口处理包，有两种方式，一种通过maven插件的形式，一种通过maven依赖的形式

maven插件的形式（官方推荐）

<build>
 <plugins>
 <plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-compiler-plugin</artifactId>
 <version>3.5.1</version>
 <configuration>
 <source>1.8</source>
 <target>1.8</target>
 <annotationProcessorPaths>
 <path>
 <groupId>org.mapstruct</groupId>
 <artifactId>mapstruct-processor</artifactId>
 <version>${org.mapstruct.version}</version>
 </path>
 </annotationProcessorPaths>
 </configuration>
 </plugin>
 </plugins>
</build>

maven依赖的形式

<dependency>
 <groupId>org.mapstruct</groupId>
 <artifactId>mapstruct-processor</artifactId>
 <version>${mapstruct.version}</version>
</dependency>

考虑到平台的maven模块数量较大，互相依赖关系比较复杂，遗漏插件依赖容易导致难以排查的bug，最终采用第二种方式，减少编码量。

新增映射类

/**
 * @author F嘉阳
 * @date 2021/2/18 12:06
 */
@Mapper(componentModel = "spring", unmappedTargetPolicy = ReportingPolicy.IGNORE)
public interface OssEntityMapper {

	OssVO ossToOssVO(Oss oss);

	OssDTO ossToOssDto(Oss oss);

	/**
	 * bladeFile转Oss
	 *
	 * @param bladeFile
	 * @return
	 */
	@Mapping(target = "fileName", source = "name")
	Oss bladeFileToOss(BladeFile bladeFile);
}

新增单元测试

由于首次使用mapstruct，编写单元测试检测代码是否成功

/**
 * @author F嘉阳
 * @date 2021/2/18 14:23
 */
@Slf4j
@ExtendWith(BladeSpringExtension.class)
@SpringBootTest(classes = ResourceApplication.class)
@BladeBootTest(appName = AppConstant.APPLICATION_RESOURCE_NAME, profile = "test", enableLoader = true)
class OssEntityMapperTest {

	@Autowired
	private OssEntityMapper mapper;

	@Test
	void ossToOssVO() {
		Oss oss = new Oss();
		oss.setFileName("filename");
		oss.setLink("link");
		oss.setOriginalName("origin");
		oss.setOssType(OssType.QI_NIU_OSS);
		oss.setIsDeleted(BladeConstant.DB_NOT_DELETED);
		OssVO ossVO = mapper.ossToOssVO(oss);
		assertEquals("filename", ossVO.getFileName());
		assertEquals("link", ossVO.getLink());
		assertEquals("origin", ossVO.getOriginalName());
	}

	@Test
	void ossToOssDto() {
		Oss oss = new Oss();
		oss.setFileName("filename");
		oss.setLink("link");
		oss.setOriginalName("origin");
		oss.setOssType(OssType.QI_NIU_OSS);
		oss.setIsDeleted(BladeConstant.DB_NOT_DELETED);
		OssDTO ossDTO = mapper.ossToOssDto(oss);
		assertEquals("filename", ossDTO.getFileName());
		assertEquals("link", ossDTO.getLink());
		assertEquals("origin", ossDTO.getOriginalName());
	}

	@Test
	void bladeFileToOss() {
		BladeFile file = new BladeFile();
		file.setName("file");
		file.setLink("link");
		file.setOriginalName("origin");
		Oss oss = mapper.bladeFileToOss(file);
		assertEquals("file", oss.getFileName());
		assertEquals("link", oss.getLink());
		assertEquals("origin", oss.getOriginalName());
	}
}

编译结果

编译结果，报错

这就很奇怪了，这是按官网最简单的方式写的，而且IDEA也安装了mapstruct插件，也没报错，能正确识别属性，怎么编译就不通过了

查看生成的代码，确实没有属性映射

爬坑之路

由于编译报错，考虑以下方法解决

移除spring容器依赖，采用类引用的方式编写。——无效

移除属性映射，尝试编译，可以编译通过，但生成的代码没有同名属性映射，只会返回空值。——无效

@Override
public Oss bladeFileToOss(BladeFile bladeFile) {
 if ( bladeFile == null ) {
 return null;
 }

 Oss oss = new Oss();

 return oss;
}

更换类名后测试通过，多次测试仍然有不通过的情况。——可能和编译环境有关？
相同代码放在单独的工程（非maven多模块）测试通过。——可能和maven多模块依赖有关？
根据网上检索的结果，普遍反映的lombok版本太低导致的，经过检查，当前的lombok版本应该是满足最低版本要求的。——无效
考虑是否和spring某些组件冲突，单独新开maven多模块工程测试，依然测试不通过。——果然和maven有关？

解决方法

最终通过给官方仓库提交issue咨询得到答案

原来是lombok版本太高导致的。。

最终降低lombok版本问题解决。

Sentinel持久化存储和镜像构建

Sun, 07 Feb 2021 12:06:58 +0800

背景

Sentinel默认只能对限流规则存储到内存中

在生产环境中使用 Sentinel — Sentinel中文文档

官方文档对于数据持久化描述如下

我们提供了 ZooKeeper, Apollo, Nacos 等的动态数据源实现。以 ZooKeeper 为例子，如果要使用第三方的配置中心作为配置管理，您需要做下面的几件事情:

实现一个公共的 ZooKeeper 客户端用于推送规则，在 Sentinel 控制台配置项中需要指定 ZooKeeper 的地址，启动时即创建 ZooKeeper Client。

我们需要针对每个应用（appName），每种规则设置不同的 path（可随时修改）；或者约定大于配置（如 path 的模式统一为 /sentinel_rules/{appName}/{ruleType}，e.g. sentinel_rules/appA/flowRule）。

规则配置页需要进行相应的改造，直接针对应用维度进行规则配置；修改同个应用多个资源的规则时可以批量进行推送，也可以分别推送。Sentinel 控制台将规则缓存在内存中（如 InMemFlowRuleStore），可以对其进行改造使其支持应用维度的规则缓存（key 为 appName），每次添加/修改/删除规则都先更新内存中的规则缓存，然后需要推送的时候从规则缓存中获取全量规则，然后通过上面实现的 Client 将规则推送到 ZooKeeper 即可。

应用客户端需要注册对应的读数据源以监听变更，可以参考相关文档。

由于alibaba/Sentinel官方并没有实现对具体数据源的写入支持，只提供的实现的接口，故无法做到开箱即用

解决方法

已经有贡献者提供了对三种动态数据源的实现PR，但还没合并进主版本中

添加持久化支持，可选持久化选项 — Fixes Issues #1696 添加持久化支持，可选持久化选项：nacos、apollo、zookeeper ; 修改 README.md 说明文档

进一步完善

本次基于jiajiangnan/Sentinelfork出自己的工程FJiayang/Sentinel，实现docker镜像打包，并基于GitHub Actions完成自动化构建和上传制品。

jiajiangnan/Sentinel — jiajiangnan的Sentinel仓库源码

FJiayang/Sentinel — FJiayang的Sentinel仓库源码

添加Dockerfile

基于最小jdk镜像制作

FROM openjdk:8-jdk-alpine3.7

MAINTAINER fjy8018@gmail.com

ENV VERSION 1.8.0

RUN mkdir -p /sentinel

WORKDIR /sentinel

ADD ./sentinel-dashboard/target/sentinel-dashboard.jar ./app-${VERSION}.jar

# 设置时区，默认为UTC
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN echo 'Asia/Shanghai' > /etc/timezone

CMD java ${PARAM} -jar app-${VERSION}.jar

添加GitHub流水线配置文件

name: Publish Docker image

on: [push]

jobs:
 build:

 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v2
 - name: Set up Maven Central Repository
 uses: actions/setup-java@v1
 with:
 java-version: 1.8

 - name: Publish package
 run: mvn --batch-mode clean package -Dmaven.test.skip=true

 - name: Push to Docker Hub
 uses: docker/build-push-action@v1
 with:
 username: ${{ secrets.DOCKER_USERNAME }}
 password: ${{ secrets.DOCKER_PASSWORD }}
 repository: fjy8018/sentinel
 tag_with_ref: true

具体的流水线编写可参考文章

使用方法

具体参数文档

项	类型	默认值	最小值	描述
sentinel.dashboard.auth.username	String	sentinel	无	登录控制台的用户名，默认为 `sentinel`
sentinel.dashboard.auth.password	String	sentinel	无	登录控制台的密码，默认为 `sentinel`
sentinel.dashboard.app.hideAppNoMachineMillis	Integer	0	60000	是否隐藏无健康节点的应用，距离最近一次主机心跳时间的毫秒数，默认关闭
sentinel.dashboard.removeAppNoMachineMillis	Integer	0	120000	是否自动删除无健康节点的应用，距离最近一次其下节点的心跳时间毫秒数，默认关闭
sentinel.dashboard.unhealthyMachineMillis	Integer	60000	30000	主机失联判定，不可关闭
sentinel.dashboard.autoRemoveMachineMillis	Integer	0	300000	距离最近心跳时间超过指定时间是否自动删除失联节点，默认关闭
datasource.provider	String	memory	无	默认为 `memory`, 可选持久化配置 `nacos`、`apollo`、`zookeeper`
datasource.provider.nacos.server-addr	String	localhost:8848	无	nacos 注册中心地址
datasource.provider.nacos.username	String		无	nacos 用户名，默认为空
datasource.provider.nacos.password	String		无	nacos 密码，默认为空
datasource.provider.nacos.namespace	String		无	nacos 名臣空间，默认为空
datasource.provider.nacos.group-id	String	SENTINEL_GROUP	无	nacos 分组，默认为 `SENTINEL_GROUP`
datasource.provider.apollo.server-addr	String	http://localhost:10034	无	apollo 注册中心地址，必须有前缀 `http://` 或 `https://`
datasource.provider.apollo.token	String	token	无	apollo 登录 token，默认为 `token`
datasource.provider.zookeeper.server-addr	String	localhost:2181	无	zookeeper 注册中心地址
datasource.provider.zookeeper.session-timeout	Integer	60000	0	zookeeper session超时时间，默认 `60000`
datasource.provider.zookeeper.connection-timeout	Integer	15000	0	zookeeper connection超时时间，默认 `15000`
datasource.provider.zookeeper.retry.max-retries	Integer	3	0	zookeeper 最大重试次数，默认 `3`
datasource.provider.zookeeper.retry.base-sleep-time	Integer	1000	1000	zookeeper 重试间隔最小时长，默认 `1000`
datasource.provider.zookeeper.retry.max-sleep-time	Integer	2147483647	0	zookeeper 重试间隔最大时长，默认 `2147483647`

对于单条Docker命令启动

$ docker run --rm -e PARAM="-Dserver.port=8858 -Dcsp.sentinel.dashboard.server=localhost:8858 -Dproject.name=sentinel-dashboard -Ddatasource.provider=nacos -Ddatasource.provider.nacos.server-addr=192.168.133.128:8848 -Ddatasource.provider.nacos.username=nacos -Ddatasource.provider.nacos.password=nacos -Ddatasource.provider.nacos.group-id=SENTINEL_GROUP" fjy8018/sentinel:1.8.0

对于docker-compose启动，此处结合nacos与mysql基础设施给出示例，mysql需要提前写入nacos相关初始化表结构

version: '3.7'
services:

 mysql57:
 image: mysql:5.7.29
 restart: always
 command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
 ports:
 - 3306:3306
 networks:
 - dev
 volumes:
 - ./mysql57/data:/var/lib/mysql
 - ./mysql57/custom-master.cnf:/etc/mysql/conf.d/custom.cnf
 - ./mysql57/log:/var/log/mysql
 environment:
 MYSQL_ROOT_PASSWORD: root

 nacos-standalone:
 image: nacos/nacos-server:1.1.4
 networks:
 - dev
 environment:
 - PREFER_HOST_MODE=hostname
 - MODE=standalone
 - SPRING_DATASOURCE_PLATFORM=mysql
 - MYSQL_MASTER_SERVICE_HOST=mysql57
 - MYSQL_MASTER_SERVICE_DB_NAME=nacos
 - MYSQL_MASTER_SERVICE_PORT=3306
 - MYSQL_SLAVE_SERVICE_HOST=mysql57
 - MYSQL_SLAVE_SERVICE_PORT=3306
 - MYSQL_MASTER_SERVICE_USER=root
 - MYSQL_MASTER_SERVICE_PASSWORD=root
 ports:
 - 8848:8848
 depends_on:
 - mysql57
 volumes:
 - ./standalone-logs/:/home/nacos/logs

 sentinel:
 image: fjy8018/sentinel:1.8.0
 environment:
 - PARAM=-Dserver.port=8858 -Dcsp.sentinel.dashboard.server=localhost:8858 -Dproject.name=sentinel-dashboard -Ddatasource.provider=nacos -Ddatasource.provider.nacos.server-addr=nacos-standalone:8848 -Ddatasource.provider.nacos.username=nacos -Ddatasource.provider.nacos.password=nacos -Ddatasource.provider.nacos.group-id=SENTINEL_GROUP
 networks:
 - dev
 ports:
 - 8858:8858
 restart: on-failure

networks:
 dev:

启动成功后可以看到nacos上成功注册了sentinel-dashboard

GitHub DevOps工作流构建Java项目并上传到DockerHub

Sun, 07 Feb 2021 11:58:43 +0800

工程准备

此处使用fork自alibaba/Sentinel的FJiayang/Sentinel项目实践

工程地址

Sentinel — Sentinel仓库源码

准备Dockerfile

该工程为标准Java项目，故准备好的Dockerfile文件如下

FROM anapsix/alpine-java:8_server-jre_unlimited

MAINTAINER fjy8018@gmail.com

ENV PARAM ""
ENV VERSION 1.8.0

RUN mkdir -p /sentinel

WORKDIR /sentinel

ADD ./sentinel-dashboard/target/sentinel-dashboard.jar ./app-${VERSION}.jar

# 设置时区，默认为UTC
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN echo 'Asia/Shanghai' > /etc/timezone

ENTRYPOINT ["java", "${PARAM}", "-jar", "app-${VERSION}.jar"]

编写GitHub工作流

GitHub DevOps工作流又叫GitHub Actions 是2019年末推出的自动化CICD持续集成工作流，有详细的样例文档

此处参考以下文档编写

使用 Maven 构建和测试 Java — GitHub Actions 指南

安全配置

由于需要建构建后的制品推送到DockerHub中，但又不能在工作流中直接使用用户名和密码登陆，故还需要配置环境变量

在DockerHub中新增秘钥

在构建工程的设置中新增用户名和密码变量

编写后的工作流如下

name: Publish Docker image

on: [push]

jobs:
 build:

 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v2
 - name: Set up Maven Central Repository
 uses: actions/setup-java@v1
 with:
 java-version: 1.8

 - name: Publish package
 run: mvn --batch-mode clean package -Dmaven.test.skip=true

 - name: Push to Docker Hub
 uses: docker/build-push-action@v1
 with:
 username: ${{ secrets.DOCKER_USERNAME }}
 password: ${{ secrets.DOCKER_PASSWORD }}
 repository: fjy8018/sentinel
 tag_with_ref: true

触发构建

当前的触发方式配置为PUSH，修改好文件后会自动触发流水线工作

Maven打包，速度很快

打包完成

检查结果

登录到DockerHub中检查推送结果，可见成功推送

总结

整体来看，GitHub工作流效率高，文档齐全，编码上手速度快，相比GitLab CICD和DockerHub CICD来说，GitHub CICD配置更加简单、友好和高效。

DevOps实践心得二

Sun, 27 Dec 2020 12:25:47 +0800

背景

上一篇DevOps实践心得编写于2019年08月16日

文章最后提到未来探索方向是

全面容器化服务治理方向

国内对容器化有非常多的实践

kubernetes作为容器调度成为大多数服务治理的选择

容器隔离、迁移和伸缩带来的巨大优势

容器与传统应用部署结合方式

以半全自动方式落地DevOps，采用Mesos+marathon方式同时兼容传统服务部署和容器化部署

兼容分布式服务治理

国外有大量的实践案例

日志收集体系完备

经过一年半的时间，完成对完整DevOps的探索

Mesos+Marathon的DevOps分布式环境探索

参考文章

该架构更加适合还未进行容器化改造的项目，其支持较为通用，但各组件都是由开源组件协调而成，故搭建较为繁琐，而且后续版本升级并不方便

全面容器化服务探索

探索历程

最终完成了DevOps容器化自动部署实践

采用kuberneties方案，其统一性更加一致，依赖的基础设施也多支持容器化部署，机器利用率较高，版本升级、维护较为方便，适用于云原生项目部署

遇到的坑

最终生产环境采用kuberneties集群方案，5个节点，但由于实施经验不足，依然遇到不少坑

后续发展展望

云原生技术还在不断发展中，不少公司也研发并开源关于kuberneties集群管理方案，例如kubesphere

kubesphere — The Kubernetes platform tailored for hybrid multicloud

经过测试，其2.x版本资源占用较高，还处于刚起步阶段，暂时不适用于小规模集群

k8s证书失效问题排查

Sun, 27 Dec 2020 12:00:58 +0800

问题背景

生产环境kuberneties集群搭建于2019年11月7日，使用kubeadm结合国内镜像搭建，参考文章

2020年11月8日突然发现生产环境集群服务全都失效并且不可用，进行紧急排查处理

处理过程

1. 排查服务是否启动

[root@k8smaster1 ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
65dff767f422 b95b1efa0436 "kube-controller-m..." 20 hours ago Up 20 hours k8s_kube-controller-manager_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_11
1b26acecbcea registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_1
9c6449df0a18 00638a24688b "kube-scheduler --..." 20 hours ago Up 20 hours k8s_kube-scheduler_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_10
028a1177cc9b registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_1
6c6bdc2c2644 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1
fe4a12daa7d7 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1

容器服务正常，但kubelet服务有报错信息，报连接不上apiserver

[root@k8smaster1 ~]$ systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
 Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
 Drop-In: /usr/lib/systemd/system/kubelet.service.d
 └─10-kubeadm.conf
 Active: active (running) since 日 2020-11-08 22:16:07 CST; 20h ago
 Docs: https://kubernetes.io/docs/
 Main PID: 15313 (kubelet)
 Tasks: 16
 Memory: 68.4M
 CGroup: /system.slice/kubelet.service
 └─15313 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1

11月 09 18:43:06 k8smaster1 kubelet[15313]: E1109 18:43:06.967416 15313 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://192.168.1.14:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.14:6443: connect: connection refused
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.009411 15313 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.109665 15313 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:43:07 k8smaster1 kubelet[15313]: I1109 18:43:07.165755 15313 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.167106 15313 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSIDriver: Get https://192.168.1.14:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 192.168.1.14:6443: connect: connection refused
11月 09 18:43:07 k8smaster1 kubelet[15313]: I1109 18:43:07.169000 15313 kubelet_node_status.go:72] Attempting to register node k8smaster1
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.209831 15313 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.310013 15313 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.362049 15313 kubelet_node_status.go:94] Unable to register node "k8smaster1" with API server: Post https://192.168.1.14:6443/api/v1/nodes: dial tcp 192.168.1.14:6443: connect: connection refused
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.410174 15313 kubelet.go:2244] node "k8smaster1" not found

2. 检查APIServer

apiserver 端口连接正常

[root@k8smaster1 ~]$ telnet 192.168.1.14 6443
Trying 192.168.1.14...
telnet: connect to address 192.168.1.14: Connection refused

重设防火墙，并重新检查，连接正常

[root@k8smaster1 ~]$ iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT
[root@k8smaster1 ~]$ telnet 192.168.1.14 6443
Trying 192.168.1.14...
Connected to 192.168.1.14.

但服务仍然无法访问

3. 重启kubelet

[root@k8smaster1 ~]$ systemctl restart kubelet
[root@k8smaster1 ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4383926ef6c9 ecf910f40d6e "kube-apiserver --..." 3 seconds ago Up 2 seconds k8s_kube-apiserver_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1362
88eb6134fe3c 2c4adeb21b4f "etcd --advertise-..." 3 seconds ago Up 2 seconds k8s_etcd_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1768
65dff767f422 b95b1efa0436 "kube-controller-m..." 20 hours ago Up 20 hours k8s_kube-controller-manager_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_11
1b26acecbcea registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_1
9c6449df0a18 00638a24688b "kube-scheduler --..." 20 hours ago Up 20 hours k8s_kube-scheduler_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_10
028a1177cc9b registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_1
6c6bdc2c2644 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1
fe4a12daa7d7 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1

检查kubelet状态，仍然报错

[root@k8smaster1 ~]$ systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
 Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
 Drop-In: /usr/lib/systemd/system/kubelet.service.d
 └─10-kubeadm.conf
 Active: active (running) since 一 2020-11-09 18:52:24 CST; 24s ago
 Docs: https://kubernetes.io/docs/
 Main PID: 31759 (kubelet)
 Tasks: 15
 Memory: 30.1M
 CGroup: /system.slice/kubelet.service
 └─31759 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1

11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.063832 31759 controller.go:115] failed to ensure node lease exists, will retry in 3.2s, error: Get https://192.168.1.14:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/k8smaster1?timeout=10s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.152723 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.252964 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.353226 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.453452 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.553682 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.653965 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.754181 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.854401 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.954576 31759 kubelet.go:2244] node "k8smaster1" not found

4.检查API Server日志

查容器ID

[root@k8smaster1 ~]$ docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
27a3b7787748 ecf910f40d6e "kube-apiserver --..." 18 seconds ago Up 18 seconds

查看日志

[root@k8smaster1 ~]$ docker logs 27a
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I1109 10:55:42.281078 1 server.go:559] external host was not specified, using 192.168.1.15
I1109 10:55:42.281264 1 server.go:146] Version: v1.14.0
I1109 10:55:42.759000 1 plugins.go:158] Loaded 9 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I1109 10:55:42.759031 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
E1109 10:55:42.759943 1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.759976 1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.760007 1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.760038 1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.760065 1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.760086 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
I1109 10:55:42.760102 1 plugins.go:158] Loaded 9 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I1109 10:55:42.760110 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I1109 10:55:42.763844 1 client.go:352] parsed scheme: ""
I1109 10:55:42.763866 1 client.go:352] scheme "" not registered, fallback to default scheme
I1109 10:55:42.764295 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I1109 10:55:42.764449 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
W1109 10:55:42.771517 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
I1109 10:55:43.756910 1 client.go:352] parsed scheme: ""
I1109 10:55:43.756939 1 client.go:352] scheme "" not registered, fallback to default scheme
I1109 10:55:43.756999 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I1109 10:55:43.757077 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
W1109 10:55:43.763355 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:43.770837 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:44.764994 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:45.541971 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:46.657328 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:47.938720 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:48.730961 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:51.899665 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:52.961615 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:59.193916 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:56:00.025333 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
F1109 10:56:02.764150 1 storage_decorator.go:57] Unable to create storage backend: config (&{ /registry {[https://127.0.0.1:2379] /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/etcd/ca.crt} false true 0xc0007a94d0 apiextensions.k8s.io/v1beta1 <nil> 5m0s 1m0s}), err (context deadline exceeded)

日志最后提示：authentication handshake failed: x509: certificate has expired or is not yet valid

以此信息查询解决方案

解决方案

外部推荐方案

最终在该网站找到解决方案

Kubeadm证书过期时间调整 — Devops、小铁匠

采用方案

由于生产环境集群版本为1.14，是支持自动轮换证书的，故首先解决生产环境问题，先手动重新签发证书，再开启证书自动续签

重新签发证书

国内网络直接签发证书会因为无法连接Google而报错

[root@k8smaster1 ~]$ kubeadm alpha certs renew all
I1109 19:33:40.266778 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:33:40.266978 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:33:51.139051 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:33:51.139111 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:01.511909 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1109 19:34:01.511967 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:11.945411 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:34:11.945504 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:22.145905 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:34:22.145975 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:32.532514 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:34:32.532561 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:43.335574 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1109 19:34:43.335628 22487 version.go:97] falling back to the local client version: v1.14.0

先准备一份连接国内镜像的kubeadm.conf文件

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.14.0 #-->这里改成你集群对应的版本
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers 
#这里使用国内的镜像仓库，否则在重新签发的时候会报错：could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt"

重新签发，即可，并重启kubelet

[root@k8smaster1 ~]$ kubeadm alpha certs renew all --config=/root/kubeadm.conf
[root@k8smaster1 ~]$ systemctl restart kubelet

重新签发后若要使用kubectl命令，要重新生成~/.kube/config配置文件

[root@k8smaster1 kubernetes]$ kubeadm init phase kubeconfig all --config=/root/kubeadm.conf
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[root@k8smaster1 kubernetes]$ cp -i /etc/kubernetes/admin.conf ~/.kube/config
cp：是否覆盖"/root/.kube/config"？ y

命令正常使用

[root@k8smaster1 kubernetes]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
glusterfs-hfdmh 1/1 Running 0 351d
glusterfs-k766z 1/1 Running 0 351d
glusterfs-rrc7x 1/1 Running 0 351d
heketi-68f9dfdfbf-2k58b 1/1 Running 0 351d
nginx 1/1 Running 1 370d
nginx-ds-29dbc 1/1 Running 5 366d
nginx-ds-4w6cn 1/1 Running 1 370d
nginx-ds-6lhsk 0/1 Evicted 0 216d
nginx-ds-xq4h7 1/1 Running 3 366d
tomcat-demo-6bc7d5b6f4-75rgc 0/1 Evicted 0 351d

重新启动集群上的所有服务，可见服务正常

证书自动续签

kubelet证书分为server和client两种， k8s 1.9默认启用了client证书的自动轮换，但server证书自动轮换需要用户开启

查询kubelet配置文件位置

[root@k8smaster1 ~]$ find / -name 10-kubeadm.conf
/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
[root@k8smaster1 ~]$ vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

添加配置如下

# 在/etc/systemd/system/kubelet.service.d/10-kubeadm.conf 增加如下参数
Environment="KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true"

添加Controller-Manager参数

apiVersion: v1
kind: Pod
metadata:
 creationTimestamp: null
 labels:
 component: kube-controller-manager
 tier: control-plane
 name: kube-controller-manager
 namespace: kube-system
spec:
 containers:
 - command:
 - kube-controller-manager
 - --allocate-node-cidrs=true
 # 证书有效期
 - --experimental-cluster-signing-duration=87600h0m0s
 # 证书自动签发
 - --feature-gates=RotateKubeletServerCertificate=true
 - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
 - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
 - --bind-address=127.0.0.1
 - --client-ca-file=/etc/kubernetes/pki/ca.crt
 - --cluster-cidr=172.22.0.0/16
 - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
 - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
 - --controllers=*,bootstrapsigner,tokencleaner
 - --kubeconfig=/etc/kubernetes/controller-manager.conf
 - --leader-elect=true
 - --node-cidr-mask-size=24
 - --requestheader-client-ca-file=/etc/k

创建RBAC对象

cat > ca-update.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 annotations:
 rbac.authorization.kubernetes.io/autoupdate: "true"
 labels:
 kubernetes.io/bootstrapping: rbac-defaults
 name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
 - certificates.k8s.io
 resources:
 - certificatesigningrequests/selfnodeserver
 verbs:
 - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: kubeadm:node-autoapprove-certificate-server
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
 kind: Group
 name: system:nodes
EOF

[root@k8smaster1 ~]$ kubectl create -f ca-update.yaml
clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeserver created
clusterrolebinding.rbac.authorization.k8s.io/kubeadm:node-autoapprove-certificate-server created

RocketMQ学习笔记6——集群实战

Wed, 23 Dec 2020 19:15:00 +0800

组建高性能MQ服务

双主双从模式
保障无单点故障问题的高可用集群

集群拓扑图

IP	资源	服务
10.139.12.52	1C2G20GB	rocketmq-nameserver1、rocketmq-master1
10.139.12.53	1C2G20GB	rocketmq-nameserver2、rocketmq-master2
10.139.12.54	1C2G20GB	rocketmq-nameserver3、rocketmq-master1-slave
10.139.12.55	1C2G20GB	rocketmq-nameserver4、rocketmqmaster2-slave

配置文件最优设置解析

修改host文件映射

10.139.12.52 rocketmq-nameserver1 rocketmq-master1
10.139.12.53 rocketmq-nameserver2 rocketmq-master2
10.139.12.54 rocketmq-nameserver3 rocketmq-master1-slave
10.139.12.55 rocketmq-nameserver4 rocketmq-master2-slave

将4个节点的hosts均改为如上内容，并将4.4.0版本包放入环境中解压

创建存储目录

$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index

建立软连接

$ cd ~/rocketmq-test
$ ln -s rocketmq-all-4.4.0-bin-release rocketmq

修改配置文件

解压后默认的配置文件如下

[user@301 ~/rocketmq-test/rocketmq/conf]$ll
total 36
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync
drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave
-rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf
-rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml
-rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml
-rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml
-rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml
-rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml

若想保障生产环境高可用，推荐使用双主双从同步双写模式，可以让一个主节点对应多个从节点，但从节点越多，同步效率越慢

本次采用双主双从同步双写模式，一个主节点对应一个从节点

[user@301 ~/rocketmq-test/rocketmq/conf/2m-2s-sync]$ll
total 16
-rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-a.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties
-rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-b.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties

用途如下

IP	资源	服务	配置文件
10.139.12.52	1C2G20GB	rocketmq-nameserver1、rocketmq-master1	broker-a.properties
10.139.12.53	1C2G20GB	rocketmq-nameserver2、rocketmq-master2	broker-b.properties
10.139.12.54	1C2G20GB	rocketmq-nameserver3、rocketmq-slave1	broker-a-s.properties
10.139.12.55	1C2G20GB	rocketmq-nameserver4、rocketmq-slave2	broker-b-s.properties

`broker-a.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=false
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=false
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间，最大100%
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-b.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-b
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=false
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=false
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间，最大100%
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-a-s.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

`broker-b-s.properties` 配置

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对
brokerName=broker-b
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort
#限制的消息大小
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000
#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE
#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

分发配置文件

$ scp broker-* ocmuser@rocketmq-master2-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync
$ scp broker-* ocmuser@rocketmq-master2:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync
$ scp broker-* ocmuser@rocketmq-master1-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync

修改日志配置文件

$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq/logs
$ cd /home/ocmuser/rocketmq-test/rocketmq/conf && sed -i 's#${user.home}#/home/ocmuser/rocketmq-test/#g' *.xml

修改脚本启动参数

/home/ocmuser/rocketmq-test/rocketmq/bin/runbroker.sh内容如下，至少保障1g内存

#======================================================================================
# 开发环境JVM Configuration
#======================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"

/home/ocmuser/rocketmq-test/rocketmq/bin/runserver.sh内容如下

#======================================================================================
# JVM Configuration
#======================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

启动 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin
$ nohup sh mqnamesrv &

启动 Master1:BrokerServerA

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master2:BrokerServerB

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master1-Slave:BrokerServerC

$ cd cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a-s.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

启动 Master2-Slave:BrokerServerD

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b-s.properties >/dev/null 2>&1 & 
$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log

查看进程和日志，可见部署完成

角色为同步双写，异步刷盘，4个节点通信正常

控制台监控验证

修改连接配置

rocketmq.config.namesrvAddr=10.139.12.51:9876;10.139.12.52:9876;10.139.12.53:9876;10.139.12.54:9876

连接正常

查看各节点角色和状态

服务停止

首先关闭 4 个 BrokerServer，再关闭 4 个 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq/bin 
$ sh mqshutdown broker 
$ sh mqshutdown namesrv

双主双从环境构建与调优

为取得可靠性与效率的平衡，建议采用同步双写，异步刷盘的模式

RocketMQ学习笔记5——原理解析

Wed, 23 Dec 2020 19:13:00 +0800

消息存储结构

CommitLog：消息主体以及元数据的存储主体，存储Producer端写入的消息主体内容,消息内容不是定长的。消息主要是顺序写入日志文件，当文件满了，写入下一个文件；其日志信息科用于数据恢复，作为MQ数据保障兜底策略。
ConsumerQueue：消息消费队列，存储消息在commit log 中所在的这个位置，也可以说是一个偏移量.引入的目的主要是提高消息消费的性能。
IndexFile：IndexFile（索引文件）提供了一种可以通过key或时间区间来查询消息的方法。
每一个topic下的每一个message queue都对应一个ConsumerQueue文件，具体存储位置通过配置文件配置。
RocketMQ采用的是混合型的存储结构，即为Broker单个实例下所有的队列（ConsumerQueue）共用一个日志数据文件（即为CommitLog）来存储。
刷盘采用顺序写随机读的模式。
消费端采用“0拷贝”模式提升性能。

同步\异步刷盘

消息存储：内存+磁盘存储，两种刷盘方式

异步刷盘

能够充分利用OS的PageCache的优势，只要消息写入PageCache（内存）即可将成功的ACK（成功信息）返回给Producer端。
消息堆积到一定量才触发刷盘。
消息刷盘采用后台异步线程提交的方式进行，降低了读写延迟，提高了MQ的性能和吞吐量。

同步刷盘

只有在消息真正持久化至磁盘后RocketMQ的Broker端才会真正返回给Producer端一个成功的ACK响应。
同步刷盘对MQ消息可靠性来说是一种不错的保障，但是性能上会有较大影响，一般适用于金融业务应用该模式较多。

同步\异步复制

针对集群模式设计，同一组Broker有Master-Slave角色
推荐采用同步双写，异步复制的模式保持性能与可靠性的平衡

同步复制

Master与Slave都成功才返回ACK（成功信息）给Producer

异步复制

Master节点刷盘成功后立即返回ACK（成功信息）给Producer
再通过异步线程将信息同步给其他Slave节点
高性能，但存在数据丢失风险

同步双写

每个Master配置一个Slave，有多对Master-Slave，HA采用同步双写方式，即只有主备都写成功，才向应用返回成功，这种模式的优缺点如下：

优点：数据与服务都无单点故障，Master宕机情况下，消息无延迟，服务可用性与数据可用性都非常高；
缺点：性能比异步复制模式略低（大约低10%左右），发送单个消息的RT会略高，且目前版本在主节点宕机后，备机不能自动切换为主机。

高可用机制

Master-Slave搭配实现高可用机制
通过BrokerId区分节点
Master支持读写，Slave只读
当Master繁忙或者不可用时，可以自动切换到Slave读取消息

最佳实践

创建Topic时，将Topic的Message Queue创建到多个Broker组中，使得逻辑上高可用
由于RocketMQ不支持主从切换，故可以考虑使用监控程序监听主节点的有效性，若连接超时，则将Slave节点的配置改为Master并重启，升级为主节点，实现高可用。

NameServer协调者

NameServer是一个非常简单的Topic路由注册中心，其角色类似Dubbo中的zookeeper，支持Broker的动态注册与发现，是整个集群的状态服务器。
主要包括两个功能：
- Broker管理，NameServer接受Broker集群的注册信息并且保存下来作为路由信息的基本数据。然后提供心跳检测机制，检查Broker是否还存活；
- 路由信息管理，每个NameServer将保存关于Broker集群的整个路由信息和用于客户端查询的队列信息。然后Producer和Conumser通过NameServer就可以知道整个Broker集群的路由信息，从而进行消息的投递和消费。
NameServer通常也是集群的方式部署，各实例间相互不进行信息通讯。Broker是向每一台NameServer注册自己的路由信息，所以每一个NameServer实例上面都保存一份完整的路由信息。
当某个NameServer因某种原因下线了，Broker仍然可以向其它NameServer同步其路由信息，Producer,Consumer仍然可以动态感知Broker的路由的信息。

为什么不用zookeeper

RocketMQ 3.x以前使用zookeeper实现，后来改为nameserver实现
zookeeper提供主从选举，广播原子性等特性在MQ中用不上，功能冗余
zk使用主从自动切换提供高可用保障，nameserver使用多副本节点保障高可用
nameserver只提供路由等基本状态信息维护，代码量小，更加轻量，便于维护。

NameServer信息维护源码

package org.apache.rocketmq.namesrv.routeinfo;

public class RouteInfoManager {
 private final ReadWriteLock lock = new ReentrantReadWriteLock();
 private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;
 private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
 private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
 private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
 private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;
}

RocketMQ学习笔记4——消费者核心

Wed, 23 Dec 2020 19:11:00 +0800

配置参数详解

PushConsumer核心参数

ConsumeFromWhere源码，支持三种参数配置，默认是CONSUME_FROM_LAST_OFFSET

package org.apache.rocketmq.common.consumer;

public enum ConsumeFromWhere {
 /**
 * 第一次启动时从队列最后开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_LAST_OFFSET,

 @Deprecated
 CONSUME_FROM_LAST_OFFSET_AND_FROM_MIN_WHEN_BOOT_FIRST,
 @Deprecated
 CONSUME_FROM_MIN_OFFSET,
 @Deprecated
 CONSUME_FROM_MAX_OFFSET,
 /**
 * 从队列最开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_FIRST_OFFSET,
 /**
 * 指定具体时间点开始消费
 * 下次启动从上次读取的位置开始消费
 */
 CONSUME_FROM_TIMESTAMP,
}

AllocateMessageQueueStrategy接口用于集群模式下消息策略，默认配置为平均消息分配器AllocateMessageQueueAveragely

默认配置源码

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Constructor specifying consumer group and enabled msg trace flag.
 *
 * @param consumerGroup Consumer group.
 * @param enableMsgTrace Switch flag instance for message trace.
 */
 public DefaultMQPushConsumer(final String consumerGroup, boolean enableMsgTrace) {
 this(null, consumerGroup, null, new AllocateMessageQueueAveragely(), enableMsgTrace, null);
 }
}

subscription订阅参数，传递两个值，一个是主题，一个是表达式tag过滤

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Subscription relationship
 */
 private Map<String /* topic */, String /* sub expression */> subscription = new HashMap<String, String>();
}

样例：

consumer.subscribe("test_model_topic2", "TagA||TagB");

consumer.subscribe("test_model_topic2", "TagA");

consumer.subscribe("test_model_topic2", "*");

offsetStore配置核心偏移量，有两个实现，用于支持远程存储和本地存储，即支持集群和单机模式

consumeThreadMin与consumeThreadMax线程池自动调整参数，默认为20

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Minimum consumer thread number
 */
 private int consumeThreadMin = 20;

 /**
 * Max consumer thread number
 */
 private int consumeThreadMax = 20;
}

流控参数consumeConcurrentlyMaxSpan与pullThresholdForQueue

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Concurrently max span offset.it has no effect on sequential consumption
 * 单个队列并行消费的最大跨度
 */
 private int consumeConcurrentlyMaxSpan = 2000;

 /**
 * Flow control threshold on queue level, each message queue will cache at most 1000 messages by default,
 * Consider the {@code pullBatchSize}, the instantaneous value may exceed the limit
 * 一个队列最大的消息消费个数
 */
 private int pullThresholdForQueue = 1000;
}

消息拉取参数pullInterval与pullBatchSize

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Message pull Interval
 * 消息拉取的时间间隔
 */
 private long pullInterval = 0;

 /**
 * Batch pull size
 * 一次拉取的数据量
 */
 private int pullBatchSize = 32;
}

consumeMessageBatchMaxSize一次消息最大拉取的数据数量配置

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Batch consumption size
 * 一次消息最大拉取的数据数量
 */
 private int consumeMessageBatchMaxSize = 1;
}

MessageListener用于消息监听配置，包括并发、顺序读取等

MessageModel配置消息模式，广播或者集群，默认为集群模式

package org.apache.rocketmq.client.consumer;

public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
 /**
 * Message model defines the way how messages are delivered to each consumer clients.
 * </p>
 *
 * RocketMQ supports two message models: clustering and broadcasting. If clustering is set, consumer clients with
 * the same {@link #consumerGroup} would only consume shards of the messages subscribed, which achieves load
 * balances; Conversely, if the broadcasting is set, each consumer client will consume all subscribed messages
 * separately.
 * </p>
 *
 * This field defaults to clustering.
 */
 private MessageModel messageModel = MessageModel.CLUSTERING;
}

集群与广播模式

集群模式

MessageModel源码支持两种模式，集群和广播

package org.apache.rocketmq.common.protocol.heartbeat;

/**
 * Message model
 */
public enum MessageModel {
 /**
 * broadcast
 */
 BROADCASTING("BROADCASTING"),
 /**
 * clustering（默认）
 */
 CLUSTERING("CLUSTERING");

 private String modeCN;

 MessageModel(String modeCN) {
 this.modeCN = modeCN;
 }

 public String getModeCN() {
 return modeCN;
 }
}

集群模式下GroupName用于把多个Consumer组织到一起，所以同一个集群，GroupName必须相同
相同的GroupName的Consumer只消费所订阅消息的一部分，即支持负载均衡
优势：集群模式下增加节点能动态增加负载均衡
一条消息只会被一个Consumer消费

样例

消息消费与奇偶性无关，随机负载均衡，并且负载均衡存在一定的差值，并非100%平分
为提高附在均衡度，可以用2个consume监听4个队列，或者3个consume监听6个队列
Producer只能保证平均分配到各个队列，不能保证被平均的消费

生产者

public class Producer {
	
 public static void main(String[] args) throws MQClientException, InterruptedException {
 	String group_name = "test_model_producer_name";
 DefaultMQProducer producer = new DefaultMQProducer(group_name);
 producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 producer.start();
 
 for (int i = 0; i < 10; i++) {
 try {
 	String tag = (i % 2 ==0) ? "TagA":"TagB";
 Message msg = new Message("test_model_topic2",// topic
 		tag,// tag
 ("信息内容" + i).getBytes()// body
 );
 SendResult sendResult = producer.send(msg);
 System.out.println(sendResult);
 }
 catch (Exception e) {
 e.printStackTrace();
 Thread.sleep(1000);
 }
 }
 
 producer.shutdown();
 }
}

消费者1

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.CLUSTERING);
//			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					//if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					//}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
		
	}
}

消费者2

public class Consumer2 {

	public Consumer2() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.CLUSTERING);
//			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					//if(tags.equals("TagB")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					//}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer2 c2 = new Consumer2();
		System.out.println("c2 start..");
		
	}
}

发送10条消息结果

消费者1结果

消费者2结果

注意，若使用集群模式，则tag表达式应该为*号，才能支持多consumer消费，若集群模式下指定tag表达式，则会导致消费结果不确定，原因如下

consumerA监听4条队列中的2条，consumerB监听4条队列中的另外2条。
Producer将包含TagA和TagB的消息均分到4条队列中。
consumerA消费TagA的消息，那么此时consumerA监听的2条队列中的TagB消息则永远不会被消费

样例

consumer1改成监听TagA，其他内容不变

consumer.subscribe("test_model_topic2", "TagA");

此时发送10条消息，consumer1监听的结果

此时将consumer2改成监听TagB并启动

consumer.subscribe("test_model_topic2", "TagB");

此时发送10条消息，consumer1监听的结果

consumer2监听结果为空，大量消息丢失

经过实测发现，只要集群模式下指定了tag，那么只有一个consumer能正常进行消费，其他consumer都为空

广播模式

同一个ConsumeGroup里的Consumer都消费订阅Topic全部信息
也就是一条消息会被每个Consume消费

样例

将consumer1和2改成广播模式，并监听所有tag

consumer.setMessageModel(MessageModel.BROADCASTING);
consumer.subscribe("test_model_topic2", "*");

发送10条消息consumer1消费结果

consumer2消费结果

注意广播模式下使用服务端tag过滤仍然有bug

将consumer1设置为只监听tagA消息

consumer.subscribe("test_model_topic2", "TagA");

consumer2设置为只监听tag2消息

consumer.subscribe("test_model_topic2", "TagB");

发送10条消息consumer1消费结果

consumer2消费结果

结果只有其中一个consumer能正常消费，与集群模式一样的问题

目前（4.4.0）若想要实现同一个topic的tag过滤，则只能让consumer监听所有消息并主动判断

样例：

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "*");
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
	}
}

发送10条消息结果

消费者1结果

消费者2结果

对于consumer消费指定队列消息，推荐做法是将消息投递到专属的队列中，由同一个consumer监听，即相当于用groupname+topic唯一定位一个consumer，该方法集群与广播模式均适用

样例

此时consumer1变为test_model_consumer_name1

public class Consumer1 {

	public Consumer1() {
		try {
			String group_name = "test_model_consumer_name1";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "TagA");
//			consumer.setMessageModel(MessageModel.CLUSTERING);
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagA")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer1 c1 = new Consumer1();
		System.out.println("c1 start..");
	}
}

consumer2变为test_model_consumer_name2

public class Consumer2 {

	public Consumer2() {
		try {
			String group_name = "test_model_consumer_name2";
			DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name);
			consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
			consumer.subscribe("test_model_topic2", "TagB");
//			consumer.setMessageModel(MessageModel.CLUSTERING);
			consumer.setMessageModel(MessageModel.BROADCASTING);
			consumer.registerMessageListener(new Listener());
			consumer.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	
	class Listener implements MessageListenerConcurrently {
		public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
			try {
				for(MessageExt msg : msgs){
					String topic = msg.getTopic();
					String msgBody = new String(msg.getBody(),"utf-8");
					String tags = msg.getTags();
					if(tags.equals("TagB")) {
						System.out.println("收到消息：" + " topic :" + topic + " ,tags : " + tags + " ,msg : " + msgBody);
					}
				}
			} catch (Exception e) {	
				e.printStackTrace();
				return ConsumeConcurrentlyStatus.RECONSUME_LATER;
			}			
			return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
		}
		
	}
	
	public static void main(String[] args) {
		Consumer2 c2 = new Consumer2();
		System.out.println("c2 start..");
	}
}

发送10条消息结果

消费者1结果

消费者2结果

消费存储核心-OffSet

Offset是消费者消费进度的核心
Offset指某个topic下的一条消息在某个MessageQueue里的位置
通过Offset可以进行定位到这条消息
Offset的存储实现分为远程文件类型和本地文件类型两种

集群模式-`RemoteBrokerOffsetStore`

默认集群模式clustering，采用远程文件存储Offset
本质上因为多消费模式，每个Consumer消费所订阅主题的一部分

广播模式-`LocalFileOffsetStore`

广播模式下，由于每个Consumer都会收到消息且消费
各个Consumer之间没有任何干扰，独立线程消费
所以使用LocalFileOffsetStore，也就是把Offset存储到本地

PushConsumer长轮询模式分析

DefaultMQPushConsumer是使用长轮询模式进行实现的
通常主流的消息获取模式：Push消息推送模式与Pull消息拉取模式
Push模式消耗broker端性能，若消费端消费速度不一致，可能导致消息堆积
pull模式消耗消费端性能，消费端不知道消息达到时间
RocketMQ采用长轮询机制（消费端主动获取）

长轮询过程

客户端发起获取消息请求，假设一次请求超时时间为15s
服务端判断是否有消息，有则直接返回
服务端无消息，则等待5s（阈值），5s后再次检查是否有消息，有则返回，无则继续等待
15s后循环检查结束，若仍然无消息，则返回给客户端

优势：

主动权在客户端，当服务端消息堆积，不会全推送到客户端

缺点：

连接返回时间较长，占用客户端资源（请求阻塞，等待服务端返回）

源码分析

源码实现位置

PullRequestHoldService继承自Runnable接口

核心类PullRequestHoldService源码全貌

package org.apache.rocketmq.broker.longpolling;

public class PullRequestHoldService extends ServiceThread {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME);
 private static final String TOPIC_QUEUEID_SEPARATOR = "@";
 private final BrokerController brokerController;
 private final SystemClock systemClock = new SystemClock();
 private ConcurrentMap<String/* topic@queueId */, ManyPullRequest> pullRequestTable =
 new ConcurrentHashMap<String, ManyPullRequest>(1024);

 public PullRequestHoldService(final BrokerController brokerController) {
 this.brokerController = brokerController;
 }

 public void suspendPullRequest(final String topic, final int queueId, final PullRequest pullRequest) {
 String key = this.buildKey(topic, queueId);
 ManyPullRequest mpr = this.pullRequestTable.get(key);
 if (null == mpr) {
 mpr = new ManyPullRequest();
 ManyPullRequest prev = this.pullRequestTable.putIfAbsent(key, mpr);
 if (prev != null) {
 mpr = prev;
 }
 }

 mpr.addPullRequest(pullRequest);
 }

 private String buildKey(final String topic, final int queueId) {
 StringBuilder sb = new StringBuilder();
 sb.append(topic);
 sb.append(TOPIC_QUEUEID_SEPARATOR);
 sb.append(queueId);
 return sb.toString();
 }

 @Override
 public void run() {
 log.info("{} service started", this.getServiceName());
 while (!this.isStopped()) {
 try {
 // 判断是否启用长轮询
 if (this.brokerController.getBrokerConfig().isLongPollingEnable()) {
 // 等待5s
 this.waitForRunning(5 * 1000);
 } else {
 this.waitForRunning(this.brokerController.getBrokerConfig().getShortPollingTimeMills());
 }

 long beginLockTimestamp = this.systemClock.now();
 // 检查请求
 this.checkHoldRequest();
 long costTime = this.systemClock.now() - beginLockTimestamp;
 if (costTime > 5 * 1000) {
 log.info("[NOTIFYME] check hold request cost {} ms.", costTime);
 }
 } catch (Throwable e) {
 log.warn(this.getServiceName() + " service has exception. ", e);
 }
 }

 log.info("{} service end", this.getServiceName());
 }

 @Override
 public String getServiceName() {
 return PullRequestHoldService.class.getSimpleName();
 }

 private void checkHoldRequest() {
 // 遍历所有请求的channel
 for (String key : this.pullRequestTable.keySet()) {
 String[] kArray = key.split(TOPIC_QUEUEID_SEPARATOR);
 if (2 == kArray.length) {
 String topic = kArray[0];
 int queueId = Integer.parseInt(kArray[1]);
 final long offset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
 try {
 // 唤醒
 this.notifyMessageArriving(topic, queueId, offset);
 } catch (Throwable e) {
 log.error("check hold request failed. topic={}, queueId={}", topic, queueId, e);
 }
 }
 }
 }

 public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset) {
 notifyMessageArriving(topic, queueId, maxOffset, null, 0, null, null);
 }

 public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset, final Long tagsCode,
 long msgStoreTime, byte[] filterBitMap, Map<String, String> properties) {
 String key = this.buildKey(topic, queueId);
 // 取出等待的channel（请求）
 ManyPullRequest mpr = this.pullRequestTable.get(key);
 if (mpr != null) {
 List<PullRequest> requestList = mpr.cloneListAndClear();
 if (requestList != null) {
 List<PullRequest> replayList = new ArrayList<PullRequest>();
 // 循环所有的请求
 for (PullRequest request : requestList) {
 long newestOffset = maxOffset;
 if (newestOffset <= request.getPullFromThisOffset()) {
 newestOffset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId);
 }

 if (newestOffset > request.getPullFromThisOffset()) {
 boolean match = request.getMessageFilter().isMatchedByConsumeQueue(tagsCode,
 new ConsumeQueueExt.CqExtUnit(tagsCode, msgStoreTime, filterBitMap));
 // match by bit map, need eval again when properties is not null.
 if (match && properties != null) {
 match = request.getMessageFilter().isMatchedByCommitLog(null, properties);
 }
 // 如果匹配上
 if (match) {
 try {
 // 唤醒request
 this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
 request.getRequestCommand());
 } catch (Throwable e) {
 log.error("execute request when wakeup failed.", e);
 }
 continue;
 }
 }

 if (System.currentTimeMillis() >= (request.getSuspendTimestamp() + request.getTimeoutMillis())) {
 try {
 this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(),
 request.getRequestCommand());
 } catch (Throwable e) {
 log.error("execute request when wakeup failed.", e);
 }
 continue;
 }

 replayList.add(request);
 }

 if (!replayList.isEmpty()) {
 mpr.addPullRequest(replayList);
 }
 }
 }
 }
}

ManyPullRequest里存储的实际为PullRequest数组

public class ManyPullRequest {
 private final ArrayList<PullRequest> pullRequestList = new ArrayList<>();
	...
}

PullRequest实际存储NIO请求

public class PullRequest {
 private final RemotingCommand requestCommand;
 /** NIO channel */
 private final Channel clientChannel;
 private final long timeoutMillis;
 private final long suspendTimestamp;
 private final long pullFromThisOffset;
 private final SubscriptionData subscriptionData;
 private final MessageFilter messageFilter;
	...
}

`DefaultMQPullConsumer`消息拉取模式

实现类DefaultMQPullConsumer
主要逻辑
- 获取Message Queue并遍历
- 维护OffsetStore
- 根据不同消息状态做不同处理

生产者样例

public class Producer {
 public static void main(String[] args) throws MQClientException, InterruptedException {
 	
 	String group_name = "test_pull_producer_name";
 DefaultMQProducer producer = new DefaultMQProducer(group_name);
 producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 producer.start();
 
 for (int i = 0; i < 10; i++) {
 try {
 Message msg = new Message("test_pull_topic",// topic
 		"TagA",// tag
 		("Hello RocketMQ " + i).getBytes()// body
 );
 // 配置延迟
 msg.setDelayTimeLevel(2);
 
 SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
 // 外部方法的arg会传递到Object中
 @Override
 public MessageQueue select(List<MessageQueue> list, Message message, Object o) {
 Integer queueNumber = (Integer) o;
 return list.get(queueNumber);
 }
 },1);

 SendStatus status = sendResult.getSendStatus();
 System.out.println(status);

 Thread.sleep(1000);
 }
 catch (Exception e) {
 e.printStackTrace();
 Thread.sleep(3000);
 }
 }
 producer.shutdown();
 }
}

发送消息后的结果，在队列1有45条消息

使用普通Pull轮询模式

public class PullConsumer {
	//Map<key, value> key为指定的队列，value为这个队列拉取数据的最后位置
 private static final Map<MessageQueue, Long> offseTable = new HashMap<MessageQueue, Long>();
 
 public static void main(String[] args) throws MQClientException {
 	
 	String group_name = "test_pull_consumer_name";
 DefaultMQPullConsumer consumer = new DefaultMQPullConsumer(group_name);
 consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 consumer.start();
 System.err.println("consumer start");
 //	从TopicTest这个主题去获取所有的队列（默认会有4个队列）
 Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("test_pull_topic");
 //	遍历每一个队列，进行拉取数据
 for (MessageQueue mq : mqs) {
 System.out.println("Consume from the queue: " + mq);
 
 SINGLE_MQ: while (true) {
 try {
 	//	从queue中获取数据，从什么位置开始拉取数据 单次对多拉取32条记录
 PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
 System.out.println(pullResult);
 System.out.println(pullResult.getPullStatus());
 System.out.println();
 // 客户端自行维护Offset
 putMessageQueueOffset(mq, pullResult.getNextBeginOffset());
 switch (pullResult.getPullStatus()) {
	 case FOUND:
	 	List<MessageExt> list = pullResult.getMsgFoundList();
	 	for(MessageExt msg : list){
	 		System.out.println(new String(msg.getBody()));
	 	}
	 break;
	 case NO_MATCHED_MSG:
	 break;
	 case NO_NEW_MSG:
	 	System.out.println("没有新的数据啦...");
	 break SINGLE_MQ;
	 case OFFSET_ILLEGAL:
	 break;
	 default:
	 break;
 }
 }
 catch (Exception e) {
 e.printStackTrace();
 }
 }
 }
 consumer.shutdown();
 }
 
 
 private static void putMessageQueueOffset(MessageQueue mq, long offset) {
 offseTable.put(mq, offset);
 }
 
 
 private static long getMessageQueueOffset(MessageQueue mq) {
 Long offset = offseTable.get(mq);
 if (offset != null)
 return offset;
 return 0;
 }
}

消费结果，其会随机选择队列读取，导致服务端积压的消息得不到有效的消费

而且该模式需要客户端编码实现Offset存储

Pull定时任务实现样例

public class PullScheduleService {

 public static void main(String[] args) throws MQClientException {
 	
 	String group_name = "test_pull_consumer_name";
 	// 使用定时任务
 final MQPullConsumerScheduleService scheduleService = new MQPullConsumerScheduleService(group_name);
 
 scheduleService.getDefaultMQPullConsumer().setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
 
 scheduleService.setMessageModel(MessageModel.CLUSTERING);
 
 scheduleService.registerPullTaskCallback("test_pull_topic", new PullTaskCallback() {
 
 @Override
 public void doPullTask(MessageQueue mq, PullTaskContext context) {
 MQPullConsumer consumer = context.getPullConsumer();
 System.err.println("-------------- queueId: " + mq.getQueueId() + "-------------");
 try {
 // 获取从哪里拉取
 long offset = consumer.fetchConsumeOffset(mq, false);
 if (offset < 0)
 offset = 0;
 
 PullResult pullResult = consumer.pull(mq, "*", offset, 32);
 switch (pullResult.getPullStatus()) {
 case FOUND:
 	List<MessageExt> list = pullResult.getMsgFoundList();
 	for(MessageExt msg : list){
 		//消费数据...
 		System.out.println(new String(msg.getBody()));
 	}
 break;
 case NO_MATCHED_MSG:
 break;
 case NO_NEW_MSG:
 case OFFSET_ILLEGAL:
 break;
 default:
 break;
 }
 // 客户端自动更新Offset，无需本地持久化维护
 consumer.updateConsumeOffset(mq, pullResult.getNextBeginOffset());
 // 设置再过3000ms后重新拉取
 context.setPullNextDelayTimeMillis(3000);
 
 }
 catch (Exception e) {
 e.printStackTrace();
 }
 }
 });
 
 scheduleService.start();
 }
}

运行结果，由于采用多线程读取所有队列，所以当服务端有消息时可以及时获取

实现原理

MQPullConsumerScheduleService中将PullTaskCallback方法作为Map存储
并使用ScheduledThreadPoolExecutor作为线程池，配置线程数为10
线程通过调度taskTable中的PullTaskImpl任务，实现方法回调

package org.apache.rocketmq.client.consumer;

public class MQPullConsumerScheduleService {
 private final InternalLogger log = ClientLogger.getLog();
 private final MessageQueueListener messageQueueListener = new MessageQueueListenerImpl();
 private final ConcurrentMap<MessageQueue, PullTaskImpl> taskTable =
 new ConcurrentHashMap<MessageQueue, PullTaskImpl>();
 private DefaultMQPullConsumer defaultMQPullConsumer;
 private int pullThreadNums = 20;
 private ConcurrentMap<String /* topic */, PullTaskCallback> callbackTable =
 new ConcurrentHashMap<String, PullTaskCallback>();
 private ScheduledThreadPoolExecutor scheduledThreadPoolExecutor;

 public MQPullConsumerScheduleService(final String consumerGroup) {
 this.defaultMQPullConsumer = new DefaultMQPullConsumer(consumerGroup);
 this.defaultMQPullConsumer.setMessageModel(MessageModel.CLUSTERING);
 }

 public void putTask(String topic, Set<MessageQueue> mqNewSet) {
 Iterator<Entry<MessageQueue, PullTaskImpl>> it = this.taskTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<MessageQueue, PullTaskImpl> next = it.next();
 if (next.getKey().getTopic().equals(topic)) {
 if (!mqNewSet.contains(next.getKey())) {
 next.getValue().setCancelled(true);
 it.remove();
 }
 }
 }

 for (MessageQueue mq : mqNewSet) {
 if (!this.taskTable.containsKey(mq)) {
 // 需要调度的任务
 PullTaskImpl command = new PullTaskImpl(mq);
 this.taskTable.put(mq, command);
 // 使用多线程调度
 this.scheduledThreadPoolExecutor.schedule(command, 0, TimeUnit.MILLISECONDS);

 }
 }
 }

 public void start() throws MQClientException {
 final String group = this.defaultMQPullConsumer.getConsumerGroup();
 this.scheduledThreadPoolExecutor = new ScheduledThreadPoolExecutor(
 this.pullThreadNums,
 new ThreadFactoryImpl("PullMsgThread-" + group)
 );

 this.defaultMQPullConsumer.setMessageQueueListener(this.messageQueueListener);

 this.defaultMQPullConsumer.start();

 log.info("MQPullConsumerScheduleService start OK, {} {}",
 this.defaultMQPullConsumer.getConsumerGroup(), this.callbackTable);
 }

 public void registerPullTaskCallback(final String topic, final PullTaskCallback callback) {
 // 放入回调任务集合
 this.callbackTable.put(topic, callback);
 this.defaultMQPullConsumer.registerMessageQueueListener(topic, null);
 }

 class MessageQueueListenerImpl implements MessageQueueListener {
 @Override
 public void messageQueueChanged(String topic, Set<MessageQueue> mqAll, Set<MessageQueue> mqDivided) {
 MessageModel messageModel =
 MQPullConsumerScheduleService.this.defaultMQPullConsumer.getMessageModel();
 switch (messageModel) {
 case BROADCASTING:
 MQPullConsumerScheduleService.this.putTask(topic, mqAll);
 break;
 case CLUSTERING:
 MQPullConsumerScheduleService.this.putTask(topic, mqDivided);
 break;
 default:
 break;
 }
 }
 }

 class PullTaskImpl implements Runnable {
 private final MessageQueue messageQueue;
 private volatile boolean cancelled = false;

 public PullTaskImpl(final MessageQueue messageQueue) {
 this.messageQueue = messageQueue;
 }

 @Override
 public void run() {
 String topic = this.messageQueue.getTopic();
 if (!this.isCancelled()) {
 // 获取回调任务
 PullTaskCallback pullTaskCallback =
 MQPullConsumerScheduleService.this.callbackTable.get(topic);
 if (pullTaskCallback != null) {
 final PullTaskContext context = new PullTaskContext();
 context.setPullConsumer(MQPullConsumerScheduleService.this.defaultMQPullConsumer);
 try {
 // 执行客户端实现的回调任务
 pullTaskCallback.doPullTask(this.messageQueue, context);
 } catch (Throwable e) {
 context.setPullNextDelayTimeMillis(1000);
 log.error("doPullTask Exception", e);
 }

 if (!this.isCancelled()) {
 MQPullConsumerScheduleService.this.scheduledThreadPoolExecutor.schedule(this,
 context.getPullNextDelayTimeMillis(), TimeUnit.MILLISECONDS);
 } else {
 log.warn("The Pull Task is cancelled after doPullTask, {}", messageQueue);
 }
 } else {
 log.warn("Pull Task Callback not exist , {}", topic);
 }
 } else {
 log.warn("The Pull Task is cancelled, {}", messageQueue);
 }
 }
 }
}

RocketMQ学习笔记3——生产者核心

Wed, 23 Dec 2020 19:09:00 +0800

配置参数

producerGroup：组名
createTopicKey：创建Topic（不允许使用，应当由架构设计定义）
defaultTopicQueueNums：创建topic时默认的队列数量（默认为4）
sendMsgTimeout：发送消息时的超时时间（默认值：3000，单位：毫秒）
compressMsgBodyOverHowmuch ：消息压缩字节（默认4096）
retryTimesWhenSendFailed ：同步模式下，在返回发送失败之前，内部尝试重新发送消息的最大次数。（默认值：2）
retryTimesWhenSendAsyncFailed：异步模式下，在发送失败之前，内部尝试重新发送消息的最大次数。（默认值：2）
retryAnotherBrokerWhenNotStoreOK：同步模式下，消息保存失败时是否重试其他broker 。（默认值：false ）
maxMessageSize：最大消息限制。（默认值：1024 * 1024 * 4，单位：字节）

主从同步机制

主从同步
同步信息：数据内容+元数据信息
元数据（配置信息）同步：Broker角色识别，为Slave则启动同步任务
- 通过Netty传输
- 实质为定时任务
消息同步（通过Socket）：HAService、HAconnection、WaitNotifyObject
- 通过原生Socket传输，性能高，数据包支持容量大

元数据同步源码分析

元信息同步工程源码位于broker中

发起定时任务源码BrokerController

package org.apache.rocketmq.broker;

public class BrokerController {
 public boolean initialize() throws CloneNotSupportedException {
 ...
 if (result) {
 ...
 // 进行角色判断
 if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) {
 if (this.messageStoreConfig.getHaMasterAddress() != null && this.messageStoreConfig.getHaMasterAddress().length() >= 6) {
 this.messageStore.updateHaMasterAddress(this.messageStoreConfig.getHaMasterAddress());
 this.updateMasterHAServerAddrPeriodically = false;
 } else {
 this.updateMasterHAServerAddrPeriodically = true;
 }
 // 启动定时任务进行元信息同步
 this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

 @Override
 public void run() {
 try {
 BrokerController.this.slaveSynchronize.syncAll();
 } catch (Throwable e) {
 log.error("ScheduledTask syncAll slave exception", e);
 }
 }
 }, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);
 } else {
 this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

 @Override
 public void run() {
 try {
 BrokerController.this.printMasterAndSlaveDiff();
 } catch (Throwable e) {
 log.error("schedule printMasterAndSlaveDiff error.", e);
 }
 }
 }, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);
 }
 }
 ...
 }
}

syncAll方法实现

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 public void syncAll() {
 // 同步Topic配置信息
 this.syncTopicConfig();
 // 同步消费者偏移量
 this.syncConsumerOffset();
 // 同步延迟偏移量
 this.syncDelayOffset();
 // 同步订阅组配置信息
 this.syncSubscriptionGroupConfig();
 }
 ...
}

同步Topic配置信息源码

syncTopicConfig方法实现如下

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 private void syncTopicConfig() {
 String masterAddrBak = this.masterAddr;
 if (masterAddrBak != null) {
 try {
 TopicConfigSerializeWrapper topicWrapper =
 this.brokerController.getBrokerOuterAPI().getAllTopicConfig(masterAddrBak);
 if (!this.brokerController.getTopicConfigManager().getDataVersion()
 .equals(topicWrapper.getDataVersion())) {

 this.brokerController.getTopicConfigManager().getDataVersion()
 .assignNewOne(topicWrapper.getDataVersion());
 this.brokerController.getTopicConfigManager().getTopicConfigTable().clear();
 this.brokerController.getTopicConfigManager().getTopicConfigTable()
 .putAll(topicWrapper.getTopicConfigTable());
 this.brokerController.getTopicConfigManager().persist();

 log.info("Update slave topic config from master, {}", masterAddrBak);
 }
 } catch (Exception e) {
 log.error("SyncTopicConfig Exception, {}", masterAddrBak, e);
 }
 }
 }
 ...
}

其中getAllTopicConfig方法实现如下

package org.apache.rocketmq.broker.out;

public class BrokerOuterAPI {
 ...
 public TopicConfigSerializeWrapper getAllTopicConfig(
 final String addr) throws RemotingConnectException, RemotingSendRequestException,
 RemotingTimeoutException, InterruptedException, MQBrokerException {
 RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_TOPIC_CONFIG, null);
 // 发起远程同步调用
 RemotingCommand response = this.remotingClient.invokeSync(MixAll.brokerVIPChannel(true, addr), request, 3000);
 assert response != null;
 switch (response.getCode()) {
 case ResponseCode.SUCCESS: {
 return TopicConfigSerializeWrapper.decode(response.getBody(), TopicConfigSerializeWrapper.class);
 }
 default:
 break;
 }

 throw new MQBrokerException(response.getCode(), response.getRemark());
 }
 ...
}

invokeSync通过调用封装过的Netty客户端实现

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingClient extends NettyRemotingAbstract implements RemotingClient {
 ...
 /**
 * 同步请求
 * @param addr 请求地址
 * @param request 请求命令
 * @param timeoutMillis 超时时间
 */
 @Override
 public RemotingCommand invokeSync(String addr, final RemotingCommand request, long timeoutMillis)
 throws InterruptedException, RemotingConnectException, RemotingSendRequestException, RemotingTimeoutException {
 long beginStartTime = System.currentTimeMillis();
 final Channel channel = this.getAndCreateChannel(addr);
 if (channel != null && channel.isActive()) {
 try {
 doBeforeRpcHooks(addr, request);
 // 算出极限耗时代价
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTime) {
 // 如果同步等待时长 < 极限耗时 说明请求不成立，抛出异常
 throw new RemotingTimeoutException("invokeSync call timeout");
 }
 // 同步请求实现
 RemotingCommand response = this.invokeSyncImpl(channel, request, timeoutMillis - costTime);
 doAfterRpcHooks(RemotingHelper.parseChannelRemoteAddr(channel), request, response);
 return response;
 } catch (RemotingSendRequestException e) {
 log.warn("invokeSync: send request exception, so close the channel[{}]", addr);
 this.closeChannel(addr, channel);
 throw e;
 } catch (RemotingTimeoutException e) {
 if (nettyClientConfig.isClientCloseSocketIfTimeout()) {
 this.closeChannel(addr, channel);
 log.warn("invokeSync: close socket because of timeout, {}ms, {}", timeoutMillis, addr);
 }
 log.warn("invokeSync: wait response timeout exception, the channel[{}]", addr);
 throw e;
 }
 } else {
 this.closeChannel(addr, channel);
 throw new RemotingConnectException(addr);
 }
 }
 ...
}

invokeSyncImpl实现逻辑

package org.apache.rocketmq.remoting.netty;

public abstract class NettyRemotingAbstract {
 ...
 public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
 final long timeoutMillis)
 throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
 final int opaque = request.getOpaque();

 try {
 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
 this.responseTable.put(opaque, responseFuture);
 final SocketAddress addr = channel.remoteAddress();
 // 使用netty channel
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 // 轮训监听
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 } else {
 responseFuture.setSendRequestOK(false);
 }
 // 异常情况
 responseTable.remove(opaque);
 responseFuture.setCause(f.cause());
 responseFuture.putResponse(null);
 log.warn("send a request command to channel <" + addr + "> failed.");
 }
 });

 RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
 if (null == responseCommand) {
 if (responseFuture.isSendRequestOK()) {
 throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
 responseFuture.getCause());
 } else {
 throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
 }
 }

 return responseCommand;
 } finally {
 this.responseTable.remove(opaque);
 }
 }
	...
}

同步消费者偏移量源码

其中syncConsumerOffset方法实现如下

package org.apache.rocketmq.broker.slave;

public class SlaveSynchronize {
 ...
 private void syncConsumerOffset() {
 String masterAddrBak = this.masterAddr;
 if (masterAddrBak != null) {
 try {
 ConsumerOffsetSerializeWrapper offsetWrapper =
 this.brokerController.getBrokerOuterAPI().getAllConsumerOffset(masterAddrBak);
 this.brokerController.getConsumerOffsetManager().getOffsetTable()
 .putAll(offsetWrapper.getOffsetTable());
 this.brokerController.getConsumerOffsetManager().persist();
 log.info("Update slave consumer offset from master, {}", masterAddrBak);
 } catch (Exception e) {
 log.error("SyncConsumerOffset Exception, {}", masterAddrBak, e);
 }
 }
 }
 ...
}

getAllConsumerOffset实现逻辑

package org.apache.rocketmq.broker.out;

public class BrokerOuterAPI {
 ...
 public ConsumerOffsetSerializeWrapper getAllConsumerOffset(
 final String addr) throws InterruptedException, RemotingTimeoutException,
 RemotingSendRequestException, RemotingConnectException, MQBrokerException {
 // 封装CMD指令向主节点发起同步offset信息请求，同步时间上限为3000ms
 RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_CONSUMER_OFFSET, null);
 RemotingCommand response = this.remotingClient.invokeSync(addr, request, 3000);
 assert response != null;
 switch (response.getCode()) {
 case ResponseCode.SUCCESS: {
 return ConsumerOffsetSerializeWrapper.decode(response.getBody(), ConsumerOffsetSerializeWrapper.class);
 }
 default:
 break;
 }

 throw new MQBrokerException(response.getCode(), response.getRemark());
 }
 ...
}

封装的CMD指令

package org.apache.rocketmq.common.protocol;

public class RequestCode {
 ...
 public static final int NOTIFY_CONSUMER_IDS_CHANGED = 40;

 public static final int LOCK_BATCH_MQ = 41;

 public static final int UNLOCK_BATCH_MQ = 42;
 public static final int GET_ALL_CONSUMER_OFFSET = 43;

 public static final int GET_ALL_DELAY_OFFSET = 45;
 ...
}

远程命令对象构建，构建后通过remotingClient发送即可

package org.apache.rocketmq.remoting.protocol;

public class RemotingCommand {
 ...
	public static RemotingCommand createRequestCommand(int code, CommandCustomHeader customHeader) {
 RemotingCommand cmd = new RemotingCommand();
 cmd.setCode(code);
 cmd.customHeader = customHeader;
 setCmdVersion(cmd);
 return cmd;
 }
 ...
}

消息同步源码

消息同步工程源码位于store中

HAConnection实现消息读写

HAservice包含节点连接实现HAClient

逻辑

Master节点：

AcceptSocketService：接收Slave节点连接
HAConnection：
- ReadSocketService：读取来自Slave节点的数据
- WriteSocketService：写入Slave节点的数据

Slave节点：

HAService：
- HAClient：对Master节点连接，读写数据

通信协议：Master节点与Slave节点通信协议较简单，主要如下

对象	用途	第几位	字段	数据类型	字节数	说明
`Slave`=>`Master`	上报`CommitLog`已经同步的物理位置	0	`maxPhyOffset`	Long	8	`CommitLog`最大物理位置
`Master`=>`Slave`	传输新的`CommitLog`数据	0	`fromPhyOffset`	Long	8	传输`CommitLog`开始的物理位置
		1	`size`	Int	4	传输`CommitLog`数据长度
		2	`body`	Bytes		传输`CommitLog`数据

连接Master源码

package org.apache.rocketmq.store.ha;

public class HAService {
 ...
 class HAClient extends ServiceThread {
 /**
 * 客户端连接Master，使用NIO函数，提高效率
 * @return
 * @throws ClosedChannelException
 */
 private boolean connectMaster() throws ClosedChannelException {
 if (null == socketChannel) {
 String addr = this.masterAddress.get();
 if (addr != null) {

 SocketAddress socketAddress = RemotingUtil.string2SocketAddress(addr);
 if (socketAddress != null) {
 this.socketChannel = RemotingUtil.connect(socketAddress);
 if (this.socketChannel != null) {
 this.socketChannel.register(this.selector, SelectionKey.OP_READ);
 }
 }
 }

 this.currentReportedOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();

 this.lastWriteTimestamp = System.currentTimeMillis();
 }

 return this.socketChannel != null;
 }
 
 /**
 * Client(Slave)主循环，实现不断从Master传输CommitLog数据
 * 上传Master自己本地的CommitLog已经同步物理位置
 */
 @Override
 public void run() {
 log.info(this.getServiceName() + " service started");

 while (!this.isStopped()) {
 try {
 if (this.connectMaster()) {

 if (this.isTimeToReportOffset()) {
 // 满足上报间隔，上报到Master进度
 // 固定间隔（默认为5s）向Master上报Slave本地CommitLog已经同步到的物理位置
 boolean result = this.reportSlaveMaxOffset(this.currentReportedOffset);
 if (!result) {
 this.closeMaster();
 }
 }

 this.selector.select(1000);
 // 处理Master传输给Slave的CommitLog数据
 boolean ok = this.processReadEvent();
 if (!ok) {
 this.closeMaster();
 }
 // 若进度有变化，上报到Master进度
 if (!reportSlaveMaxOffsetPlus()) {
 continue;
 }
 // Master超时未返回，关闭连接
 long interval =
 HAService.this.getDefaultMessageStore().getSystemClock().now()
 - this.lastWriteTimestamp;
 if (interval > HAService.this.getDefaultMessageStore().getMessageStoreConfig()
 .getHaHousekeepingInterval()) {
 log.warn("HAClient, housekeeping, found this connection[" + this.masterAddress
 + "] expired, " + interval);
 this.closeMaster();
 log.warn("HAClient, master not response some time, so close connection");
 }
 } else {
 this.waitForRunning(1000 * 5);
 }
 } catch (Exception e) {
 log.warn(this.getServiceName() + " service has exception. ", e);
 this.waitForRunning(1000 * 5);
 }
 }

 log.info(this.getServiceName() + " service end");
 }
 ...
 }
 ...
}

RemotingUtil使用原生Socket实现

package org.apache.rocketmq.remoting.common;

public class RemotingUtil {
 public static SocketAddress string2SocketAddress(final String addr) {
 String[] s = addr.split(":");
 InetSocketAddress isa = new InetSocketAddress(s[0], Integer.parseInt(s[1]));
 return isa;
 }
 
 public static SocketChannel connect(SocketAddress remote, final int timeoutMillis) {
 SocketChannel sc = null;
 try {
 // 打开channel
 sc = SocketChannel.open();
 // 设置同步阻塞，可能存在bug
 sc.configureBlocking(true);
 // 设置关闭Socket的延迟事件，那么当线程执行Socket的close方法时，会进入阻塞状态
 // 直到Socket发送完所有剩余数据，或者超过了方法设置的延迟时间，才从close方法返回
 sc.socket().setSoLinger(false, -1);
 // 禁止使用Nagle算法，使用小数据传输
 // Nagle算法通过将缓冲区的小封包自动相连，组成较大的包
 sc.socket().setTcpNoDelay(true);
 // 设置接收缓冲区
 sc.socket().setReceiveBufferSize(1024 * 64);
 // 设置发送缓冲区
 sc.socket().setSendBufferSize(1024 * 64);
 // 连接
 sc.socket().connect(remote, timeoutMillis);
 // bug？又设置非同步阻塞了
 sc.configureBlocking(false);
 return sc;
 } catch (Exception e) {
 if (sc != null) {
 try {
 sc.close();
 } catch (IOException e1) {
 e1.printStackTrace();
 }
 }
 }

 return null;
 }
 ...
}

核心读请求实现

package org.apache.rocketmq.store.ha;

public class HAService {
 ...
 class HAClient extends ServiceThread {
 private boolean processReadEvent() {
 int readSizeZeroTimes = 0;
 while (this.byteBufferRead.hasRemaining()) {
 try {
 int readSize = this.socketChannel.read(this.byteBufferRead);
 if (readSize > 0) {
 lastWriteTimestamp = HAService.this.defaultMessageStore.getSystemClock().now();
 readSizeZeroTimes = 0;
 boolean result = this.dispatchReadRequest();
 if (!result) {
 log.error("HAClient, dispatchReadRequest error");
 return false;
 }
 } else if (readSize == 0) {
 if (++readSizeZeroTimes >= 3) {
 break;
 }
 } else {
 log.info("HAClient, processReadEvent read socket < 0");
 return false;
 }
 } catch (IOException e) {
 log.info("HAClient, processReadEvent read socket exception", e);
 return false;
 }
 }

 return true;
 }
 
 /**
 * 核心读请求实现
 * 
 * 读取Master传输的CommitLog数据，并返回是否正确
 * 如果读取到数据，写入CommitLog
 * 异常原因：
 * 1. Master传输来的数据开始位置offset(masterPhyOffset)不等于Slave的CommitLog数最大offset(slavePhyOffset)
 * 2. 上报到Master进度失败
 * @return 是否正确
 */
 private boolean dispatchReadRequest() {
 final int msgHeaderSize = 8 + 4; // phyoffset + size
 int readSocketPos = this.byteBufferRead.position();

 while (true) {
 // 读取到请求数据
 int diff = this.byteBufferRead.position() - this.dispatchPostion;
 if (diff >= msgHeaderSize) {
 // 读取masterPhyOffset、bodySize
 long masterPhyOffset = this.byteBufferRead.getLong(this.dispatchPostion);
 int bodySize = this.byteBufferRead.getInt(this.dispatchPostion + 8);
 // 设置Slave节点CommitLog文件最大的offset位置
 long slavePhyOffset = HAService.this.defaultMessageStore.getMaxPhyOffset();
 // 校验Master传输来的数据开始位置offset和slave的commitLog数最大offset是否相同
 if (slavePhyOffset != 0) {
 if (slavePhyOffset != masterPhyOffset) {
 log.error("master pushed offset not equal the max phy offset in slave, SLAVE: "
 + slavePhyOffset + " MASTER: " + masterPhyOffset);
 return false;
 }
 }
 // 读取消息
 if (diff >= (msgHeaderSize + bodySize)) {
 // 写入CommitLog
 byte[] bodyData = new byte[bodySize];
 this.byteBufferRead.position(this.dispatchPostion + msgHeaderSize);
 this.byteBufferRead.get(bodyData);

 HAService.this.defaultMessageStore.appendToCommitLog(masterPhyOffset, bodyData);
 // 设置处理到的位置
 this.byteBufferRead.position(readSocketPos);
 this.dispatchPostion += msgHeaderSize + bodySize;
 // 上报到Master进度
 if (!reportSlaveMaxOffsetPlus()) {
 return false;
 }
 // 继续读数据
 continue;
 }
 }
 // 空间写满，重新分配空间
 if (!this.byteBufferRead.hasRemaining()) {
 this.reallocateByteBuffer();
 }

 break;
 }

 return true;
 }
 ...
 }
 ...
}

同步\异步消息发送

实现分类

同步消息发送
- 发送：producer.send(Message msg)
- 核心实现：DefaultMQProducerImpl
异步消息发送
- 发送：producer.send(Message msg,SendCallback sendCallback)
- 核心实现：DefaultMQProducerImpl

实例

public class Producer {

	public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
		
		DefaultMQProducer producer = new DefaultMQProducer("test_quick_producer_name");
		
	producer.setNamesrvAddr(Const.NAMESRV_ADDR_MASTER_SLAVE);
		
		producer.start();
		
		for(int i = 0 ; i <5; i ++) {
			//	1.	创建消息
			Message message = new Message("test_quick_topic",	//	主题
					"TagA", //	标签
					"key" + i,	// 	用户自定义的key ,唯一的标识
					("Hello RocketMQ" + i).getBytes());	//	消息内容实体（byte[]）
			// 异步发送消息
			producer.send(message, new SendCallback() {
				@Override
				public void onSuccess(SendResult sendResult) {
					System.err.println("msgId: " + sendResult.getMsgId() + ", status: " + sendResult.getSendStatus());
					producer.shutdown();
				}
				@Override
				public void onException(Throwable e) {
					e.printStackTrace();
					System.err.println("------发送失败");
				}
			});
		}	
	}
}

同步消息发送源码分析

发送消息方法

package org.apache.rocketmq.client.producer;

public class DefaultMQProducer extends ClientConfig implements MQProducer {
 ...
 @Override
 public SendResult send(Message msg,
 long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 return this.defaultMQProducerImpl.send(msg, timeout);
 }
}

模式枚举

package org.apache.rocketmq.client.impl;

public enum CommunicationMode {
 SYNC,
 ASYNC,
 // 单向，不需要响应
 ONEWAY,
}

消息发送实现类

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public SendResult send(Message msg,
 long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 return this.sendDefaultImpl(msg, CommunicationMode.SYNC, null, timeout);
 }
 
 private SendResult sendDefaultImpl(
 Message msg,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final long timeout
 ) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 // 校验运行状态
 this.makeSureStateOK();
 // 检查消息格式
 Validators.checkMessage(msg, this.defaultMQProducer);
 // 回调ID，用户标记是否同一次消息
 final long invokeID = random.nextLong();
 // 开始发送消息的时间
 long beginTimestampFirst = System.currentTimeMillis();
 long beginTimestampPrev = beginTimestampFirst;
 long endTimestamp = beginTimestampFirst;
 // 查找topic
 TopicPublishInfo topicPublishInfo = this.tryToFindTopicPublishInfo(msg.getTopic());
 if (topicPublishInfo != null && topicPublishInfo.ok()) {
 boolean callTimeout = false;
 // 要发送的消息队列
 MessageQueue mq = null;
 Exception exception = null;
 SendResult sendResult = null;
 // 重试次数，从配置参数中读取
 int timesTotal = communicationMode == CommunicationMode.SYNC ? 1 + this.defaultMQProducer.getRetryTimesWhenSendFailed() : 1;
 int times = 0;
 // 存储每次发送消息选择的broker名称
 String[] brokersSent = new String[timesTotal];
 for (; times < timesTotal; times++) {
 // 最后一次消息发送的brokerName进行存储
 String lastBrokerName = null == mq ? null : mq.getBrokerName();
 // 消息队列选择策略
 MessageQueue mqSelected = this.selectOneMessageQueue(topicPublishInfo, lastBrokerName);
 if (mqSelected != null) {
 mq = mqSelected;
 brokersSent[times] = mq.getBrokerName();
 try {
 beginTimestampPrev = System.currentTimeMillis();
 // 计算代价
 long costTime = beginTimestampPrev - beginTimestampFirst;
 if (timeout < costTime) {
 callTimeout = true;
 break;
 }
 // 消息发送实现
 sendResult = this.sendKernelImpl(msg, mq, communicationMode, sendCallback, topicPublishInfo, timeout - costTime);
 endTimestamp = System.currentTimeMillis();
 // 更新broker可用信息（容错）
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
 switch (communicationMode) {
 case ASYNC:
 return null;
 case ONEWAY:
 return null;
 case SYNC:
 // 同步发送等待返回结果
 if (sendResult.getSendStatus() != SendStatus.SEND_OK) {
 if (this.defaultMQProducer.isRetryAnotherBrokerWhenNotStoreOK()) {
 continue;
 }
 }

 return sendResult;
 default:
 break;
 }
 } catch (RemotingException e) {
 endTimestamp = System.currentTimeMillis();
 // 抛异常则更新broker容错信息
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 continue;
 } catch (MQClientException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 continue;
 } catch (MQBrokerException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
 log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());
 exception = e;
 // 抛出如下异常则进行重试
 switch (e.getResponseCode()) {
 case ResponseCode.TOPIC_NOT_EXIST:
 case ResponseCode.SERVICE_NOT_AVAILABLE:
 case ResponseCode.SYSTEM_ERROR:
 case ResponseCode.NO_PERMISSION:
 case ResponseCode.NO_BUYER_ID:
 case ResponseCode.NOT_IN_CURRENT_UNIT:
 continue;
 default:
 if (sendResult != null) {
 return sendResult;
 }

 throw e;
 }
 } catch (InterruptedException e) {
 endTimestamp = System.currentTimeMillis();
 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
 log.warn(String.format("sendKernelImpl exception, throw exception, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
 log.warn(msg.toString());

 log.warn("sendKernelImpl exception", e);
 log.warn(msg.toString());
 throw e;
 }
 } else {
 break;
 }
 }

 if (sendResult != null) {
 return sendResult;
 }

 String info = String.format("Send [%d] times, still failed, cost [%d]ms, Topic: %s, BrokersSent: %s",
 times,
 System.currentTimeMillis() - beginTimestampFirst,
 msg.getTopic(),
 Arrays.toString(brokersSent));

 info += FAQUrl.suggestTodo(FAQUrl.SEND_MSG_FAILED);

 MQClientException mqClientException = new MQClientException(info, exception);
 if (callTimeout) {
 throw new RemotingTooMuchRequestException("sendDefaultImpl call timeout");
 }

 if (exception instanceof MQBrokerException) {
 mqClientException.setResponseCode(((MQBrokerException) exception).getResponseCode());
 } else if (exception instanceof RemotingConnectException) {
 mqClientException.setResponseCode(ClientErrorCode.CONNECT_BROKER_EXCEPTION);
 } else if (exception instanceof RemotingTimeoutException) {
 mqClientException.setResponseCode(ClientErrorCode.ACCESS_BROKER_TIMEOUT);
 } else if (exception instanceof MQClientException) {
 mqClientException.setResponseCode(ClientErrorCode.BROKER_NOT_EXIST_EXCEPTION);
 }

 throw mqClientException;
 }

 List<String> nsList = this.getmQClientFactory().getMQClientAPIImpl().getNameServerAddressList();
 if (null == nsList || nsList.isEmpty()) {
 throw new MQClientException(
 "No name server address, please set it." + FAQUrl.suggestTodo(FAQUrl.NAME_SERVER_ADDR_NOT_EXIST_URL), null).setResponseCode(ClientErrorCode.NO_NAME_SERVER_EXCEPTION);
 }

 throw new MQClientException("No route info of this topic, " + msg.getTopic() + FAQUrl.suggestTodo(FAQUrl.NO_TOPIC_ROUTE_INFO),
 null).setResponseCode(ClientErrorCode.NOT_FOUND_TOPIC_EXCEPTION);
 }
}

查找topic方法实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 private TopicPublishInfo tryToFindTopicPublishInfo(final String topic) {
 // 使用Map缓存
 TopicPublishInfo topicPublishInfo = this.topicPublishInfoTable.get(topic);
 if (null == topicPublishInfo || !topicPublishInfo.ok()) {
 // 当无可用topic发布信息时，主动向NameServer拉取一次
 this.topicPublishInfoTable.putIfAbsent(topic, new TopicPublishInfo());
 this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
 topicPublishInfo = this.topicPublishInfoTable.get(topic);
 }
 // 若可用
 if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) {
 return topicPublishInfo;
 } else {
 // 否则根据设置判断是否新建Topic
 this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, true, this.defaultMQProducer);
 topicPublishInfo = this.topicPublishInfoTable.get(topic);
 return topicPublishInfo;
 }
 }
}

客户端获取路由信息实现

package org.apache.rocketmq.client.impl.factory;

public class MQClientInstance {
 public boolean updateTopicRouteInfoFromNameServer(final String topic) {
 return updateTopicRouteInfoFromNameServer(topic, false, null);
 }
 
 public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault,
 DefaultMQProducer defaultMQProducer) {
 try {
 if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
 try {
 TopicRouteData topicRouteData;
 if (isDefault && defaultMQProducer != null) {
 topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(),
 1000 * 3);
 if (topicRouteData != null) {
 for (QueueData data : topicRouteData.getQueueDatas()) {
 int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
 data.setReadQueueNums(queueNums);
 data.setWriteQueueNums(queueNums);
 }
 }
 } else {
 topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, 1000 * 3);
 }
 if (topicRouteData != null) {
 TopicRouteData old = this.topicRouteTable.get(topic);
 boolean changed = topicRouteDataIsChange(old, topicRouteData);
 if (!changed) {
 changed = this.isNeedUpdateTopicRouteInfo(topic);
 } else {
 log.info("the topic[{}] route info changed, old[{}] ,new[{}]", topic, old, topicRouteData);
 }

 if (changed) {
 TopicRouteData cloneTopicRouteData = topicRouteData.cloneTopicRouteData();

 for (BrokerData bd : topicRouteData.getBrokerDatas()) {
 this.brokerAddrTable.put(bd.getBrokerName(), bd.getBrokerAddrs());
 }

 // Update Pub info
 {
 TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData);
 publishInfo.setHaveTopicRouterInfo(true);
 Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<String, MQProducerInner> entry = it.next();
 MQProducerInner impl = entry.getValue();
 if (impl != null) {
 impl.updateTopicPublishInfo(topic, publishInfo);
 }
 }
 }

 // Update sub info
 {
 Set<MessageQueue> subscribeInfo = topicRouteData2TopicSubscribeInfo(topic, topicRouteData);
 Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
 while (it.hasNext()) {
 Entry<String, MQConsumerInner> entry = it.next();
 MQConsumerInner impl = entry.getValue();
 if (impl != null) {
 impl.updateTopicSubscribeInfo(topic, subscribeInfo);
 }
 }
 }
 log.info("topicRouteTable.put. Topic = {}, TopicRouteData[{}]", topic, cloneTopicRouteData);
 this.topicRouteTable.put(topic, cloneTopicRouteData);
 return true;
 }
 } else {
 log.warn("updateTopicRouteInfoFromNameServer, getTopicRouteInfoFromNameServer return null, Topic: {}", topic);
 }
 } catch (Exception e) {
 if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX) && !topic.equals(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC)) {
 log.warn("updateTopicRouteInfoFromNameServer Exception", e);
 }
 } finally {
 this.lockNamesrv.unlock();
 }
 } else {
 log.warn("updateTopicRouteInfoFromNameServer tryLock timeout {}ms", LOCK_TIMEOUT_MILLIS);
 }
 } catch (InterruptedException e) {
 log.warn("updateTopicRouteInfoFromNameServer Exception", e);
 }

 return false;
 }
}

消息队列选择策略实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
 return this.mqFaultStrategy.selectOneMessageQueue(tpInfo, lastBrokerName);
 }
}

负载均衡消息选择策略实现

package org.apache.rocketmq.client.latency;

public class MQFaultStrategy {
 private final static InternalLogger log = ClientLogger.getLog();
 private final LatencyFaultTolerance<String> latencyFaultTolerance = new LatencyFaultToleranceImpl();
 // 发送消息延迟容错开关
 private boolean sendLatencyFaultEnable = false;
 // 消息发送时长的延迟级别数组
 private long[] latencyMax = {50L, 100L, 550L, 1000L, 2000L, 3000L, 15000L};
 // broker预估不可用时长级别数组
 private long[] notAvailableDuration = {0L, 0L, 30000L, 60000L, 120000L, 180000L, 600000L};
 /**
 * 负载均衡消息选择策略
 */
 public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
 if (this.sendLatencyFaultEnable) {
 try {
 // 获取可用队列
 int index = tpInfo.getSendWhichQueue().getAndIncrement();
 for (int i = 0; i < tpInfo.getMessageQueueList().size(); i++) {
 int pos = Math.abs(index++) % tpInfo.getMessageQueueList().size();
 if (pos < 0)
 pos = 0;
 MessageQueue mq = tpInfo.getMessageQueueList().get(pos);
 if (latencyFaultTolerance.isAvailable(mq.getBrokerName())) {
 if (null == lastBrokerName || mq.getBrokerName().equals(lastBrokerName))
 return mq;
 }
 }
 // 选择最优队列
 final String notBestBroker = latencyFaultTolerance.pickOneAtLeast();
 int writeQueueNums = tpInfo.getQueueIdByBroker(notBestBroker);
 if (writeQueueNums > 0) {
 final MessageQueue mq = tpInfo.selectOneMessageQueue();
 if (notBestBroker != null) {
 mq.setBrokerName(notBestBroker);
 mq.setQueueId(tpInfo.getSendWhichQueue().getAndIncrement() % writeQueueNums);
 }
 return mq;
 } else {
 latencyFaultTolerance.remove(notBestBroker);
 }
 } catch (Exception e) {
 log.error("Error occurred when selecting message queue", e);
 }

 return tpInfo.selectOneMessageQueue();
 }

 return tpInfo.selectOneMessageQueue(lastBrokerName);
 }
}

更新broker可用信息（容错）实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) {
 this.mqFaultStrategy.updateFaultItem(brokerName, currentLatency, isolation);
 }
}

与负载均衡消息选择策略位于相同类中

package org.apache.rocketmq.client.latency;

public class MQFaultStrategy {
 	 public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) {
 if (this.sendLatencyFaultEnable) {
 long duration = computeNotAvailableDuration(isolation ? 30000 : currentLatency);
 this.latencyFaultTolerance.updateFaultItem(brokerName, currentLatency, duration);
 }
 } 
}

消息发送实现

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 private SendResult sendKernelImpl(final Message msg,
 final MessageQueue mq,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final TopicPublishInfo topicPublishInfo,
 final long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 long beginStartTime = System.currentTimeMillis();
 // 获得地址
 String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
 if (null == brokerAddr) {
 tryToFindTopicPublishInfo(mq.getTopic());
 brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
 }

 SendMessageContext context = null;
 if (brokerAddr != null) {
 brokerAddr = MixAll.brokerVIPChannel(this.defaultMQProducer.isSendMessageWithVIPChannel(), brokerAddr);
 // 记录消息内容，因为下面逻辑可能改变消息内容，例如消息压缩
 byte[] prevBody = msg.getBody();
 try {
 //for MessageBatch,ID has been set in the generating process
 if (!(msg instanceof MessageBatch)) {
 MessageClientIDSetter.setUniqID(msg);
 }

 int sysFlag = 0;
 boolean msgBodyCompressed = false;
 if (this.tryToCompressMessage(msg)) {
 sysFlag |= MessageSysFlag.COMPRESSED_FLAG;
 msgBodyCompressed = true;
 }

 final String tranMsg = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
 if (tranMsg != null && Boolean.parseBoolean(tranMsg)) {
 sysFlag |= MessageSysFlag.TRANSACTION_PREPARED_TYPE;
 }
 // 消息发送前校验
 if (hasCheckForbiddenHook()) {
 CheckForbiddenContext checkForbiddenContext = new CheckForbiddenContext();
 checkForbiddenContext.setNameSrvAddr(this.defaultMQProducer.getNamesrvAddr());
 checkForbiddenContext.setGroup(this.defaultMQProducer.getProducerGroup());
 checkForbiddenContext.setCommunicationMode(communicationMode);
 checkForbiddenContext.setBrokerAddr(brokerAddr);
 checkForbiddenContext.setMessage(msg);
 checkForbiddenContext.setMq(mq);
 checkForbiddenContext.setUnitMode(this.isUnitMode());
 this.executeCheckForbiddenHook(checkForbiddenContext);
 }
 // 消息发送前回调处理
 if (this.hasSendMessageHook()) {
 context = new SendMessageContext();
 context.setProducer(this);
 context.setProducerGroup(this.defaultMQProducer.getProducerGroup());
 context.setCommunicationMode(communicationMode);
 context.setBornHost(this.defaultMQProducer.getClientIP());
 context.setBrokerAddr(brokerAddr);
 context.setMessage(msg);
 context.setMq(mq);
 String isTrans = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
 if (isTrans != null && isTrans.equals("true")) {
 context.setMsgType(MessageType.Trans_Msg_Half);
 }

 if (msg.getProperty("__STARTDELIVERTIME") != null || msg.getProperty(MessageConst.PROPERTY_DELAY_TIME_LEVEL) != null) {
 context.setMsgType(MessageType.Delay_Msg);
 }
 this.executeSendMessageHookBefore(context);
 }
 // 构建消息发送请求头
 SendMessageRequestHeader requestHeader = new SendMessageRequestHeader();
 requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup());
 requestHeader.setTopic(msg.getTopic());
 requestHeader.setDefaultTopic(this.defaultMQProducer.getCreateTopicKey());
 requestHeader.setDefaultTopicQueueNums(this.defaultMQProducer.getDefaultTopicQueueNums());
 requestHeader.setQueueId(mq.getQueueId());
 requestHeader.setSysFlag(sysFlag);
 requestHeader.setBornTimestamp(System.currentTimeMillis());
 requestHeader.setFlag(msg.getFlag());
 requestHeader.setProperties(MessageDecoder.messageProperties2String(msg.getProperties()));
 requestHeader.setReconsumeTimes(0);
 requestHeader.setUnitMode(this.isUnitMode());
 requestHeader.setBatch(msg instanceof MessageBatch);
 if (requestHeader.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
 String reconsumeTimes = MessageAccessor.getReconsumeTime(msg);
 if (reconsumeTimes != null) {
 requestHeader.setReconsumeTimes(Integer.valueOf(reconsumeTimes));
 MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_RECONSUME_TIME);
 }

 String maxReconsumeTimes = MessageAccessor.getMaxReconsumeTimes(msg);
 if (maxReconsumeTimes != null) {
 requestHeader.setMaxReconsumeTimes(Integer.valueOf(maxReconsumeTimes));
 MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_MAX_RECONSUME_TIMES);
 }
 }

 SendResult sendResult = null;
 // 消息发送逻辑
 switch (communicationMode) {
 // 异步发送
 case ASYNC:
 Message tmpMessage = msg;
 if (msgBodyCompressed) {
 //If msg body was compressed, msgbody should be reset using prevBody.
 //Clone new message using commpressed message body and recover origin massage.
 //Fix bug:https://github.com/apache/rocketmq-externals/issues/66
 tmpMessage = MessageAccessor.cloneMessage(msg);
 msg.setBody(prevBody);
 }
 long costTimeAsync = System.currentTimeMillis() - beginStartTime;
 if (timeout < costTimeAsync) {
 throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
 }
 sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
 brokerAddr,
 mq.getBrokerName(),
 tmpMessage,
 requestHeader,
 timeout - costTimeAsync,
 communicationMode,
 sendCallback,
 topicPublishInfo,
 this.mQClientFactory,
 this.defaultMQProducer.getRetryTimesWhenSendAsyncFailed(),
 context,
 this);
 break;
 case ONEWAY:
 // 同步发送
 case SYNC:
 long costTimeSync = System.currentTimeMillis() - beginStartTime;
 if (timeout < costTimeSync) {
 throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
 }
 sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
 brokerAddr,
 mq.getBrokerName(),
 msg,
 requestHeader,
 timeout - costTimeSync,
 communicationMode,
 context,
 this);
 break;
 default:
 assert false;
 break;
 }

 if (this.hasSendMessageHook()) {
 // 发送结果放入上下文后返回
 context.setSendResult(sendResult);
 this.executeSendMessageHookAfter(context);
 }

 return sendResult;
 } catch (RemotingException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } catch (MQBrokerException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } catch (InterruptedException e) {
 if (this.hasSendMessageHook()) {
 context.setException(e);
 this.executeSendMessageHookAfter(context);
 }
 throw e;
 } finally {
 msg.setBody(prevBody);
 }
 }

 throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
 }
}

使用MQClientAPIImpl真正发起Netty网络请求

package org.apache.rocketmq.client.impl;

public class MQClientAPIImpl {
 ...
 public SendResult sendMessage(
 final String addr,
 final String brokerName,
 final Message msg,
 final SendMessageRequestHeader requestHeader,
 final long timeoutMillis,
 final CommunicationMode communicationMode,
 final SendMessageContext context,
 final DefaultMQProducerImpl producer
 ) throws RemotingException, MQBrokerException, InterruptedException {
 return sendMessage(addr, brokerName, msg, requestHeader, timeoutMillis, communicationMode, null, null, null, 0, context, producer);
 }

 /**
 * 封装 RemotingCommand 对象后真正发起网络请求
 */
 public SendResult sendMessage(
 final String addr,
 final String brokerName,
 final Message msg,
 final SendMessageRequestHeader requestHeader,
 final long timeoutMillis,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final TopicPublishInfo topicPublishInfo,
 final MQClientInstance instance,
 final int retryTimesWhenSendFailed,
 final SendMessageContext context,
 final DefaultMQProducerImpl producer
 ) throws RemotingException, MQBrokerException, InterruptedException {
 long beginStartTime = System.currentTimeMillis();
 RemotingCommand request = null;
 if (sendSmartMsg || msg instanceof MessageBatch) {
 SendMessageRequestHeaderV2 requestHeaderV2 = SendMessageRequestHeaderV2.createSendMessageRequestHeaderV2(requestHeader);
 request = RemotingCommand.createRequestCommand(msg instanceof MessageBatch ? RequestCode.SEND_BATCH_MESSAGE : RequestCode.SEND_MESSAGE_V2, requestHeaderV2);
 } else {
 request = RemotingCommand.createRequestCommand(RequestCode.SEND_MESSAGE, requestHeader);
 }
 // 设置请求体
 request.setBody(msg.getBody());

 switch (communicationMode) {
 case ONEWAY:
 this.remotingClient.invokeOneway(addr, request, timeoutMillis);
 return null;
 case ASYNC:
 final AtomicInteger times = new AtomicInteger();
 long costTimeAsync = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTimeAsync) {
 throw new RemotingTooMuchRequestException("sendMessage call timeout");
 }
 this.sendMessageAsync(addr, brokerName, msg, timeoutMillis - costTimeAsync, request, sendCallback, topicPublishInfo, instance,
 retryTimesWhenSendFailed, times, context, producer);
 return null;
 case SYNC:
 long costTimeSync = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTimeSync) {
 throw new RemotingTooMuchRequestException("sendMessage call timeout");
 }
 return this.sendMessageSync(addr, brokerName, msg, timeoutMillis - costTimeSync, request);
 default:
 assert false;
 break;
 }

 return null;
 }

 private SendResult sendMessageSync(
 final String addr,
 final String brokerName,
 final Message msg,
 final long timeoutMillis,
 final RemotingCommand request
 ) throws RemotingException, MQBrokerException, InterruptedException {
 // 同步发网络请求，返回响应结果
 RemotingCommand response = this.remotingClient.invokeSync(addr, request, timeoutMillis);
 assert response != null;
 return this.processSendResponse(brokerName, msg, response);
 }
}

底层Netty通信

类图

`RemotingClient`源码

package org.apache.rocketmq.remoting;

public interface RemotingClient extends RemotingService {

 /**
 * 更新协调服务地址列表
 * @param addrs
 */
 void updateNameServerAddressList(final List<String> addrs);

 /**
 * 更新协调服务地址列表
 * @return
 */
 List<String> getNameServerAddressList();

 /**
 * 同步消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 RemotingCommand invokeSync(final String addr, final RemotingCommand request,
 final long timeoutMillis) throws InterruptedException, RemotingConnectException,
 RemotingSendRequestException, RemotingTimeoutException;

 /**
 * 异步消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeAsync(final String addr, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback) throws InterruptedException, RemotingConnectException,
 RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException;

 /**
 * 单次消息发送
 * @param addr
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingConnectException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeOneway(final String addr, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingConnectException, RemotingTooMuchRequestException,
 RemotingTimeoutException, RemotingSendRequestException;

 /**
 *用于注册一些底层通信服务，比如进行元数据信息的同步工作，commitlog的同步等
 * @param requestCode 底层业务通信规则码
 * @param processor 注册器（单线程）
 * @param executor 线程池，用于执行注册器的业务逻辑
 */
 void registerProcessor(final int requestCode, final NettyRequestProcessor processor,
 final ExecutorService executor);

 /**
 * 回调函数执行线程设置，用于生产者发送消息后的回调函数
 * @param callbackExecutor
 */
 void setCallbackExecutor(final ExecutorService callbackExecutor);

 /**
 * 获取回调池：NameServer地址
 * @return
 */
 ExecutorService getCallbackExecutor();
 
 /**
 * 用于远程服务地址判断是否可以进行通信
 * @param addr
 * @return
 */
 boolean isChannelWritable(final String addr);
}

registerProcessor方法在实现类中用于初始化时进行注册逻辑

package org.apache.rocketmq.client.impl;

public class MQClientAPIImpl {
 /**
 * Netty 服务启动时初始化工作
 * @param nettyClientConfig 配置信息
 * @param clientRemotingProcessor
 * @param rpcHook
 * @param clientConfig
 */
 public MQClientAPIImpl(final NettyClientConfig nettyClientConfig,
 final ClientRemotingProcessor clientRemotingProcessor,
 RPCHook rpcHook, final ClientConfig clientConfig) {
 this.clientConfig = clientConfig;
 topAddressing = new TopAddressing(MixAll.getWSAddr(), clientConfig.getUnitName());
 this.remotingClient = new NettyRemotingClient(nettyClientConfig, null);
 this.clientRemotingProcessor = clientRemotingProcessor;

 this.remotingClient.registerRPCHook(rpcHook);
 this.remotingClient.registerProcessor(RequestCode.CHECK_TRANSACTION_STATE, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.NOTIFY_CONSUMER_IDS_CHANGED, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.RESET_CONSUMER_CLIENT_OFFSET, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.GET_CONSUMER_STATUS_FROM_CLIENT, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.GET_CONSUMER_RUNNING_INFO, this.clientRemotingProcessor, null);

 this.remotingClient.registerProcessor(RequestCode.CONSUME_MESSAGE_DIRECTLY, this.clientRemotingProcessor, null);
 }
 ...
}

setCallbackExecutor方法用于回调函数执行线程设置，用于生产者发送消息后的回调线程池，例如DefaultMQProducerImpl中的使用方式

package org.apache.rocketmq.client.impl.producer;

public class DefaultMQProducerImpl implements MQProducerInner {
 ...
 public void setCallbackExecutor(final ExecutorService callbackExecutor) {
	 this.mQClientFactory.getMQClientAPIImpl().getRemotingClient().setCallbackExecutor(callbackExecutor);
 }
 
 /**
 * It will be removed at 4.4.0 cause for exception handling and the wrong Semantics of timeout.
 * A new one will be provided in next version
 * @param msg
 * @param sendCallback
 * @param timeout the <code>sendCallback</code> will be invoked at most time
 * @throws RejectedExecutionException
 */
 @Deprecated
 public void send(final Message msg, final SendCallback sendCallback, final long timeout)
 throws MQClientException, RemotingException, InterruptedException {
 final long beginStartTime = System.currentTimeMillis();
 // 获取回调线程池
 ExecutorService executor = this.getCallbackExecutor();
 try {
 executor.submit(new Runnable() {
 @Override
 public void run() {
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeout > costTime) {
 try {
 sendDefaultImpl(msg, CommunicationMode.ASYNC, sendCallback, timeout - costTime);
 } catch (Exception e) {
 sendCallback.onException(e);
 }
 } else {
 sendCallback.onException(
 new RemotingTooMuchRequestException("DEFAULT ASYNC send call timeout"));
 }
 }

 });
 } catch (RejectedExecutionException e) {
 throw new MQClientException("executor rejected ", e);
 }

 }
	...
}

`RemotingServer`源码

package org.apache.rocketmq.remoting;

public interface RemotingServer extends RemotingService {

 /**
 * 用于注册一些底层通信服务，比如进行元数据信息的同步工作，commitlog的同步等
 * @param requestCode 底层业务通信规则码
 * @param processor 注册器（单线程）
 * @param executor 线程池，用于执行注册器的业务逻辑
 */
 void registerProcessor(final int requestCode, final NettyRequestProcessor processor,
 final ExecutorService executor);

 /**
 * 默认注册方法：比如注册：{@link org.apache.rocketmq.broker.processor.AdminBrokerProcessor}管理Broker工作线程或者namesrv有关的流程器，
 * 比如同步地址、路由topic等信息
 * @param processor
 * @param executor
 */
 void registerDefaultProcessor(final NettyRequestProcessor processor, final ExecutorService executor);

 /**
 * 获取端口
 * @return
 */
 int localListenPort();

 /**
 * 根据业务规则获取Pair<执行器，线程池>
 * @param requestCode 业务规则码
 * @return
 */
 Pair<NettyRequestProcessor, ExecutorService> getProcessorPair(final int requestCode);

 /**
 * 同步消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 RemotingCommand invokeSync(final Channel channel, final RemotingCommand request,
 final long timeoutMillis) throws InterruptedException, RemotingSendRequestException,
 RemotingTimeoutException;

 /**
 * 异步消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeAsync(final Channel channel, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback) throws InterruptedException,
 RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException;

 /**
 * 单向消息发送
 * @param channel
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 void invokeOneway(final Channel channel, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException,
 RemotingSendRequestException;

}

`NettyRemotingServer`源码

其唯一实现类NettyRemotingServer，主要实现Netty编程模型

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingServer extends NettyRemotingAbstract implements RemotingServer {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);
 /**
 * NettyServer辅助类
 */
 private final ServerBootstrap serverBootstrap;
 /**
 * selector组（用于注册并监听event）
 */
 private final EventLoopGroup eventLoopGroupSelector;
 /**
 * 主线程组（用于接收TCP连接）
 */
 private final EventLoopGroup eventLoopGroupBoss;
 /**
 * NettyServer配置
 */
 private final NettyServerConfig nettyServerConfig;

 /**
 * 主要使用在registerProcessor方法中
 */
 private final ExecutorService publicExecutor;

 /**
 * channel事件监听
 */
 private final ChannelEventListener channelEventListener;

 public NettyRemotingServer(final NettyServerConfig nettyServerConfig,
 final ChannelEventListener channelEventListener) {
 super(nettyServerConfig.getServerOnewaySemaphoreValue(), nettyServerConfig.getServerAsyncSemaphoreValue());
 this.serverBootstrap = new ServerBootstrap();
 this.nettyServerConfig = nettyServerConfig;
 this.channelEventListener = channelEventListener;

 int publicThreadNums = nettyServerConfig.getServerCallbackExecutorThreads();
 if (publicThreadNums <= 0) {
 publicThreadNums = 4;
 }

 this.publicExecutor = Executors.newFixedThreadPool(publicThreadNums, new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyServerPublicExecutor_" + this.threadIndex.incrementAndGet());
 }
 });

 this.eventLoopGroupBoss = new NioEventLoopGroup(1, new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyBoss_%d", this.threadIndex.incrementAndGet()));
 }
 });

 // 如果是Linux平台，并且开启了native epoll，就用EpollEventLoopGroup，这个也就是用JNI，调用c写的epoll；否则就用java NIO的NioEventLoopGroup
 if (useEpoll()) {
 this.eventLoopGroupSelector = new EpollEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);
 private int threadTotal = nettyServerConfig.getServerSelectorThreads();

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyServerEPOLLSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
 }
 });
 } else {
 this.eventLoopGroupSelector = new NioEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
 private AtomicInteger threadIndex = new AtomicInteger(0);
 private int threadTotal = nettyServerConfig.getServerSelectorThreads();

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, String.format("NettyServerNIOSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
 }
 });
 }

 loadSslContext();
 }
 
 @Override
 public void start() {
 // 根据配置创建工作线程组
 this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
 nettyServerConfig.getServerWorkerThreads(),
 new ThreadFactory() {

 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyServerCodecThread_" + this.threadIndex.incrementAndGet());
 }
 });
 // Netty编程模型
 ServerBootstrap childHandler =
 this.serverBootstrap.group(this.eventLoopGroupBoss, this.eventLoopGroupSelector)
 .channel(useEpoll() ? EpollServerSocketChannel.class : NioServerSocketChannel.class)
 // sync + accept = backlog
 .option(ChannelOption.SO_BACKLOG, 1024)
 // 允许重复使用本地地址和端口
 .option(ChannelOption.SO_REUSEADDR, true)
 .option(ChannelOption.SO_KEEPALIVE, false)
 .childOption(ChannelOption.TCP_NODELAY, true)
 .childOption(ChannelOption.SO_SNDBUF, nettyServerConfig.getServerSocketSndBufSize())
 .childOption(ChannelOption.SO_RCVBUF, nettyServerConfig.getServerSocketRcvBufSize())
 .localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort()))
 .childHandler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) throws Exception {
 ch.pipeline()
 .addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME,
 new HandshakeHandler(TlsSystemConfig.tlsMode))
 .addLast(defaultEventExecutorGroup,
 new NettyEncoder(),
 new NettyDecoder(),
 new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()),
 new NettyConnectManageHandler(),
 new NettyServerHandler()
 );
 }
 });

 if (nettyServerConfig.isServerPooledByteBufAllocatorEnable()) {
 childHandler.childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
 }

 try {
 ChannelFuture sync = this.serverBootstrap.bind().sync();
 InetSocketAddress addr = (InetSocketAddress) sync.channel().localAddress();
 this.port = addr.getPort();
 } catch (InterruptedException e1) {
 throw new RuntimeException("this.serverBootstrap.bind().sync() InterruptedException", e1);
 }

 if (this.channelEventListener != null) {
 this.nettyEventExecutor.start();
 }

 this.timer.scheduleAtFixedRate(new TimerTask() {

 @Override
 public void run() {
 try {
 NettyRemotingServer.this.scanResponseTable();
 } catch (Throwable e) {
 log.error("scanResponseTable exception", e);
 }
 }
 }, 1000 * 3, 1000);
 }
 ...
}

`NettyRemotingAbstract`源码

NettyRemotingAbstract具体实现三种发送方法

package org.apache.rocketmq.remoting.netty;

public abstract class NettyRemotingAbstract {
 /**
 * 异步调用具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @param invokeCallback
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis,
 final InvokeCallback invokeCallback)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
 long beginStartTime = System.currentTimeMillis();
 final int opaque = request.getOpaque();
 boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
 if (acquired) {
 final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync);
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTime) {
 throw new RemotingTooMuchRequestException("invokeAsyncImpl call timeout");
 }

 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis - costTime, invokeCallback, once);
 this.responseTable.put(opaque, responseFuture);
 try {
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 }
 requestFail(opaque);
 log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel));
 }
 });
 } catch (Exception e) {
 responseFuture.release();
 log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e);
 throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
 }
 } else {
 if (timeoutMillis <= 0) {
 throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast");
 } else {
 String info =
 String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
 timeoutMillis,
 this.semaphoreAsync.getQueueLength(),
 this.semaphoreAsync.availablePermits()
 );
 log.warn(info);
 throw new RemotingTimeoutException(info);
 }
 }
 }
 /**
 * 单向发送具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @throws InterruptedException
 * @throws RemotingTooMuchRequestException
 * @throws RemotingTimeoutException
 * @throws RemotingSendRequestException
 */
 public void invokeOnewayImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis)
 throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
 request.markOnewayRPC();
 boolean acquired = this.semaphoreOneway.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
 if (acquired) {
 final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreOneway);
 try {
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 once.release();
 if (!f.isSuccess()) {
 log.warn("send a request command to channel <" + channel.remoteAddress() + "> failed.");
 }
 }
 });
 } catch (Exception e) {
 once.release();
 log.warn("write send a request command to channel <" + channel.remoteAddress() + "> failed.");
 throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
 }
 } else {
 if (timeoutMillis <= 0) {
 throw new RemotingTooMuchRequestException("invokeOnewayImpl invoke too fast");
 } else {
 String info = String.format(
 "invokeOnewayImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
 timeoutMillis,
 this.semaphoreOneway.getQueueLength(),
 this.semaphoreOneway.availablePermits()
 );
 log.warn(info);
 throw new RemotingTimeoutException(info);
 }
 }
 }
 /**
 * 同步调用具体实现
 * @param channel
 * @param request
 * @param timeoutMillis
 * @return
 * @throws InterruptedException
 * @throws RemotingSendRequestException
 * @throws RemotingTimeoutException
 */
 public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
 final long timeoutMillis)
 throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
 final int opaque = request.getOpaque();

 try {
 final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
 this.responseTable.put(opaque, responseFuture);
 final SocketAddress addr = channel.remoteAddress();
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
 @Override
 public void operationComplete(ChannelFuture f) throws Exception {
 if (f.isSuccess()) {
 responseFuture.setSendRequestOK(true);
 return;
 } else {
 responseFuture.setSendRequestOK(false);
 }

 responseTable.remove(opaque);
 responseFuture.setCause(f.cause());
 responseFuture.putResponse(null);
 log.warn("send a request command to channel <" + addr + "> failed.");
 }
 });

 RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
 if (null == responseCommand) {
 if (responseFuture.isSendRequestOK()) {
 throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
 responseFuture.getCause());
 } else {
 throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
 }
 }

 return responseCommand;
 } finally {
 this.responseTable.remove(opaque);
 }
 }
 ...
}

Netty编码解码源码解析——`RemotingCommand`源码

netty消息数据包结构

RemotingCommand中编码、解码方法体现了其数据包格式和用法

package org.apache.rocketmq.remoting.protocol;

public class RemotingCommand {
 
 public ByteBuffer encode() {
 // 1> header length size
 int length = 4;

 // 2> header data length
 byte[] headerData = this.headerEncode();
 length += headerData.length;

 // 3> body data length
 if (this.body != null) {
 length += body.length;
 }

 ByteBuffer result = ByteBuffer.allocate(4 + length);

 // length
 result.putInt(length);

 // header length
 result.put(markProtocolType(headerData.length, serializeTypeCurrentRPC));

 // header data
 result.put(headerData);

 // body data;
 if (this.body != null) {
 result.put(this.body);
 }

 result.flip();

 return result;
 }
 public static RemotingCommand decode(final ByteBuffer byteBuffer) {
 int length = byteBuffer.limit();
 int oriHeaderLen = byteBuffer.getInt();
 int headerLength = getHeaderLength(oriHeaderLen);

 byte[] headerData = new byte[headerLength];
 // 此处使用的是java nio buffer
 byteBuffer.get(headerData);
 // header封装，长度+类型 对应数据包结构第二位
 RemotingCommand cmd = headerDecode(headerData, getProtocolType(oriHeaderLen));

 int bodyLength = length - 4 - headerLength;
 byte[] bodyData = null;
 if (bodyLength > 0) {
 bodyData = new byte[bodyLength];
 byteBuffer.get(bodyData);
 }
 cmd.body = bodyData;

 return cmd;
 }
 ...
}

其支持的序列化类型有两个

SerializeType序列化类型枚举

package org.apache.rocketmq.remoting.protocol;

public enum SerializeType {
 JSON((byte) 0),
 ROCKETMQ((byte) 1);
 ...
}

在NettyRemotingClient中的Netty编程模型也有编码、解码配置，其使用的是netty buffer

package org.apache.rocketmq.remoting.netty;

public class NettyRemotingClient extends NettyRemotingAbstract implements RemotingClient {
 @Override
 public void start() {
 this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
 nettyClientConfig.getClientWorkerThreads(),
 new ThreadFactory() {

 private AtomicInteger threadIndex = new AtomicInteger(0);

 @Override
 public Thread newThread(Runnable r) {
 return new Thread(r, "NettyClientWorkerThread_" + this.threadIndex.incrementAndGet());
 }
 });

 Bootstrap handler = this.bootstrap.group(this.eventLoopGroupWorker).channel(NioSocketChannel.class)
 .option(ChannelOption.TCP_NODELAY, true)
 .option(ChannelOption.SO_KEEPALIVE, false)
 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, nettyClientConfig.getConnectTimeoutMillis())
 .option(ChannelOption.SO_SNDBUF, nettyClientConfig.getClientSocketSndBufSize())
 .option(ChannelOption.SO_RCVBUF, nettyClientConfig.getClientSocketRcvBufSize())
 .handler(new ChannelInitializer<SocketChannel>() {
 @Override
 public void initChannel(SocketChannel ch) throws Exception {
 ChannelPipeline pipeline = ch.pipeline();
 if (nettyClientConfig.isUseTLS()) {
 if (null != sslContext) {
 pipeline.addFirst(defaultEventExecutorGroup, "sslHandler", sslContext.newHandler(ch.alloc()));
 log.info("Prepend SSL handler");
 } else {
 log.warn("Connections are insecure as SSLContext is null!");
 }
 }
 pipeline.addLast(
 // 使用工作线程
 defaultEventExecutorGroup,
 // 编码
 new NettyEncoder(),
 // 解码
 new NettyDecoder(),
 new IdleStateHandler(0, 0, nettyClientConfig.getClientChannelMaxIdleTimeSeconds()),
 // 连接管理器
 new NettyConnectManageHandler(),
 new NettyClientHandler());
 }
 });

 this.timer.scheduleAtFixedRate(new TimerTask() {
 @Override
 public void run() {
 try {
 NettyRemotingClient.this.scanResponseTable();
 } catch (Throwable e) {
 log.error("scanResponseTable exception", e);
 }
 }
 }, 1000 * 3, 1000);

 if (this.channelEventListener != null) {
 this.nettyEventExecutor.start();
 }
 }
}

NettyEncoder源码

package org.apache.rocketmq.remoting.netty;

import io.netty.buffer.ByteBuf;
import java.nio.ByteBuffer;

public class NettyEncoder extends MessageToByteEncoder<RemotingCommand> {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);

 @Override
 public void encode(ChannelHandlerContext ctx, RemotingCommand remotingCommand, ByteBuf out)
 throws Exception {
 try {
 ByteBuffer header = remotingCommand.encodeHeader();
 out.writeBytes(header);
 byte[] body = remotingCommand.getBody();
 if (body != null) {
 out.writeBytes(body);
 }
 } catch (Exception e) {
 log.error("encode exception, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e);
 if (remotingCommand != null) {
 log.error(remotingCommand.toString());
 }
 RemotingUtil.closeChannel(ctx.channel());
 }
 }
}

NettyDecoder源码

其继承LengthFieldBasedFrameDecoder，支持灵活配置编解码参数

package org.apache.rocketmq.remoting.netty;

import io.netty.buffer.ByteBuf;
import io.netty.handler.codec.LengthFieldBasedFrameDecoder;
import java.nio.ByteBuffer;

public class NettyDecoder extends LengthFieldBasedFrameDecoder {
 private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);

 private static final int FRAME_MAX_LENGTH =
 Integer.parseInt(System.getProperty("com.rocketmq.remoting.frameMaxLength", "16777216"));

 public NettyDecoder() {
 super(FRAME_MAX_LENGTH, 0, 4, 0, 4);
 }

 @Override
 public Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception {
 ByteBuf frame = null;
 try {
 frame = (ByteBuf) super.decode(ctx, in);
 if (null == frame) {
 return null;
 }

 ByteBuffer byteBuffer = frame.nioBuffer();

 return RemotingCommand.decode(byteBuffer);
 } catch (Exception e) {
 log.error("decode exception, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e);
 RemotingUtil.closeChannel(ctx.channel());
 } finally {
 if (null != frame) {
 frame.release();
 }
 }

 return null;
 }
}

消息返回状态详解

SEND_OK：消息发送成功。
FLUSH_DISK_TIMEOUT：消息发送成功，但消息刷盘失败，此时服务器宕机则消息丢失
FLUSH_SLAVE_TIMEOUT：主从同步消息时超时，从节点已经收到消息，若此时从节点宕机，则从节点消息丢失
SLAVE_NOT_AVAILABLE：从节点不可用，此时主节点宕机，则消息丢失

对于可靠消息投递，后三种情况均应该考虑补偿。

延迟消息投递

场景：消息发送到Broker后，要特定的时间才会被Consumer消费

特点：目前只支持固定精度的定时消息

固定精度消息原因：支持任意精度消息时，对消息持久化、排序产生较大性能开销

使用方法：对message设置setDelayTimeLevel级别即可

核心类：MessageStoreConfig和ScheduleMessageService

MessageStoreConfig源码

package org.apache.rocketmq.store.config;

public class MessageStoreConfig {
 // 延时投递级别，固定精度
 private String messageDelayLevel = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h";
 ...
}

自定义消息发送规则

场景：把消息发送到指定的队列（Message Queue）

处理类：MessageQueueSelector

用法：调用send(Message msg, MessageQueueSelector selector, Object arg)方法即可

实现代码片段，指定发送到id为1的队列中

SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
 // 外部方法的arg会传递到Object中
 @Override
 public MessageQueue select(List<MessageQueue> list, Message message, Object o) {
 Integer queueNumber = (Integer) o;
 return list.get(queueNumber);
 }
},1);

RocketMQ学习笔记2——入门

Wed, 23 Dec 2020 19:06:00 +0800

生产者模型使用

步骤

创建生产者对象DefaultMQProducer
设置NamesrvAddr
启动生产者服务
创建消息并发送

简单消息发送

创建测试主题

常量定义

public class Const {

	public static final String NAMESRV_ADDR_SINGLE = "10.139.12.52:9876";
}

public class Producer {

	public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
		
		DefaultMQProducer producer = new DefaultMQProducer("test_quick_producer_name");
		
		producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
		producer.start();
		
		for(int i = 0 ; i <5; i ++) {
			//	1.	创建消息
			Message message = new Message("test_quick_topic",	//	主题
					"TagA", //	标签
					"key" + i,	// 	用户自定义的key ,唯一的标识
					("Hello RocketMQ" + i).getBytes());	//	消息内容实体（byte[]）
			SendResult sr = producer.send(message, new MessageQueueSelector() {
				@Override
				public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
					Integer queueNumber = (Integer)arg;
					return mqs.get(queueNumber);
				}
			}, 2);
			System.err.println(sr);
			System.err.println("消息发出: " + sr);
		}
		producer.shutdown();
	}
}

运行结果

SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3]
SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4]
消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4]
10:45:15.129 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10911] result: true
10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10909] result: true
10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:9876] result: true

消费者模型使用

步骤

创建消费者对象DefaultMQPushConsumer
设置NamesrvAddr及消费位置ConsumeFromWhere
进行主题订阅Subscribe
注册监听并消费registerMessageListener

简单消息消费

public class Consumer {

	public static void main(String[] args) throws MQClientException {
		// 采用Push方式
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test_quick_consumer_name");
		
		consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
		// 匹配所有Tag
		consumer.subscribe("test_quick_topic", "*");
		
		consumer.registerMessageListener(new MessageListenerConcurrently() {
			
			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
				MessageExt me = msgs.get(0);
				try {
					String topic = me.getTopic();
					String tags = me.getTags();
					String keys = me.getKeys();
					
					String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET);
					System.err.println("topic: " + topic + ",tags: " + tags + ", keys: " + keys + ",body: " + msgBody);
				} catch (Exception e) {
					e.printStackTrace();
 // 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
					return ConsumeConcurrentlyStatus.RECONSUME_LATER;
				}
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		consumer.start();
		System.err.println("consumer start...");
	}
}

模拟异常重试

public class Consumer {

	public static void main(String[] args) throws MQClientException {
		// 采用Push方式
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test_quick_consumer_name");
		
		consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE);
		
		consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
		// 匹配所有Tag
		consumer.subscribe("test_quick_topic", "*");
		
		consumer.registerMessageListener(new MessageListenerConcurrently() {
			
			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
				MessageExt me = msgs.get(0);
				try {
					String topic = me.getTopic();
					String tags = me.getTags();
					String keys = me.getKeys();
					// 模拟异常
					if(keys.equals("key1")) {
						System.err.println("消息消费失败..");
						int a = 1/0;
					}
					
					String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET);
					System.err.println("topic: " + topic + ",tags: " + tags + ", keys: " + keys + ",body: " + msgBody);
				} catch (Exception e) {
					e.printStackTrace();
					// 重发次数
					int recousumeTimes = me.getReconsumeTimes();
					System.err.println("recousumeTimes: " + recousumeTimes);
					// 超过3次不再重发
					if(recousumeTimes == 3) {
						//		记录日志....
						// 	做补偿处理
						return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
					}
					// 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
					return ConsumeConcurrentlyStatus.RECONSUME_LATER;
				}
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		consumer.start();
		System.err.println("consumer start...");
	}
}

日志输出

consumer start...
topic: test_quick_topic,tags: TagA, keys: key0,body: Hello RocketMQ0
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 0
topic: test_quick_topic,tags: TagA, keys: key2,body: Hello RocketMQ2
topic: test_quick_topic,tags: TagA, keys: key3,body: Hello RocketMQ3
topic: test_quick_topic,tags: TagA, keys: key4,body: Hello RocketMQ4
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 1
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 2
消息消费失败..
java.lang.ArithmeticException: / by zero
	at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40)
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
recousumeTimes: 3

四种集群环境

集群环境拓扑图

分类

单点模式（不推荐）
主从模式
双主模式（不推荐）
双主双从、多主多从模式

主从集群环境构建

优势：

主从模式可以保障消息的即时性与可靠性
当投递一条消息后关闭主节点，从节点可以继续提供消费者进行消费，但不能接受消息
主节点上线后会进行消费进度offset同步

基础环境

选择2台服务器进行部署验证

IP	IP
GY11-SyOCM33-101	10.139.12.52	主
GY11-SyOCM33-201	10.139.12.53	从

53节点按照单点部署时操作好对应目录和日志配置

修改主节点配置

52节点修改主节点配置文件broker-a.properties如下

修改了namesrvAddr

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876;10.139.12.53:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

修改从节点配置

修改从节点配置文件broker-a-s.properties如下

拷贝主节点配置后修改id和brokerRole

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=1
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876;10.139.12.53:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=SLAVE

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

主从配置同步到53节点即可

启动NameSrv

启动Namesrv（主从节点均执行）

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin
$ nohup sh mqnamesrv &

启动Broker

主节点启动

$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &

从节点启动

$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a-s.properties >/dev/null 2>&1 &

启动控制台

修改控制台配置地址参数为

rocketmq.config.namesrvAddr=10.139.12.52:9876;10.139.12.53:9876

数据高可用机制故障演练

投递一条消息

关闭主节点，剩余一个slave

启动消费者，消费正常

主节点上线，同步消息位点（offset）使消费端不会重复消费

RocketMQ学习笔记1——初探

Wed, 23 Dec 2020 19:04:00 +0800

RocketMQ整体认知

优势

支持集群模型、负载均衡、水平扩展能力
亿级别的消息堆积能力
采用零拷贝的原理、顺序写盘、随机读
丰富的API使用
代码优秀，底层通信框架采用Netty NIO框架
NameServer代替Zookeeper
强调集群无单点，可扩展，任意一点高可用，水平可扩展
消息失败重试机制，消息可查询
开源社区活跃，成熟度高（经过双十一考验）

核心概念模型

基本概念

消息生产者（Producer）：负责生产消息，一般由业务系统负责生产消息。
消息消费者（Consumer）：负责消费消息，一般是后台系统负责异步消费。
推动式消费（Push Consumer）： Consumer消费的一种类型，该模式下Broker收到数据后会主动推送给消费端。
拉取式消费（Pull Consumer）： Consumer消费的一种类型，应用通常主动调用Consumer的拉消息方法从Broker服务器拉消息。
生产者组（Producer Group）：同一类Producer的集合，这类Producer发送同一类消息且发送逻辑一致。
消费者组（Consumer Group）：同一类Consumer的集合，这类Consumer通常消费同一类消息且消费逻辑一致。
Broker：MQ消息服务（中转角色，用于消息存储于生产消息转发）

源码结构

源码分析以4.4.0为例

源码地址： https://github.com/apache/rocketmq

结构：

结构解析：

rocketmq-common：公用数据结构
rocketmq-distribution：编译模块，编译输出
rocketmq-filter：进行Broker过滤的不感兴趣的消息传输，减小带宽压力
rocketmq-logappender、rocketmq-logging：日志服务
rocketmq-namesrv：Namesrv服务，用于服务协调
rocketmq-openmessaging：对外提供服务
rocketmq-remoting：远程调用接口，封装Netty底层通信
rocketmq-srvutil：提供一些公用的工具方法，比如命令行解析参数
rocketmq-store：消息存储
rocketmq-test、rocketmq-example
rocketmq-tools：管理工具，比如有名的mqadmin工具

编译部署&环境搭建

选择1台服务器进行部署验证

IP	IP
GY11-SyOCM33-101	10.139.12.52

编译部署

在源码目录执行一下命令编译工程

$ mvn -Prelease-all -DskipTests clean package

编译成功后在rocketmq\distribution\target目录下生成apache-rocketmq.tar.gz包，使用该包解压与从官网下载一致，其他步骤参考二进制部署。

二进制包部署（单点）

此处从官网下载4.4.0版本并上传至服务器解压

下载地址

解压到~/rocketmq-test/rocketmq-all-4.4.0-bin-release/如下

$ ll
total 36
drwxr-xr-x 2 ocmuser gpocm 83 Jan 17 2019 benchmark
drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 bin
drwxr-xr-x 5 ocmuser gpocm 196 Jan 17 2019 conf
drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 lib
-rw-r--r-- 1 ocmuser gpocm 17336 Jan 17 2019 LICENSE
-rw-r--r-- 1 ocmuser gpocm 1337 Jan 17 2019 NOTICE
-rw-r--r-- 1 ocmuser gpocm 2481 Jan 17 2019 README.md

创建存储目录

$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index

RocketMQ 配置文件

进入提供好的样例配置文件目录，其中包含双主双从异步复制、双主双从同步复制和双主集群配置文件，此处选择双主双从异步复制作为配置模板

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf]$ll
total 36
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async
drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync
drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave
-rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf
-rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml
-rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml
-rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml
-rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml
-rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml

选择broker-a配置文件

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async]$ll
total 16
-rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-a.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties
-rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-b.properties
-rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties

删除原有配置，加入如下配置（单点）

#所属集群名字
brokerClusterName=rocketmq-cluster
#broker 名字，注意此处不同的配置文件填写的不一样
brokerName=broker-a
#0 表示 Master，>0 表示 Slave
brokerId=0
#nameServer 地址，分号分割
namesrvAddr=10.139.12.52:9876
#在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数
defaultTopicQueueNums=4
#是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭
autoCreateTopicEnable=true
#是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true
#Broker 对外服务的监听端口
listenPort=10911
#删除文件时间点，默认凌晨 4 点
deleteWhen=04
#文件保留时间，默认 48 小时
fileReservedTime=120
#commitLog 每个文件的大小默认 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store
#commitLog 存储路径
storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog
#消费队列存储路径存储路径
storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue
#消息索引存储路径
storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index
#checkpoint 文件存储路径
storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint
#abort 文件存储路径
abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort
#限制的消息大小
maxMessageSize=65536

#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Broker 的角色
#- ASYNC_MASTER 异步复制 Master
#- SYNC_MASTER 同步双写 Master
#- SLAVE
brokerRole=ASYNC_MASTER

#刷盘方式
#- ASYNC_FLUSH 异步刷盘
#- SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

#checkTransactionMessageEnable=false
#发消息线程池数量
#sendMessageThreadPoolNums=128
#拉消息线程池数量
#pullMessageThreadPoolNums=128

修改日志配置文件

$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs 
$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf && sed -i 's#${user.home}#/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release#g' *.xml

修改启动脚本参数

将默认8g的内存配置改成1g

$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runbroker.sh

runbroker.sh

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"
...

同理

$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runserver.sh

runserver.sh

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m"
...

启动 NameServer

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ nohup sh mqnamesrv &

检查进程

[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin]$jps
65322 NamesrvStartup
65534 Jps

启动 BrokerServer A

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &

$ netstat -ntlp 
$ jps 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/broker.log 
$ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/namesrv.log

停止与数据清理

$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin 
$ sh mqshutdown broker 
$ sh mqshutdown namesrv 
# --等待停止 
$ rm -rf /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue 
$ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index 
# --按照上面步骤重启 NameServer 与 BrokerServer

SpringBoot 控制台讲解

克隆工程：https://github.com/apache/rocketmq-externals.git

进入控制台配置文件rocketmq-externals\rocketmq-console\src\main\resources\application.properties修改配置

server.contextPath=
server.port=9090

## SSL setting
#server.ssl.key-store=classpath:rmqcngkeystore.jks
#server.ssl.key-store-password=rocketmq
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=rmqcngkey

#spring.application.index=true
spring.application.name=rocketmq-console
spring.http.encoding.charset=UTF-8
spring.http.encoding.enabled=true
spring.http.encoding.force=true
logging.config=classpath:logback.xml
#if this value is empty,use env value rocketmq.config.namesrvAddr NAMESRV_ADDR | now, you can set it in ops page.default localhost:9876
rocketmq.config.namesrvAddr=10.139.12.52:9876
#if you use rocketmq version < 3.5.8, rocketmq.config.isVIPChannel should be false.default true
rocketmq.config.isVIPChannel=
#rocketmq-console's data path:dashboard/monitor
rocketmq.config.dataPath=./data
#set it false if you don't want use dashboard.default true
rocketmq.config.enableDashBoardCollect=true
#set the message track trace topic if you don't want use the default one

SpringBoot源码分析2——走向自动装配

Thu, 10 Dec 2020 17:28:00 +0800

Spring Framework 手动装配

Spring模式注解装配

定义:一种用于声明在应用中扮演“组件”角色的注解
举例:@Component、@Service、@Configuration等
装配:< context: component-scan>或@Componentscan

官方仓库说明

Stereotype Annotations
A *stereotype annotation* is an annotation that is used to declare the role that a component plays within the application. For example, the @Repository annotation in the Spring Framework is a marker for any class that fulfills the role or stereotype of a repository (also known as Data Access Object or DAO).

@Component is a generic stereotype for any Spring-managed component. Any component annotated with @Component is a candidate for component scanning. Similarly, any component annotated with an annotation that is itself meta-annotated with @Component is also a candidate for component scanning. For example, @Service is meta-annotated with @Component.

Core Spring provides several stereotype annotations out of the box, including but not limited to: @Component, @Service, @Repository, @Controller, @RestController, and @Configuration. @Repository, @Service, etc. are specializations of @Component.

模式注解是一种用于声明在应用中扮演“组件”角色的注解。如 Spring Framework 中的 @Repository 标注在任何类上，用于扮演仓储角色的模式注解。

@Component 作为一种由 Spring 容器托管的通用模式组件，任何被 @Component 标准的组件均为组件扫描的候选对象。类似地，凡是被 @Component 元标注（meta-annotated）的注解，如 @Service ，当任何组件标注它时，也被视作组件扫描的候选对象（派生性）

模式注解举例

Spring Framework 注解	场景说明	起始版本
`@Repository`	数据仓储模式注解	2.0
`@Component`	通用组件模式注解	2.5
`@Service`	服务模式注解	2.5
`@Controller`	Web控制器模式注解	2.5
`@Configuration`	配置类模式注解	3.0

装配方式

<context:component-scan> 方式

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:context="http://www.springframework.org/schema/context"
 xsi:schemaLocation="http://www.springframework.org/schema/beans 
http://www.springframework.org/schema/beans/spring-beans.xsd
 http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-
context.xsd">
 
 <!-- 激活注解驱动特性 -->
 <context:annotation-config />
 
 <!-- 找寻被 @Component 或者其派生 Annotation 标记的类（Class），将它们注册为 Spring Bean -->
 <context:component-scan base-package="top.fjy8018.spring.boot" />
 
</beans>

@ComponentScan 方式

@ComponentScan(basePackages = "top.fjy8018.spring.boot")
public class SpringConfiguration {
 ...
}

Spring Framework手动装配

自定义模式注解

`@Component` “派生性”

package top.fjy8018.springboot.annotation;

import org.springframework.stereotype.Component;
import org.springframework.stereotype.Repository;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 10:36
 */
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Repository
public @interface FirstLevelRepository {

 /**
 * 保证签名一致性
 */
 String value() default "";
}

三个级别并无明显区别，体现注解派生性，即使用@FirstLevelRepository和使用@Component效果一致

派生关系

@Component
- @Repository
  - @FirstLevelRepository

package org.springframework.stereotype;

/**
 * @since 2.5
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Indexed
public @interface Component {

 /**
 * Component的签名
 */
 String value() default "";

}

Repository中元标注了@Component

package org.springframework.stereotype;

/**
 * @since 2.0
 * @see Component
 */
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Component
public @interface Repository {

 /**
 * 签名与Component一致
 */
 @AliasFor(annotation = Component.class)
 String value() default "";
}

`@Component` “层次性”

package top.fjy8018.springboot.annotation;

import org.springframework.stereotype.Component;
import org.springframework.stereotype.Repository;

import java.lang.annotation.*;

/**
 * Component层次性
 * 顶级 {@link Component}
 * 一级 {@link Repository @Repository}
 * 二级 {@link FirstLevelRepository}
 * 三级 {@link SecondLevelRepository}
 *
 * @author F嘉阳
 * @date 2018-09-09 10:54
 */
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@FirstLevelRepository
public @interface SecondLevelRepository {

 /**
 * 层次性、派生性都必须保证签名一致
 */
 String value() default "";
}

@Component
- @Repository
  - FirstLevelRepository
    - SecondLevelRepository

注解目标类，value 为bean的名称，此处不管用@FirstLevelRepository还是用SecondLevelRepository，效果都和使用@Component一致

package top.fjy8018.springboot.repository;

import top.fjy8018.springboot.annotation.FirstLevelRepository;

/**
 * @author F嘉阳
 * @date 2018-09-09 10:40
 */
//@Component("myRepository")
@FirstLevelRepository(value = "myRepository")
//@SecondLevelRepository(value = "myRepository")
public class RepositoryDemo {
}

仓储引导类，使用@ComponentScan指定扫描Bean路径

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.ComponentScan;
import top.fjy8018.springboot.repository.RepositoryDemo;

/**
 * @author F嘉阳
 * @date 2018-09-09 10:43
 */
@ComponentScan(basePackages = "top.fjy8018.springboot.repository")
public class RepositoryBootstrapDemo {

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(RepositoryBootstrapDemo.class)
 .web(WebApplicationType.NONE)
 .run(args);

 // 通过bean名称获取bean
 RepositoryDemo myRepository = applicationContext.getBean("myRepository", RepositoryDemo.class);

 // myRepository是否存在，若不存在则报错
 System.out.println("myRepository是否存在：" + myRepository);

 // 关闭上下文
 applicationContext.close();
 }
}

执行结果

SpringBoot源码中也有体现该层次关系

@SpringBootApplication
- @SpringBootConfiguration
  - @Configuration
    - @Component
- @EnableAutoConfiguration
  - @AutoConfigurationPackage

Spring @Enable 模块装配

Spring Framework 3.1 开始支持“@Enable 模块驱动”。所谓“模块”是指具备相同领域的功能组件集合，组合所形成一个独立的单元。比如 Web MVC 模块、AspectJ代理模块、Caching（缓存）模块、JMX（Java 管理扩展）模块、Async（异步处理）模块等。

举例:@EnableWebmvc、@EnableAutoConfiguration等
实现:注解方式、编程方式

@Enable 注解模块举例

框架实现	`@Enable` 注解模块	激活模块
Spring Framework	`@EnableWebMvc`	Web
	`@EnableTransactionManagement`	事务管理模块
	`@EnableCaching`	Caching 模块
	`@EnableMBeanExport`	JMX 模块
	`@EnableAsync`	异步处理模块
	`EnableWebFlux`	Web Flux 模块
	`@EnableAspectJAutoProxy`	AspectJ 代理模块

Spring Boot	`@EnableAutoConfiguration`	自动装配模块
	`@EnableManagementContext`	Actuator 管理模块
	`@EnableConfigurationProperties`	配置属性绑定模块
	`@EnableOAuth2Sso`	OAuth2 单点登录模块

Spring Cloud	`@EnableEurekaServer`	Eureka服务器模块
	`@EnableConfigServer`	配置服务器模块
	`@EnableFeignClients`	Feign客户端模块
	`@EnableZuulProxy`	服务网关
	`@EnableCircuitBreaker`	服务熔断模块

实现方式

注解驱动方式

Spring源码，使用@Import注入配置Bean

package org.springframework.web.servlet.config.annotation;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Import(DelegatingWebMvcConfiguration.class)
public @interface EnableWebMvc {}

接口编程方式

Spring源码——激活缓存注解

package org.springframework.cache.annotation;

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Import(CachingConfigurationSelector.class)
public @interface EnableCaching {...}

CachingConfigurationSelector类图

ImportSelector接口源码，该接口在3.1引入，目的在于增加灵活度

selectImports方法返回数组，即可以死是多个配置Bean

package org.springframework.context.annotation;

/**
 * @since 3.1
 */
public interface ImportSelector {
 
 String[] selectImports(AnnotationMetadata importingClassMetadata);
}

@Enable注解方式自定义实现

激活HelloWorld模块，使用@Import(HelloWorldConfiguration.class)直接注入HelloWorldConfiguration模块

package top.fjy8018.springboot.annotation;

import org.springframework.context.annotation.Import;
import top.fjy8018.springboot.configuration.HelloWorldConfiguration;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:35
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Import(HelloWorldConfiguration.class)
public @interface EnableHelloWorld {}

目标类，其注解@Configuration在Spring3.0时引入

package top.fjy8018.springboot.configuration;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:31
 */
@Configuration
public class HelloWorldConfiguration {

 /**
 * 方法名称即Bean名称
 * @return Hello World!
 */
 @Bean
 public String helloWorld() {
 return "Hello world!";
 }
}

package org.springframework.context.annotation;

/**
 * @since 3.0
 */
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Component
public @interface Configuration {
 @AliasFor(annotation = Component.class)
 String value() default "";
}

@Enable接口编程方式自定义实现

参考Spring包名路径创建自定义Selector，通过HelloWorldImportSelector实现HelloWorldConfiguration的配置Bean导入

package top.fjy8018.springboot.annotation;

import org.springframework.context.annotation.ImportSelector;
import org.springframework.core.type.AnnotationMetadata;
import top.fjy8018.springboot.configuration.HelloWorldConfiguration;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:28
 */
public class HelloWorldImportSelector implements ImportSelector {

 /**
 * @param importingClassMetadata 元数据信息
 * @return 一个或多个配置Bean
 */
 @Override
 public String[] selectImports(AnnotationMetadata importingClassMetadata) {
 return new String[]{HelloWorldConfiguration.class.getName()};
 }
}

激活模块方式变化，使用@Import(HelloWorldImportSelector.class)注入HelloWorldImportSelector模块，在Selector里加载并返回 HelloWorldConfiguration 模块

package top.fjy8018.springboot.annotation;

import org.springframework.context.annotation.Import;
import top.fjy8018.springboot.configuration.HelloWorldConfiguration;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:35
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
//@Import(HelloWorldConfiguration.class)
@Import(HelloWorldImportSelector.class)
public @interface EnableHelloWorld {
}

实现逻辑

graph LR
A(EnableHelloWorld) --> B(HelloWorldImportSelector) 
 B-->C(HelloWorldConfiguration)
 C --> D(helloWorld)

此方法比直接注解驱动更有弹性，可增加switch条件返回不同的模块，源码样例CachingConfigurationSelector

package org.springframework.cache.annotation;
public class CachingConfigurationSelector extends AdviceModeImportSelector<EnableCaching> {
 
 @Override
 public String[] selectImports(AdviceMode adviceMode) {
 switch (adviceMode) {
 case PROXY:
 return getProxyImports();
 case ASPECTJ:
 return getAspectJImports();
 default:
 return null;
 }
 }
 ...
}

Spring条件装配

从 Spring Framework 3.1 开始，允许在 Bean 装配时增加前置条件判断

Spring 注解	场景说明	起始版本
`@Profile`	配置化条件装配	3.1
`@Conditional`	编程条件装配	4.0

条件注解实现方式

配置方式 - `@Profile`

在spring4后实现方式发生变化，依赖@Conditional和ProfileCondition实现

package org.springframework.context.annotation;

/**
* @since 3.1
*/
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(ProfileCondition.class)
public @interface Profile {...}

package org.springframework.context.annotation;

/**
 * @since 4.0
 */
class ProfileCondition implements Condition {...}

通过上下文对象和注解元信息判断

package org.springframework.context.annotation;

/**
 * @since 4.0
 */
@FunctionalInterface
public interface Condition {
 boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata);
}

编程方式 - `@Conditional`

ConditionalOnClass用于判断标识类在当前classpath下是否存在，该注解使用@Conditional搭配OnClassCondition实现

package org.springframework.boot.autoconfigure.condition;

/**
 * {@link Conditional} that only matches when the specified classes are on the classpath.
 */
@Target({ ElementType.TYPE, ElementType.METHOD })
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(OnClassCondition.class)
public @interface ConditionalOnClass {...}

package org.springframework.boot.autoconfigure.condition;

/**
 * {@link Condition} that checks if properties are defined in environment.
 * @since 1.1.0
 * @see ConditionalOnProperty
 */
@Order(Ordered.HIGHEST_PRECEDENCE + 40)
class OnPropertyCondition extends SpringBootCondition {
 @Override
 public ConditionOutcome getMatchOutcome(ConditionContext context,
 AnnotatedTypeMetadata metadata) {
 // 获取元信息
 List<AnnotationAttributes> allAnnotationAttributes = annotationAttributesFromMultiValueMap(
 metadata.getAllAnnotationAttributes(
 ConditionalOnProperty.class.getName()));
 List<ConditionMessage> noMatch = new ArrayList<>();
 List<ConditionMessage> match = new ArrayList<>();
 for (AnnotationAttributes annotationAttributes : allAnnotationAttributes) {
 ConditionOutcome outcome = determineOutcome(annotationAttributes,
 context.getEnvironment());
 (outcome.isMatch() ? match : noMatch).add(outcome.getConditionMessage());
 }
 if (!noMatch.isEmpty()) {
 return ConditionOutcome.noMatch(ConditionMessage.of(noMatch));
 }
 return ConditionOutcome.match(ConditionMessage.of(match));
 }
 ...
}

OnPropertyCondition为抽象类，实际仍然是Condition接口的实现

package org.springframework.boot.autoconfigure.condition;

public abstract class SpringBootCondition implements Condition {...}

ConditionalOnProperty用于判断配置在上下文中是否存在

package org.springframework.boot.autoconfigure.condition;

/**
 * @since 1.1.0
 */
@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.TYPE, ElementType.METHOD })
@Documented
@Conditional(OnPropertyCondition.class)
public @interface ConditionalOnProperty {...}

自定义条件装配

目标服务

package top.fjy8018.springboot.service;

/**
 * 计算模块
 * @author F嘉阳
 * @date 2018-09-09 11:56
 */
public interface CalculateService {

 /**
 * 多整数求和
 * @param values 多个整数
 * @return 累加值
 */
 Integer sum(Integer... values);
}

基于配置方式

服务实现类，分别使用 for循环实现和Lambda实现，均标注@Service作为Bean自动注入

指定@Profile("Java7")，当条件为Java7时才装配

package top.fjy8018.springboot.service.impl;

import org.springframework.context.annotation.Profile;
import org.springframework.stereotype.Service;
import top.fjy8018.springboot.service.CalculateService;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:57
 */
@Profile("Java7")
@Service
public class Java7CalculateServiceImpl implements CalculateService {

 /**
 * 多整数求和
 *
 * @param values 多个整数
 * @return 累加值
 */
 @Override
 public Integer sum(Integer... values) {
 System.out.println("计算模块java7 for循环实现");
 int sum = 0;
 for (int i = 0; i < values.length; i++) {
 sum += values[i];
 }
 return sum;
 }
}

指定@Profile("Java8")，当条件为Java7时才装配

package top.fjy8018.springboot.service.impl;

import org.springframework.context.annotation.Profile;
import org.springframework.stereotype.Service;
import top.fjy8018.springboot.service.CalculateService;

import java.util.stream.Stream;

/**
 * @author F嘉阳
 * @date 2018-09-09 11:57
 */
@Profile("Java8")
@Service
public class Java8CalculateServiceImpl implements CalculateService {

 /**
 * 多整数求和
 * @param values 多个整数
 * @return 累加值
 */
 @Override
 public Integer sum(Integer... values) {
 System.out.println("计算模块java8 Lambda实现");
 return Stream.of(values).reduce(0, Integer::sum);
 }
}

启动类，通过scanBasePackages指定报名扫描，该方法与@ComponentScan(basePackages = "xxx")效果一致

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import top.fjy8018.springboot.service.CalculateService;

/**
 * @author F嘉阳
 * @date 2018-09-09 12:06
 */
@SpringBootApplication(scanBasePackages = "top.fjy8018.springboot.service")
public class CalculateServiceDemo {

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(CalculateServiceDemo.class)
 .profiles("Java8")
 .web(WebApplicationType.NONE)
 .run(args);

 // 直接用类获取bean
 CalculateService calculateService = applicationContext.getBean(CalculateService.class);

 // calculateService Bean 是否存在，若不添加Profile则必有报错
 Integer[] values = new Integer[10];
 for (int i = 1; i <= values.length; i++) {
 values[i - 1] = i;
 }
 System.out.println("calculateService Bean 求和: " + calculateService.sum(values));

 // 关闭上下文
 applicationContext.close();
 }
}

执行结果——java8

执行结果——java7

若不指定配置则报错

基于编程方式

系统属性条件判断实现类

package top.fjy8018.springboot.condition;

import org.springframework.context.annotation.Condition;
import org.springframework.context.annotation.ConditionContext;
import org.springframework.core.type.AnnotatedTypeMetadata;

import java.util.Map;

/**
 * @author F嘉阳
 * @date 2018-09-09 17:15
 */
public class OnSystemPropertyCondition implements Condition {

 /**
 * 条件匹配规则，此处可实现复杂逻辑判断
 * @param context 上下文
 * @param metadata 元信息
 * @return 是否匹配
 */
 @Override
 public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {

 // 通过类名称取元信息
 Map<String, Object> annotationAttributes = metadata.getAnnotationAttributes(ConditionalOnSystemProperty.class.getName());

 String propertyName = String.valueOf(annotationAttributes.get("name"));
 String propertyValue = String.valueOf(annotationAttributes.get("value"));

 // 取系统属性
 String systemPropertyValue = System.getProperty(propertyName);

 return propertyValue.equals(systemPropertyValue);
 }

Java 系统属性条件判断类装配，系统属性通过Java API获取java.lang.System#getProperty(java.lang.String)方法获取的属性值

package top.fjy8018.springboot.condition;

import org.springframework.context.annotation.Conditional;

import java.lang.annotation.*;

/**
 * @author F嘉阳
 * @date 2018-09-09 17:14
 */
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(OnSystemPropertyCondition.class)
public @interface ConditionalOnSystemProperty {

 /**
 * Java 系统属性名称
 * @return 系统属性名称
 */
 String name();

 /**
 * 系统属性值
 * @return 属性值
 */
 String value();
}

启动类，helloWorld仅在系统条件满足时自动装配，注解若作用于类上语义相同

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.Bean;
import top.fjy8018.springboot.condition.ConditionalOnSystemProperty;

/**
 * 系统属性条件引导类
 *
 * @author F嘉阳
 * @date 2018-09-09 17:26
 */
public class SystemPropertyConditionBootstrap {

 @Bean
 @ConditionalOnSystemProperty(name = "user.name", value = "11860")
 public String helloWorld() {
 return "Hello SpringBoot2.0";
 }

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(SystemPropertyConditionBootstrap.class)
 .web(WebApplicationType.NONE)
 .run(args);

 // 通过类型和方法名获取Bean
 String helloWorldBean = applicationContext.getBean("helloWorld", String.class);
 // 若系统参数条件不匹配则报找不到Bean错误，以此实现编程方式条件加载
 System.out.println("ConditionalOnSystemProperty 加载内容：" + helloWorldBean);

 applicationContext.close();
 }
}

执行结果

条件不满足则报错

Spring Boot 自动装配

在 Spring Boot 场景下，基于规约大于配置的原则，实现 Spring 组件自动装配的目的（0配置）。

装配: 模式注解、@Eηable模块、条件装配、工厂加载机制
实现: 激活自动装配、实现自动装配、配置自动装配实现

底层装配技术

Spring 模式注解装配
Spring @Enable 模块装配
Spring 条件装配装配
Spring 工厂加载机制
- 实现类： SpringFactoriesLoader
- 配置资源： META-INF/spring.factories

package org.springframework.core.io.support;

/**
 * @since 3.2
 */
public abstract class SpringFactoriesLoader {

 public static final String FACTORIES_RESOURCE_LOCATION = "META-INF/spring.factories";
 
 public static <T> List<T> loadFactories(Class<T> factoryClass, @Nullable ClassLoader classLoader) {
 Assert.notNull(factoryClass, "'factoryClass' must not be null");
 // 获取类加载器
 ClassLoader classLoaderToUse = classLoader;
 if (classLoaderToUse == null) {
 classLoaderToUse = SpringFactoriesLoader.class.getClassLoader();
 }
 List<String> factoryNames = loadFactoryNames(factoryClass, classLoaderToUse);
 if (logger.isTraceEnabled()) {
 logger.trace("Loaded [" + factoryClass.getName() + "] names: " + factoryNames);
 }
 List<T> result = new ArrayList<>(factoryNames.size());
 // 迭代装配
 for (String factoryName : factoryNames) {
 result.add(instantiateFactory(factoryName, factoryClass, classLoaderToUse));
 }
 AnnotationAwareOrderComparator.sort(result);
 return result;
 }
 ...
}

自动装配举例

参考 META-INF/spring.factories

实现方法

激活自动装配 - @EnableAutoConfiguration
实现自动装配 - XXXAutoConfiguration
配置自动装配实现 - META-INF/spring.factories

目的在于使用META-INF/spring.factories文件配置替代@ComponentScan手动装配注解，从而实现自动装配

而Spring大多数自动装配都由starter模块负责

自定义自动装配

设计逻辑

HelloWorldAutoConfiguration

条件判断： user.name == "Mercy"
模式注解： @Configuration
@Enable 模块：流程图

引导类

package top.fjy8018.springboot.bootstrap;

import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.context.ConfigurableApplicationContext;

/**
 * @author F嘉阳
 * @date 2018-09-10 17:13
 */
@EnableAutoConfiguration
public class HelloWorldAutoConfigurationBootstrap {

 public static void main(String[] args) {
 ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(HelloWorldAutoConfigurationBootstrap.class)
 .web(WebApplicationType.NONE)
 .run(args);

 // 通过bean名称获取helloWorld Bean
 String helloWorldBean = applicationContext.getBean("helloWorld", String.class);

 // helloWorld Bean 是否存在
 System.out.println("helloWorld Bean 内容: " + helloWorldBean);

 // 关闭上下文
 applicationContext.close();
 }
}

配置Bean，使用模式注解标注 @Configuration，该类实际上实现了条件装配、模式注解、自动装配

package top.fjy8018.springboot.configuration;

import org.springframework.context.annotation.Configuration;
import top.fjy8018.springboot.annotation.EnableHelloWorld;
import top.fjy8018.springboot.condition.ConditionalOnSystemProperty;

/**
 * @author F嘉阳
 * @date 2018-09-10 17:23
 */
@Configuration
@EnableHelloWorld
@ConditionalOnSystemProperty(name = "user.name", value = "11860")
public class HelloWorldAutoConfiguration {}

创建spring.factories，每行全类名顶格开始，防止在某些版本下产生不稳定的BUG

# 自动装配
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
top.fjy8018.springboot.configuration.HelloWorldAutoConfiguration

执行结果与之前一致

SpringBoot源码分析1——总览

Thu, 10 Dec 2020 17:11:00 +0800

SpringBoot 2.0 特点

组件自动装配

规约大于配置,专注核心业务

模式注解、@ Enable模块、条件装配、加载机制

外部化配置

一次构建、按需调配,到处运行

Environment抽象、生命周期、破坏性变更

嵌入式容器

内置容器、无需部署、独立运行

Servlet Web容器、 Reactive Web容器

Spring Boot Starter

简化依赖、按需装配、自我包含

依赖管理、装配条件、装配顺序

Production-Ready

一站式运维、生态无缝整合

健康检査、数据指标、@ Endpoint管控

SpringBoot 与Java EE规范

Web: Servlet( JSR-315, JSR-340）
SQL: JDBC(SR-221)
数据校验: Bean validation(JSR303、JSR-349)
缓存: Java Caching API(JSR-107)
Web Sockets: Java api for Web Socket(JSR-356
Web services: JAX-WS( JSR-224）
Java管理:JMX(JSR3)
消息:JMS(JSR-914)

核心特性介绍

三大特性

组件自动装配: Web mvc、 Web flux、JDBC等
嵌入式Web容器: Tomcat、 Jetty以及 Undertow
生产准备特性:指标、健康检査、外部化配置等

组件自动装配

激活： @EnableAutoConfiguration 配置： /META-INF/spring.factories 实质为key-value形式实现： XXXAutoConfiguration

官方文档样例

import org.springframework.boot.*;
import org.springframework.boot.autoconfigure.*;
import org.springframework.web.bind.annotation.*;

@RestController
@EnableAutoConfiguration
public class Example {

 @RequestMapping("/")
 String home() {
 return "Hello World!";
 }

 public static void main(String[] args) throws Exception {
 SpringApplication.run(Example.class, args);
 }

}

实际上@SpringBootApplication已经包含该注解

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@SpringBootConfiguration
@EnableAutoConfiguration
@ComponentScan(excludeFilters = {
 @Filter(type = FilterType.CUSTOM, classes = TypeExcludeFilter.class),
 @Filter(type = FilterType.CUSTOM, classes = AutoConfigurationExcludeFilter.class) })
public @interface SpringBootApplication {
}

在IDEA中使用ctrl+shift+n可查看Spring源码中包含大量spring.factories文件

样例文件：

org/springframework/boot/spring-boot-autoconfigure/2.0.4.RELEASE/spring-boot-autoconfigure-2.0.4.RELEASE.jar!/META-INF/spring.factories

# Initializers
org.springframework.context.ApplicationContextInitializer=\
org.springframework.boot.autoconfigure.SharedMetadataReaderFactoryContextInitializer,\
org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener

# Application Listeners
org.springframework.context.ApplicationListener=\
org.springframework.boot.autoconfigure.BackgroundPreinitializer

# Auto Configuration Import Listeners
org.springframework.boot.autoconfigure.AutoConfigurationImportListener=\
org.springframework.boot.autoconfigure.condition.ConditionEvaluationReportAutoConfigurationImportListener

# Auto Configuration Import Filters
org.springframework.boot.autoconfigure.AutoConfigurationImportFilter=\
org.springframework.boot.autoconfigure.condition.OnClassCondition

# 自动装配配置，自动装载Bean列表
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
org.springframework.boot.autoconfigure.admin.SpringApplicationAdminJmxAutoConfiguration,\
org.springframework.boot.autoconfigure.web.servlet.WebMvcAutoConfiguration,\
...

WebMvcAutoConfiguration源码，其包含大量条件装配，若未检测到该类的依赖，则不会被装配

@Configuration
@ConditionalOnWebApplication(type = Type.SERVLET)
@ConditionalOnClass({ Servlet.class, DispatcherServlet.class, WebMvcConfigurer.class })
@ConditionalOnMissingBean(WebMvcConfigurationSupport.class)
@AutoConfigureOrder(Ordered.HIGHEST_PRECEDENCE + 10)
@AutoConfigureAfter({ DispatcherServletAutoConfiguration.class,
 ValidationAutoConfiguration.class })
public class WebMvcAutoConfiguration {
...

嵌入式Web容器

Web servlet: Tomcat、 Jetty和 Undertow
Web reactive Netty Web Server

生产准备特性

指标:/ actuator/ metrics
健康检查:/ actuator/ health
外部化配置:/ actuator/ configprops

Web应用介绍

传统 Servlet应用概览

Servlet组件: Servlet、 Filter、 Listener
Servlet注册: Servlet注解、 Spring Bean、 RegistrationBean
异步非阻塞:异步 Servlet、非阻塞 Servlet

传统 Servelt 应用

加入springboot应用通用依赖前缀多为spring-boot-starter-xxx

加入Web依赖

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
</dependency>

Maven依赖变化

由此可看出默认的Web容器为Tomcat（内嵌Servlet容器）

Servlet 组件

Servlet
- 实现
  - @WebServlet
  - HttpServlet
  - 注册
- URL 映射
  - @WebServlet(urlPatterns = "/my/servlet")
- 注册 @ServletComponentScan(basePackages = "top.fjy8018.springboot.web.servlet")
Filter
Listener

Servlet 注册

Servlet 注解

@ServletComponentScan +
- @WebServlet
- @WebFilter
- @WebListener

Spring Bean

@Bean +
- Servlet
- Filter
- Listener
RegistrationBean
- ServletRegistrationBean
- FilterRegistrationBean
- ServletListenerRegistrationBean

Servlet 实现

参考源码包位置，由于出现了Reactive Web容器，SpringBoot 1.x的包路径与2.x有所不同，2.x包路径变更为org.springframework.boot.autoconfigure.web.servlet

HttpServlet

package top.fjy8018.springboot.web.servlet;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

/**
 * 结合servlet规范注册
 *
 * @author F嘉阳
 * @date 2018-09-08 19:17
 */
@WebServlet(urlPatterns = "/myservlet")
public class MyServlet extends HttpServlet {

 @Override
 protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
 resp.getWriter().print("hello world");
 }
}

Servlet 3.0规范要求必须实现HttpServlet
其内部提供多种方法实现（ctrl+F12）
urlPatterns = "/myservlet"指明浏览器映射

注册

在主类中注册（包扫描）

package top.fjy8018.springboot;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.servlet.ServletComponentScan;

@SpringBootApplication
@ServletComponentScan(basePackages = "top.fjy8018.springboot.web.servlet")
public class OverviewApplication {

 public static void main(String[] args) {
 SpringApplication.run(OverviewApplication.class, args);
 }
}

启动后从控制台可看出映射已经成功

异步非阻塞 Servlet

异步 Servlet

javax.servlet.ServletRequest#startAsync()
javax.servlet.AsyncContext

非阻塞 Servlet

javax.servlet.ServletInputStream#setReadListener
- javax.servlet.ReadListener
javax.servlet.ServletOutputStream#setWriteListener
- javax.servlet.WriteListener

异步非阻塞Servlet实现

package top.fjy8018.springboot.web.servlet;

import javax.servlet.AsyncContext;
import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

/**
 * 结合servlet规范注册
 * asyncSupported指定异步servlet，默认false
 *
 * @author F嘉阳
 * @date 2018-09-08 19:17
 */
@WebServlet(urlPatterns = "/myservlet", asyncSupported = true)
public class MyServlet extends HttpServlet {

 @Override
 protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
 AsyncContext asyncContext = req.startAsync();

 // 执行任务
 asyncContext.start(() -> {
 try {
 resp.getWriter().print("hello world");

 // 触发完成，必须步骤，否则客户端发生超时
 asyncContext.complete();
 } catch (IOException e) {
 e.printStackTrace();
 }
 });
 }
}

asyncSupported = true 启用异步支持，否则报500错误

Spring Web MVC 应用介绍

Web mvo视图:模板引擎、内容协商、异常处理等
Web mVc rest:资源服务、资源跨域、服务发现等

Web MVC 视图

ViewResolver
View

源码

package org.springframework.web.servlet;

import java.util.Locale;
import org.springframework.lang.Nullable;

public interface ViewResolver {
 @Nullable
 View resolveViewName(String viewName, Locale locale) throws Exception;

}

通过指定名称和路径加载视图

视图渲染View

package org.springframework.web.servlet;

public interface View {
 ...

 void render(@Nullable Map<String, ?> model, HttpServletRequest request, HttpServletResponse response)
 throws Exception;

}

提供接口供不同的模板引擎实现ViewResolver

模板引擎

Thymeleaf
Freemarker
JSP

若同时有多种模板引擎，则需要内容协商

内容协商

ContentNegotiationConfigurer
ContentNegotiationStrategy
ContentNegotiatingViewResolver

异常处理

@ExceptionHandler
HandlerExceptionResolver
- ExceptionHandlerExceptionResolver
BasicErrorController (Spring Boot默认白页404)

源码

@Controller
@RequestMapping("${server.error.path:${error.path:/error}}")
public class BasicErrorController extends AbstractErrorController {
 
 @RequestMapping(produces = "text/html")
 public ModelAndView errorHtml(HttpServletRequest request,
 HttpServletResponse response) {
 HttpStatus status = getStatus(request);
 Map<String, Object> model = Collections.unmodifiableMap(getErrorAttributes(
 request, isIncludeStackTrace(request, MediaType.TEXT_HTML)));
 response.setStatus(status.value());
 ModelAndView modelAndView = resolveErrorView(request, response, status, model);
 return (modelAndView != null) ? modelAndView : new ModelAndView("error", model);
 }
 
 @RequestMapping
 @ResponseBody
 public ResponseEntity<Map<String, Object>> error(HttpServletRequest request) {
 Map<String, Object> body = getErrorAttributes(request,
 isIncludeStackTrace(request, MediaType.ALL));
 HttpStatus status = getStatus(request);
 return new ResponseEntity<>(body, status);
 }
}

其根据浏览器类型提供两种方式渲染，分别为html和json

{
 "timestamp": "2019-03-13T04:47:18.445+0000",
 "status": 404,
 "error": "Not Found",
 "message": "Not Found",
 "path": "/hello"
}

Web MVC REST

资源服务

@RequestMapping
@GetMapping
@ResponseBody
@RequestBody

从源码分析可知@GetMapping是从@RequestMapping变化而来，通过委派（别名）方式实现，并且在SpringMVC 4.3后引入

package org.springframework.web.bind.annotation;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import org.springframework.core.annotation.AliasFor;

/**
 * Annotation for mapping HTTP {@code GET} requests onto specific handler
 * methods.
 *
 * <p>Specifically, {@code @GetMapping} is a <em>composed annotation</em> that
 * acts as a shortcut for {@code @RequestMapping(method = RequestMethod.GET)}.
 *
 *
 * @author Sam Brannen
 * @since 4.3
 * @see PostMapping
 * @see PutMapping
 * @see DeleteMapping
 * @see PatchMapping
 * @see RequestMapping
 */
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@RequestMapping(method = RequestMethod.GET)
public @interface GetMapping {
 /**
 * 委派目标 {@link RequestMapping#name}.
 */
 @AliasFor(annotation = RequestMapping.class)
 String name() default "";
 ...
}

资源跨域

CrossOrigin
WebMvcConfigurer#addCorsMappings
传统解决方案
- IFrame
- JSONP

源码

package org.springframework.web.servlet.config.annotation;

/**
 * @since 3.1
 */
public interface WebMvcConfigurer {
 /**
 * 跨域配置
 * @since 4.2
 */
 default void addCorsMappings(CorsRegistry registry) {
 }
}

注解实现方式

package org.springframework.web.bind.annotation;

/**
 * @since 4.2
 */
@Target({ ElementType.METHOD, ElementType.TYPE })
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface CrossOrigin {...}

两者均来自springframework，属于不同的实现维度（接口编程 / 注解驱动）

服务发现

HATEOS

核心组件

DispatcherServlet——Servlet实现请求转发到映射URL
HandlerMapping——处理映射URL方法
HandlerAdapter——方法处理
ViewResolver——交由视图处理
…

Spring WebFlux （Spring 5+）

Reactor 基础

Java Lambda —— Reactive Stream实现（JDK 9 flow api、RxJava）
Mono
Flux

Web Flux 核心

Web MVC 注解兼容

@Controller
@RequestMapping
@ResponseBody
@RequestBody
…

函数式声明

RouterFunction

异步非阻塞

Servlet 3.1 +
Netty Reactor

使用场景

页面渲染
REST 应用

性能测试

国外博客

结论：Webflux 性能不会显著提升，但Webflux提升吞吐量

Web Server 应用

切换 Web server
自定义 Servlet Web server
自定义 Reactive Web server

切换 Web Server

切换其他 Servlet 容器

Tomcat -> Jetty

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-web</artifactId>
 <exclusions>
 <!-- tomcat加载优先级高于jetty，若不加排除，则仍然加载tomcat -->
 <exclusion>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-tomcat</artifactId>
 </exclusion>
 </exclusions>
</dependency>
<!-- 使用 Jetty 容器 -->
<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-jetty</artifactId>
</dependency>

替换 Servlet 容器

WebFlux

<!--使用webflux必须注释web依赖，当web和flux共存时，web优先级更高-->
<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

注释后相关类和的注解也已经失效，需要删除

@SpringBootApplication
//@ServletComponentScan(basePackages = "top.fjy8018.springboot.web.servlet")
public class SpringbootApplication {

 public static void main(String[] args) {
 SpringApplication.run(SpringbootApplication.class, args);
 }
}

依赖变化

启动后变为Netty容器

自定义 Servlet Web Server

WebServerFactoryCustomizer

SpringBoot 2.0 新增支持接口

package org.springframework.boot.web.server;

/**
 * @since 2.0.0
 * @see WebServerFactoryCustomizerBeanPostProcessor
 */
@FunctionalInterface
public interface WebServerFactoryCustomizer<T extends WebServerFactory> {
 
 void customize(T factory);
}

该接口为主接口，其下有大量实现，IDEA通过ctrl+alt+B或者ctrl+alt+左键点击查看

其中TomcatServletWebServerFactoryCustomizer就是实现类之一，但在启用Webflux后该类无效

package org.springframework.boot.autoconfigure.web.servlet;

/**
 * {@link WebServerFactoryCustomizer} to apply {@link ServerProperties} to Tomcat web
 * servers.
 *
 * @author Brian Clozel
 * @author Phillip Webb
 * @since 2.0.0
 */
public class TomcatServletWebServerFactoryCustomizer
 implements WebServerFactoryCustomizer<TomcatServletWebServerFactory>, Ordered {
 @Override
 public void customize(TomcatServletWebServerFactory factory) {
 ServerProperties.Tomcat tomcatProperties = this.serverProperties.getTomcat();
 if (!ObjectUtils.isEmpty(tomcatProperties.getAdditionalTldSkipPatterns())) {
 factory.getTldSkipPatterns()
 .addAll(tomcatProperties.getAdditionalTldSkipPatterns());
 }
 if (tomcatProperties.getRedirectContextRoot() != null) {
 customizeRedirectContextRoot(factory,
 tomcatProperties.getRedirectContextRoot());
 }
 if (tomcatProperties.getUseRelativeRedirects() != null) {
 customizeUseRelativeRedirects(factory,
 tomcatProperties.getUseRelativeRedirects());
 }
 }
 ...
}

自定义 Reactive Web Server

ReactiveWebServerFactoryCustomizer

其源码显示提供了最基础的实现，包括端口、SSL等，允许自行扩展实现

package org.springframework.boot.autoconfigure.web.reactive;

/**
 * {@link WebServerFactoryCustomizer} to apply {@link ServerProperties} to reactive
 * servers.
 * @since 2.0.0
 */
public class ReactiveWebServerFactoryCustomizer implements
 WebServerFactoryCustomizer<ConfigurableReactiveWebServerFactory>, Ordered {

 private final ServerProperties serverProperties;

 public ReactiveWebServerFactoryCustomizer(ServerProperties serverProperties) {
 this.serverProperties = serverProperties;
 }

 @Override
 public int getOrder() {
 return 0;
 }

 @Override
 public void customize(ConfigurableReactiveWebServerFactory factory) {
 PropertyMapper map = PropertyMapper.get().alwaysApplyingWhenNonNull();
 map.from(this.serverProperties::getPort).to(factory::setPort);
 map.from(this.serverProperties::getAddress).to(factory::setAddress);
 map.from(this.serverProperties::getSsl).to(factory::setSsl);
 map.from(this.serverProperties::getCompression).to(factory::setCompression);
 map.from(this.serverProperties::getHttp2).to(factory::setHttp2);
 }
}

数据相关介绍

关系型数据

JDBC:数据源、 JdbcTemplate、自动装配
JPA:实体映射关系、实体操作、自动装配
事务∶ Spring事务抽象、JDBC事务处理、自动装配

JDBC

依赖

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>

依赖变化，在SpringBoot2.0 后依赖新的连接池 HikariCP

数据源

javax.sql.DataSource
JdbcTemplate
自动装配
DataSourceAutoConfiguration

DataSource

连接池多数实现是对DataSource的二次封装，提高执行效率

package javax.sql;

/**
 * @since 1.4
 */

public interface DataSource extends CommonDataSource, Wrapper {
 Connection getConnection() throws SQLException;
 
 Connection getConnection(String username, String password)
 throws SQLException;
}

自动装配

JdbcTemplate自动装配，其依赖数据源配置项，故有@AutoConfigureAfter注解标识，该类执行于DataSourceAutoConfiguration之后，额外还有多个条件装配的标识

package org.springframework.boot.autoconfigure.jdbc;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for {@link JdbcTemplate} and
 * {@link NamedParameterJdbcTemplate}.
 * @since 1.4.0
 */
@Configuration
@ConditionalOnClass({ DataSource.class, JdbcTemplate.class })
@ConditionalOnSingleCandidate(DataSource.class)
@AutoConfigureAfter(DataSourceAutoConfiguration.class)
@EnableConfigurationProperties(JdbcProperties.class)
public class JdbcTemplateAutoConfiguration {...}

DataSourceAutoConfiguration

package org.springframework.boot.autoconfigure.jdbc;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for {@link DataSource}.
 */
@Configuration
@ConditionalOnClass({ DataSource.class, EmbeddedDatabaseType.class })
@EnableConfigurationProperties(DataSourceProperties.class)
@Import({ DataSourcePoolMetadataProvidersConfiguration.class,
 DataSourceInitializationConfiguration.class })
public class DataSourceAutoConfiguration {...}

该类属于自动装配，故必然在spring.factories中，而数据源下即JdbcTemplateAutoConfiguration

此时直接启动会得到缺少数据源的错误报告

***************************
APPLICATION FAILED TO START
***************************

Description:

Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.

Reason: Failed to determine a suitable driver class


Action:

Consider the following:
 If you want an embedded database (H2, HSQL or Derby), please put it on the classpath.
 If you have database settings to be loaded from a particular profile you may need to activate it (no profiles are currently active).

JPA

依赖

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>

其属于SpringData项目，基于hibernate实现，同时也自动依赖JDBC

实体映射关系

@javax.persistence.OneToOne
@javax.persistence.OneToMany
@javax.persistence.ManyToOne
@javax.persistence.ManyToMany
…

实体操作

javax.persistence.EntityManager

属于Java类库的API

package javax.persistence;

public interface EntityManager {

 /**
 * 存
 */
 public void persist(Object entity);

 /**
 * 更新
 */
 public <T> T merge(T entity);

 /**
 * 删除
 */
 public void remove(Object entity);

 /**
 * 查询以及多个重载方法
 */
 public <T> T find(Class<T> entityClass, Object primaryKey);

 public <T> T find(Class<T> entityClass, Object primaryKey,
 Map<String, Object> properties);
 ...
}

自动装配

HibernateJpaAutoConfiguration

该类也同属于自动装配，且需要JDBC提供数据源，故其加载顺序标识为DataSourceAutoConfiguration之后

package org.springframework.boot.autoconfigure.orm.jpa;

/**
 * {@link EnableAutoConfiguration Auto-configuration} for Hibernate JPA.
 */
@Configuration
@ConditionalOnClass({ LocalContainerEntityManagerFactoryBean.class, EntityManager.class })
@Conditional(HibernateEntityManagerCondition.class)
@EnableConfigurationProperties(JpaProperties.class)
@AutoConfigureAfter({ DataSourceAutoConfiguration.class })
@Import(HibernateJpaConfiguration.class)
public class HibernateJpaAutoConfiguration {...}

事务（Transaction）

依赖

<dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-tx</artifactId>
</dependency>

Spring 事务抽象

PlatformTransactionManager

该接口属于事务根接口，不仅限于数据库事务，默认实现包含分布式事务JtaTransactionManager和数据库事务DataSourceTransactionManager

package org.springframework.transaction;

import org.springframework.lang.Nullable;

/**
 * <p>The default implementations of this strategy interface are
 * {@link org.springframework.transaction.jta.JtaTransactionManager} and
 * {@link org.springframework.jdbc.datasource.DataSourceTransactionManager}

 * @see org.springframework.transaction.support.TransactionTemplate
 * @see org.springframework.transaction.interceptor.TransactionInterceptor
 * @see org.springframework.transaction.interceptor.TransactionProxyFactoryBean
 */
public interface PlatformTransactionManager {...}

JDBC 事务处理

DataSourceTransactionManager

内部通过IOC自动注入数据源接口

package org.springframework.jdbc.datasource;

@SuppressWarnings("serial")
public class DataSourceTransactionManager extends AbstractPlatformTransactionManager
 implements ResourceTransactionManager, InitializingBean {

 @Nullable
 private DataSource dataSource;
 ...
}

自动装配

TransactionAutoConfiguration

自动装配的条件是Spring事务依赖存在PlatformTransactionManager

package org.springframework.boot.autoconfigure.transaction;

/**
 * {@link org.springframework.boot.autoconfigure.EnableAutoConfiguration
 * Auto-configuration} for Spring transaction.
 * @since 1.3.0
 */
@Configuration
@ConditionalOnClass(PlatformTransactionManager.class)
@AutoConfigureAfter({ JtaAutoConfiguration.class, HibernateJpaAutoConfiguration.class,
 DataSourceTransactionManagerAutoConfiguration.class,
 Neo4jDataAutoConfiguration.class })
@EnableConfigurationProperties(TransactionProperties.class)
public class TransactionAutoConfiguration {...}

功能扩展介绍

Spring Boot 应用

Spring Application:失败分析、应用特性、事件监听等
Spring Boot配置:外部化配置、 Profile、配置属性
Spring Boot starter: Starter开发、最佳实践

SpringApplication

失败分析

FailureAnalysisReporter

该类在Spring Boot 1.4后开始支持

package org.springframework.boot.diagnostics;

/**
 * Reports a {@code FailureAnalysis} to the user.
 * @since 1.4.0
 */
@FunctionalInterface
public interface FailureAnalysisReporter {...}

应用特性

SpringApplication Fluent API

原始默认启动方法为SpringApplication.run(SpringbootApplication.class, args)其实际返回可配置上下文对象ConfigurableApplicationContext，可通过Builder模式重写

package top.fjy8018.springboot;

@SpringBootApplication
public class SpringbootApplication {

 public static void main(String[] args) {
 // Fluent API
 // 等价建造者设计模式调用方法
 new SpringApplicationBuilder(SpringbootApplication.class)
 // 不使用web模式
 .web(WebApplicationType.NONE)
 // 自定义参数
 .properties("name=fjy")
 .run(args);
 }
}

2.0 版本下提供3种WEB容器类型

package org.springframework.boot;

/**
 * @since 2.0.0
 */
public enum WebApplicationType {
 NONE,
 SERVLET,
 REACTIVE
}

Spring Boot 配置

外部化配置

ConfigurationProperty
@Profile

从SpringBoot 2.0 后引入，Origin类型支持其他配资源引入，非传统仅支持键值对，由此支持多数据源

package org.springframework.boot.context.properties.source;

/**
 * @since 2.0.0
 */
public final class ConfigurationProperty
 implements OriginProvider, Comparable<ConfigurationProperty> {
 
 private final ConfigurationPropertyName name;

 private final Object value;

 private final Origin origin;
 ...
}

官方文档

此处官方文档存在一定疏漏，部分场景配置无法实现

Spring Boot uses a very particular PropertySource order that is designed to allow sensible overriding of values. Properties are considered in the following order:

Devtools global settings properties on your home directory (~/.spring-boot-devtools.properties when devtools is active).

@TestPropertySource annotations on your tests.

@SpringBootTest#properties annotation attribute on your tests.

Command line arguments.

Properties from SPRING_APPLICATION_JSON (inline JSON embedded in an environment variable or system property).

ServletConfig init parameters.

ServletContext init parameters.

JNDI attributes from java:comp/env.

Java System properties (System.getProperties()).

OS environment variables.

A RandomValuePropertySource that has properties only in random.*.

Profile-specific application properties outside of your packaged jar (application-{profile}.properties and YAML variants).

Profile-specific application properties packaged inside your jar (application-{profile}.properties and YAML variants).

Application properties outside of your packaged jar (application.properties and YAML variants).

Application properties packaged inside your jar (application.properties and YAML variants).

@PropertySource annotations on your @Configuration classes.

Default properties (specified by setting SpringApplication.setDefaultProperties).

Profile支持以注解方式条件开启配置，用于区分开发环境和生产环境等

package org.springframework.context.annotation;

/**
 * @since 3.1
 */
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Conditional(ProfileCondition.class)
public @interface Profile {
 
 String[] value();
}

配置属性

PropertySources

提供两种编程方式：接口、注解，支持保存多个配置，按照17种顺序排列，该接口在SpringBoot中重新设计变为ConfigurationProperty

package org.springframework.core.env;

/**
 * @since 3.1
 * @see PropertySource
 */
public interface PropertySources extends Iterable<PropertySource<?>> {...}

运维管理介绍

官方文档

Spring Boot Actuator

端点（Endpoints）

Web Endpoints
JMX Endpoints

依赖

<!--运维监控-->
<dependency>
 <groupId>org.springframework.boot</groupId> 
 <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

启动效果

访问健康检查接口/actuator/health返回应用状态

{
 "status": "UP"
}

访问根路径

{
 "_links": {
 "self": {
 "href": "http://localhost:8080/actuator",
 "templated": false
 },
 "health": {
 "href": "http://localhost:8080/actuator/health",
 "templated": false
 },
 "info": {
 "href": "http://localhost:8080/actuator/info",
 "templated": false
 }
 }
}

依据SpringBoot打印的进程号或者依据应用名称使用Jconsole监控，该操作使用的为JMX端点

健康检查（Health Checks）

Health
HealthIndicator

使用JMX在默认情况下能获取比Web端点更多的信息，原因是JMX支持使用安全连接，具有更好的安全性

官方文档指出默认状态只暴露health和info，可通过配置文件调整

28.4 Actuator Security
For security purposes, all actuators other than /health and /info are disabled by default. The management.endpoints.web.exposure.include property can be used to enable the actuators.

在application.properties文件中加入配置重启应用

# 暴露所有健康监控点
management.endpoints.web.exposure.include=*

从日志输出可知增加了更多的URL映射

但尽管配置后，JMX显示的信息仍然比Web端更丰富，例如在JMX中不仅看到状态，还能看到磁盘、内存等信息，而Web端仅看到状态

指标（Metrics）

内建 Metrics
Web Endpoint : /actuator/metrics
自定义 Metrics

Web端点列举支持指标

{
 "names": [
 "jvm.memory.max",
 "jvm.gc.memory.promoted",
 "jvm.memory.used",
 "jvm.gc.max.data.size",
 "jvm.gc.pause",
 "jvm.memory.committed",
 "system.cpu.count",
 "logback.events",
 "jvm.buffer.memory.used",
 "jvm.threads.daemon",
 "system.cpu.usage",
 "jvm.gc.memory.allocated",
 "jvm.threads.live",
 "jvm.threads.peak",
 "process.uptime",
 "http.server.requests",
 "process.cpu.usage",
 "jvm.classes.loaded",
 "jvm.classes.unloaded",
 "jvm.gc.live.data.size",
 "jvm.buffer.count",
 "jvm.buffer.total.capacity",
 "process.start.time"
 ]
}

访问/actuator/metrics/jvm.memory.max可查看相关指标，转换后可知最大内存为5.2G

{
 "name": "jvm.memory.max",
 "description": "The maximum amount of memory in bytes that can be used for memory management",
 "baseUnit": "bytes",
 "measurements": [
 {
 "statistic": "VALUE",
 "value": 5595725823
 }
 ],
 "availableTags": [
 {
 "tag": "area",
 "values": [
 "heap",
 "nonheap"
 ]
 },
 {
 "tag": "id",
 "values": [
 "Compressed Class Space",
 "PS Survivor Space",
 "PS Old Gen",
 "Metaspace",
 "PS Eden Space",
 "Code Cache"
 ]
 }
 ]
}

金融企业会计笔记

Thu, 13 Aug 2020 11:20:00 +0800

会计科目分类

按资产性质划分

资产类科目

资产类科目是金融企业各项经济资源的分类名称，用来核算金融企业的各种资金运用结果。

此科目包括流动资产、固定资产、无形资产和其他资产等。

负债类科目

负债类科目是金融企业各项债务的分类名称用来核算金融企业的各种债务性资金来源状况。

此科目包括短期负债、长期负债和其他负债等。

所有者权益类科目

所有者权益类科目是金融企业资本项目的分类名称，用来核算所有者投入资金以及经营所得或亏损情况。

前者包括实收资本和资本公积，后者包括盈余公积和未分配利润。

损益类科目

损益类科目是金融企业各项经营收人和费用支出的分类名称用来核算金融企业的营业收支等情况。

此科目包括利息收入、利息支出、投资收益、营业费用等。(4)损益类科目。

资产负债共同类科目

资产负债共同类科目既能反映商业银行的资产业务，又能反映商业银行的负债业务。

此科目的余额在借方表现为资产，余额在贷方表现为负债。

在编制会计报表时应根据余额方向，分别纳入资产类或负债类科目反映。

按与资产负债表的关系分类

表内科目

表内科目是指反映金融企业资金实际增减变化而纳入资产负债表内的科目

表外科目

表外科目是指反映金融企业确已发生而尚未涉及资金的实际增减变化,或不涉及资金增减变化而不列人资产负债表的科目,如“有价单证”、“空白重要凭证”等

按反映经济业务内容详略程度分类

一级科目
二级科目

记账方法

复式记账法（借贷记账法）

借贷记账法的记账原理。借贷记账法以“资产=负债+所有者权益”的会计平衡公式为记账原理。这一平衡公式体现了资产总额与负债和所有者权益总额之间数量上的平衡关系。
借贷记账法的记账符号。借贷记账法的记账符号是“借”和“贷”。“借”在账户的左方，“贷“在账户的右方。
借贷记账法的记账规则。借贷记账法以“有借必有贷，借贷必相等”为记账规则。
- 资产和费用的增加记入借方
- 资产和费用的减少记入贷方
- 余额反映在借方
- 负债、权益和收入的增加记入贷方
- 负债、权益和收入的减少记入借方
- 余额反映在贷方。遵守记账规则
关键是在把经济业务记入账户之前，正确确定账户对应关系，编制会计分录。
借贷记账法的试算平衡
- 发生额平衡公式
  
  各科目借方本期发生额 = 各科目贷方本期发生额
- 余额平衡公式
  
  各科目借方余额合计=各科目贷方余额合计

单式记账法

单式记账法就是对发生的每一项经济业务只在一个账户中进行登记的记账方法。

单式记账设置收入、付出、余额三栏,业务发生或增加时记“收入”,销账或减少时记“付出”,“余额”表示结存。

账簿

序时账簿

序时账簿通常称为日记账,它是按照全部经济业务完成时间的先后顺序进行登记的账簿。

在实际工作中,按照会计部门收到凭证的先后顺序即按照凭证的编号顺序进行登记

分类账簿

分类账簿是全面、系统地对资金循环和收支活动所作的分类记录。

按照分类的概括程度不同,分类账簿又可分为

明细分类账
总分类账

明细分类账

明细分类账是按账户设置和登记的账簿。

金融企业的明细分类账按每一会计科目下的开户单位或具体核算对象分户立账,并根据记账凭证的业务顺序逐笔连续记载

总分类账

总分类账是按会计科目设置的，是汇总明细分类账的主要工具。

总分类账于每日营业终了，根据各科目日结单借、贷方发生额和今日余额进行登记。

备查账簿

备查账簿是对某些在日记账和分类账等主要账簿中未能记载的事项进行补充登记的账簿。

这种账簿属于辅助性登记，没有固定的格式，可根据需要加以设计。

账务组织

账务组织是指总的账务结构体系,它包括明细核算和综合核算两个系统。明细核算对综合核算起到补充作用,综合核算对明细核算起到统驭作用,二者构成了一套完整的、科学的、严密的账务组织体系

明细核算

明细核算是按账户核算，反映各项资金增减变化及其结果的详细情况的核算系统。

它的具体形式和所使用的账表包括分户账、登记簿、余额表等。

它的核算程序为：根据会计凭证登记分户账或登记簿，再根据分户账或登记簿编制余额表，最后同总账进行核对。

分户账

分户账是企业会计账簿中具体反映经济业务的明细分类账簿，是明细核算的主要形式。分户账的格式分为甲、乙、丙、丁四种。

登记簿

登记簿又称登记卡，是为了适应某些业务需要而设置的专用账簿。登记簿的格式根据业务需要可自行设计，一般都采用收入、付出、余额三栏的通用格式，根据表外科目传票采取单式记账方法进行登记处理。

余额表

余额表是核对总账与分户账余额和计算利息的重要工具。余额表分为计息余额表和一般余额表。

综合核算

综合核算是按会计科目进行核算,综合、概括地反映各科目的资金增减变化情况,由各种总分类账组成的核算系统。

综合核算所采用的主要形式包括科目日结单、总账、日计表等。

其记账程序为:通过业务传票的审核编制科目日结单,然后根据科目日结单登记各科目总账,最后编制日计表

科目日结单

科目日结单又称总账记账凭证，是每天会计科目的当日借贷方发生额和凭证、附件张数的汇总记录，是监督明细账发生额、轧平当日账务的重要工具，也是记载总账的依据。

每日营业终了，应按同一科目的凭证区分现金和转账凭证、借方或贷方发生额，各自加计总数填入科目日结单的有关栏目，并注明凭证张数。

总账

总账是综合核算与明细核算相互核对，统驭明细分户账的主要工具，是各科目的总括记录，也是编制日计表、月计表、业务状况报告表、资产负债表的依据。总账按科目设置，设有借、贷方发生额和借、贷方余额四栏，账页每月更换一次。

总账的发生额，应于每日营业终了，根据各科目日结单的借贷方发生额合计数分别填记；余额根据本科目的上日余额加减本日发生额后求出。

检验科目总账余额是否正确应视不同情况而定

如果上日借方余额大于贷方余额，可使用以下公式进行检验：

上日借方余额-上日贷方余额+本日借方发生额-本日贷方发生额=本日借方余额-本日贷方余额
如果上日贷方余额大于借方余额，可使用以下公式进行检验：

上日贷方余额-上日借方余额+本日贷方发生额-本日借方发生额=本日贷方余额-本日借方余额

日计表

日计表是反映当日业务活动和轧平全部账务的主要工具。日计表应每日编制，日计表上当日发生额和余额应根据总账各科目当日发生额和余额填记。

日计表上借、贷方发生额和借、贷方余额合计数必须各自平衡

存款业务

存款业务的分类

按存款期限划分

活期存款
定期存款

按存款对象划分

单位存款
储蓄存款

按资金性质划分

财政性存款
一般存款

按存款产生的来源划分

原始存款
派生存款

按存款币种划分

人民币存款
外币存款

存款账户的种类

基本存款账户

它是存款人办理日常转账结算和现金收付的账户。

一般存款账户

它是存款人在基本存款账户以外的银行借款转存、与基本存款账户不在同一地点的、附属非独立核算单位开立的账户。

临时存款账户

它是存款人因临时经营活动需要开立的账户。

专用存款账户

它是存款人因为特定用途需要开立的账户。

单位活期存款业务的核算

现金存入

支票户现金存入核算

分录：

借:现金贷:吸收存款——活期存款——XX单位户

存折户现金存入核算

分录：

借:现金贷:吸收存款——活期存款——XX单位户

现金支取

支票户现金支取核算

分录：

借:吸收存款——活期存款——XX单位户贷:现金

存折户现金支取核算

分录：

借:吸收存款一一活期存款一一XX单位户贷:现金

活期储蓄存款业务的核算

开户

分录：

借:现金贷:吸收存款——活期储蓄存款——XX存款人户

销户

分录：

借:吸收存款——活期储蓄存款——XX存款人户利息支出——活期储蓄存款利息支出户贷:现金

续存

分录：

借:现金贷:吸收存款——活期储蓄存款——XX存款人户

支取

分录：

借:吸收存款——活期储蓄存款——XX存款人户贷:现金

设计模式学习——行为型

Wed, 22 Jul 2020 16:50:17 +0800

模板方法模式

介绍

定义:定义了一个算法的骨架,并允许子类为一个或多个步骤提供实现

模板方法使得子类可以在不改变算法结构的情况下,重新定义算法的某些步骤

类型：行为型

适用场景

一次性实现一个算法的不变的部分,并将可变的行为留给子类来实现
各子类中公共的行为被提取出来并集中到一个公共父类中, 从而避免代码重复

优点

提高复用性
提高扩展性
符合开闭原则

缺点

类数目增加
增加了系统实现的复杂度
继承关系自身缺点,如果父类添加新的抽象方法,所有子类都要改一遍

分类和扩展

钩子方法
- 提供了默认的行为，子类可以在必要时进行扩展

相关设计模式

模板方法模式和工厂方法模式
- 工厂方法是模板方法的一种特殊实现
模板方法模式和策略模式
- 策略模式在于实现算法可以互相替换而不影响应用层，不改变算法流程
- 模板方法定义了一个算法的骨架，不明确的交给子类实现，可以改变算法流程

代码样例

类图

抽象课程

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;
import org.apache.ibatis.mapping.MappedStatement;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.Collection;

/**
 * 抽象课程
 * <p>
 * {@link java.util.AbstractList} 提供抽象List实现
 * 例如 {@link java.util.AbstractList#addAll(int, Collection)} 定义通过模板
 * {@link java.util.AbstractMap} 同理
 * <p>
 * {@link javax.servlet.http.HttpServlet#doGet(HttpServletRequest, HttpServletResponse)} 也是模板定义
 * {@link org.apache.ibatis.executor.BaseExecutor#doUpdate(MappedStatement, Object)} 等方法交由子类实现
 * 例如 {@link org.apache.ibatis.executor.SimpleExecutor#doUpdate(MappedStatement, Object)}
 *
 * @author F嘉阳
 * @date 2020/3/4 10:59
 */
@Slf4j
public abstract class ACourse {

 /**
 * 固定不变的行为
 * 定义模板，不允许子类覆盖，防止影响流程执行顺序
 */
 protected final void makeCourse() {
 this.makePPT();
 this.makeVideo();
 // 调用可选方法
 if (needWriteArticle()) {
 this.writeArticle();
 }
 this.packageCourse();
 }

 /**
 * 固定不变的行为
 * 满足所有子类需求
 */
 final void makePPT() {
 log.info("制作PPT");
 }

 final void makeVideo() {
 log.info("制作视频");
 }

 /**
 * 可选行为
 * <p>
 * 手记可写可不写，但写的手记的流程还是一致的，故也声明为final
 * 通过定义钩子方法，允许子类覆盖
 */
 final void writeArticle() {
 log.info("编写手记");
 }

 /**
 * 钩子方法，提供默认实现
 *
 * @return
 */
 protected boolean needWriteArticle() {
 return false;
 }

 /**
 * 必须实现的行为
 * 课程内容打包
 */
 abstract void packageCourse();
}

具体课程实现

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;

/**
 * 具体课程实现
 *
 * @author F嘉阳
 * @date 2020/3/4 11:05
 */
@Slf4j
public class JavaCourse extends ACourse {
 /**
 * 必须实现的行为
 * 课程内容打包
 */
 @Override
 void packageCourse() {
 log.info("提供Java学习课程源码");
 }
}

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;

/**
 * 具体课程实现
 * @author F嘉阳
 * @date 2020/3/4 11:06
 */
@Slf4j
public class VueCourse extends ACourse {

 private boolean needWriteArticleFlag = false;

 /**
 * 通过构造器开放给实例决定
 * @param needWriteArticleFlag
 */
 public VueCourse(boolean needWriteArticleFlag) {
 this.needWriteArticleFlag = needWriteArticleFlag;
 }

 /**
 * 必须实现的行为
 * 课程内容打包
 */
 @Override
 void packageCourse() {
 log.info("提供Vue学习课程源码");
 log.info("提供Vue学习课程多媒体素材");
 }

 /**
 * 钩子方法，提供默认实现
 *
 * @return
 */
 @Override
 protected boolean needWriteArticle() {
 return this.needWriteArticleFlag;
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.templatemethod;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/4 11:08
 */
@Slf4j
class ACourseTest {

 @Test
 void makeJavaCourse() {
 ACourse javaCourse = new JavaCourse();
 javaCourse.makeCourse();
 }

 @Test
 void makeVueCourse() {
 ACourse vueCourse = new VueCourse(true);
 vueCourse.makeCourse();
 }
}

输出

11:27:56.942 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作PPT
11:27:56.957 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作视频
11:27:56.957 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.JavaCourse - 提供Java学习课程源码


11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作PPT
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作视频
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 编写手记
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.VueCourse - 提供Vue学习课程源码
11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.VueCourse - 提供Vue学习课程多媒体素材

迭代器模式

介绍

定义提供一种方法,顺序访问一个集合对象中的各个元素,而又不暴露该对象的内部表示

类型：行为型

适用场景

访问一个集合对象的内容而无需暴露它的内部表示
为遍历不同的集合结构提供一个统一的接口

优点

分离了集合对象的遍历行为

缺点

类的个数成对增加

相关设计模式

迭代器模式和访问者模式
- 访问者模式扩展开放的部分，再作用于对象的操作上
- 迭代器模式扩展开放的部分，再作用于对象的种类上

代码样例

迭代器一般使用现成的，而不会自己实现

类图

课程接口

package top.fjy8018.designpattern.pattern.behavior.iterator;

import lombok.AllArgsConstructor;
import lombok.Getter;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:49
 */
@AllArgsConstructor
@Getter
public class Course {

 private String name;

}

迭代器

package top.fjy8018.designpattern.pattern.behavior.iterator;

import java.util.ArrayList;

/**
 * JDK迭代器
 * {@link java.util.Iterator}
 * 集合内部类实现 {@link ArrayList.Itr#hasNext()}
 * 同时做了进一步扩展 {@link ArrayList.ListItr}
 *
 * mybatis中 {@link org.apache.ibatis.cursor.defaults.DefaultCursor}
 * 其持有游标迭代器对象 {@link org.apache.ibatis.cursor.defaults.DefaultCursor.CursorIterator}
 *
 * @author F嘉阳
 * @date 2020/3/4 14:49
 */
public interface CourseIterator {

 /**
 * 获取下一个课程
 *
 * @return
 */
 Course nextCourse();

 /**
 * 是否是最后一个课程
 * @return
 */
 boolean isLastCourse();
}

迭代器实现

package top.fjy8018.designpattern.pattern.behavior.iterator;

import lombok.extern.slf4j.Slf4j;

import java.util.List;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:50
 */
@Slf4j
public class CourseIteratorImpl implements CourseIterator {

 private List courseList;

 private int position;

 private Course course;

 public CourseIteratorImpl(List courseList){
 this.courseList=courseList;
 }

 @Override
 public Course nextCourse() {
 log.info("返回课程,位置是: {}",position);
 course=(Course)courseList.get(position);
 position++;
 return course;
 }

 @Override
 public boolean isLastCourse(){
 if(position< courseList.size()){
 return false;
 }
 return true;
 }
}

对课程进行处理

package top.fjy8018.designpattern.pattern.behavior.iterator;

/**
 * 对课程进行处理
 *
 * @author F嘉阳
 * @date 2020/3/4 14:48
 */
public interface CourseAggregate {

 void addCourse(Course course);

 void removeCourse(Course course);

 /**
 * 获取课程迭代器
 * @return
 */
 CourseIterator getCourseIterator();
}

实现

package top.fjy8018.designpattern.pattern.behavior.iterator;

import java.util.ArrayList;
import java.util.List;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:50
 */
public class CourseAggregateImpl implements CourseAggregate {

 private List courseList;

 public CourseAggregateImpl() {
 this.courseList = new ArrayList();
 }

 @Override
 public void addCourse(Course course) {
 courseList.add(course);
 }

 @Override
 public void removeCourse(Course course) {
 courseList.remove(course);
 }

 @Override
 public CourseIterator getCourseIterator() {
 return new CourseIteratorImpl(courseList);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.iterator;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/4 14:52
 */
@Slf4j
class CourseIteratorTest {

 @Test
 void nextCourse() {
 Course course1 = new Course("Java电商一期");
 Course course2 = new Course("Java电商二期");
 Course course3 = new Course("Java设计模式精讲");
 Course course4 = new Course("Python课程");
 Course course5 = new Course("算法课程");
 Course course6 = new Course("前端课程");


 CourseAggregate courseAggregate = new CourseAggregateImpl();

 courseAggregate.addCourse(course1);
 courseAggregate.addCourse(course2);
 courseAggregate.addCourse(course3);
 courseAggregate.addCourse(course4);
 courseAggregate.addCourse(course5);
 courseAggregate.addCourse(course6);

 log.info("-----课程列表-----");
 printCourses(courseAggregate);

 courseAggregate.removeCourse(course4);
 courseAggregate.removeCourse(course5);

 log.info("-----删除操作之后的课程列表-----");
 printCourses(courseAggregate);
 }

 public static void printCourses(CourseAggregate courseAggregate){
 CourseIterator courseIterator= courseAggregate.getCourseIterator();
 while(!courseIterator.isLastCourse()){
 Course course=courseIterator.nextCourse();
 log.info(course.getName());
 }
 }
}

输出

15:18:23.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - -----课程列表-----
15:18:23.124 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 0
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java分布式
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 1
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java系统设计
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 2
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java设计模式精讲
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 3
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Python课程
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 4
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 算法课程
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 5
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 前端课程
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - -----删除操作之后的课程列表-----
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 0
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java分布式
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 1
15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java系统设计
15:18:23.127 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 2
15:18:23.127 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java设计模式精讲
15:18:23.132 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 3
15:18:23.132 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 前端课程

策略模式

介绍

定义：定义了算法家族,分别封装起来,让它们之间可以互相替换, 此模式让算法的变化不会影响到使用算法的用户。

例如有大量if…else的情况

类型：行为型

适用场景

系统有很多类，而他们的区别仅仅在于他们的行为不同
一个系统需要动态地在几种算法中选择一种

优点

开闭原则
避免使用多重条件转移语句
提高算法的保密性和安全性

缺点

客户端必须知道所有的策略类,并自行决定使用哪一个策略类。
产生很多策略类

相关设计模式

策略模式和工厂模式
策略模式和状态模式

代码样例

改造前——初级策略模式

无法避免if…else

类图

促销接口

package top.fjy8018.designpattern.pattern.behavior.strategy;

/**
 * 促销接口
 *
 * @author F嘉阳
 * @date 2020/3/4 15:37
 */
public interface PromotionStrategy {

 void doPromotion();
}

满减促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 满减促销策略
 *
 * @author F嘉阳
 * @date 2020/3/4 15:38
 */
@Slf4j
public class ManJianPromotionStrategy implements PromotionStrategy {
 @Override
 public void doPromotion() {
 log.info("满减促销，满200-20");
 }
}

立减促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 立减促销策略
 *
 * @author F嘉阳
 * @date 2020/3/4 15:39
 */
@Slf4j
public class LiJianPromotionStrategy implements PromotionStrategy{
 @Override
 public void doPromotion() {
 log.info("立减促销，课程价格直接减去配置的价格");
 }
}

返现促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 返现促销策略
 *
 * @author F嘉阳
 * @date 2020/3/4 15:38
 */
@Slf4j
public class FanXianPromotionStrategy implements PromotionStrategy{
 @Override
 public void doPromotion() {
 log.info("返现促销，返回金额存放到用户余额中");
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/4 15:46
 */
@Slf4j
class PromotionStrategyTest {

 @Test
 void existIfElse() {
 String promotionKey = "LIJIAN";
 PromotionActivity activity = null;
 if (StringUtils.equals(promotionKey,"LIJIAN")){
 // 方法和类本身线程安全
 activity= new PromotionActivity(new LiJianPromotionStrategy());
 }else if(StringUtils.equals(promotionKey,"MANJIAN")){
 activity = new PromotionActivity(new FanXianPromotionStrategy());
 }

 activity.executePromotionStrategy();
 }
}

输出

16:08:28.582 [main] INFO top.fjy8018.designpattern.pattern.behavior.strategy.LiJianPromotionStrategy - 立减促销，课程价格直接减去配置的价格

改造后——结合工厂和策略模式

类图

空促销策略

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;

/**
 * 空策略
 *
 * @author F嘉阳
 * @date 2020/3/4 16:16
 */
@Slf4j
public class EmptyPromotionStrategy implements PromotionStrategy {
 @Override
 public void doPromotion() {
 log.info("空促销策略");
 }
}

促销策略工厂

package top.fjy8018.designpattern.pattern.behavior.strategy;

import org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory;
import org.springframework.beans.factory.support.CglibSubclassingInstantiationStrategy;
import org.springframework.beans.factory.support.SimpleInstantiationStrategy;
import org.springframework.core.io.FileSystemResource;
import org.springframework.core.io.PathResource;

import java.util.Arrays;
import java.util.Comparator;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

/**
 * 促销策略工厂
 * <p>
 * {@link java.util.Comparator} 比较器，类似于抽象策略，有大量实现类，所有具体策略都要实现定义好的接口方法
 * 例如 {@link com.google.common.collect.ExplicitOrdering}
 * 应用层可以逐级传递比较器 {@link Arrays#sort(Object[], Comparator)}
 * {@link java.util.TreeMap#compare(Object, Object)} 利用内部的比较器处理
 * <p>
 * Spring资源策略抽象 {@link org.springframework.core.io.Resource}
 * 提供大量策略实现 {@link PathResource} {@link FileSystemResource}
 * <p>
 * {@link org.springframework.beans.factory.support.InstantiationStrategy} 初始化策略接口
 * 初始化具体策略实现 {@link SimpleInstantiationStrategy} {@link CglibSubclassingInstantiationStrategy}
 * 初始化策略在 {@link AbstractAutowireCapableBeanFactory} 中被使用
 *
 * @author F嘉阳
 * @date 2020/3/4 16:09
 */
public class PromotionStrategyFactory {

 private static Map<String, PromotionStrategy> STRATEGY_MAP = new ConcurrentHashMap<>();

 /**
 * 无促销
 */
 private static final PromotionStrategy NON_PROMOTION = new EmptyPromotionStrategy();

 static {
 STRATEGY_MAP.put(PromotionKey.FANXIAN, new FanXianPromotionStrategy());
 STRATEGY_MAP.put(PromotionKey.MANJIAN, new ManJianPromotionStrategy());
 STRATEGY_MAP.put(PromotionKey.LIJIAN, new LiJianPromotionStrategy());
 }

 /**
 * 禁止实例化
 */
 private PromotionStrategyFactory() {
 }

 /**
 * 开放获取策略的方法
 *
 * @param promotionKey
 * @return
 */
 public static PromotionStrategy getPromotionStrategy(String promotionKey) {
 PromotionStrategy strategy = STRATEGY_MAP.get(promotionKey);
 return strategy == null ? NON_PROMOTION : strategy;
 }

 /**
 * 常量定义
 */
 private interface PromotionKey {
 String LIJIAN = "LIJIAN";
 String FANXIAN = "FANXIAN";
 String MANJIAN = "MANJIAN";
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.strategy;

import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/3/4 15:46
 */
@Slf4j
class PromotionStrategyTest {

 @Test
 void nonIfElse() {
 String promotionKey = "LIJIAN";
 PromotionActivity activity = new PromotionActivity(PromotionStrategyFactory.getPromotionStrategy(promotionKey));
 activity.executePromotionStrategy();
 }
}

输出

16:08:28.582 [main] INFO top.fjy8018.designpattern.pattern.behavior.strategy.LiJianPromotionStrategy - 立减促销，课程价格直接减去配置的价格

解释器模式

介绍

定义：给定一个语言，定义它的文法的一种表示，并定义一个解释器，这个解释器使用该表示来解释语言中的句子。
为了解释一种语言，而为语言创建的解释器
类型：行为型

适用场景（较少）

某个特定类型问题发生频率足够高

优点

语法由很多类表示,容易改变及扩展此语言

缺点

当语法规则数目太多时,增加了系统复杂度

相关设计模式

解释器模式和适配器模式

代码样例

类图

解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * 解释器
 * <p>
 * JDK 源码
 * {@link java.util.regex.Pattern} 正则解释器
 * {@link org.springframework.expression.spel.standard.SpelExpressionParser} Spring解释器
 *
 * @author F嘉阳
 * @date 2020/3/5 14:45
 */
public interface Interpreter {
 int interpret();
}

加法解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * 加法解释器
 *
 * @author F嘉阳
 * @date 2020/3/5 14:46
 */
public class AddInterpreter implements Interpreter {

 /**
 * 定义两个表达式
 */
 private Interpreter firstExpression, secondExpression;

 public AddInterpreter(Interpreter firstExpression, Interpreter secondExpression) {
 this.firstExpression = firstExpression;
 this.secondExpression = secondExpression;
 }

 @Override
 public int interpret() {
 // 此时传入为NumberInterpreter类型，调interpret方法转int类型
 return this.firstExpression.interpret() + this.secondExpression.interpret();
 }

 @Override
 public String toString() {
 return "+";
 }
}

乘法解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * 乘法解释器
 *
 * @author F嘉阳
 * @date 2020/3/5 14:47
 */
public class MultiInterpreter implements Interpreter {

 private Interpreter firstExpression, secondExpression;

 public MultiInterpreter(Interpreter firstExpression, Interpreter secondExpression) {
 this.firstExpression = firstExpression;
 this.secondExpression = secondExpression;
 }

 @Override
 public int interpret() {
 return this.firstExpression.interpret() * this.secondExpression.interpret();
 }

 @Override
 public String toString() {
 return "*";
 }
}

类型解释器

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:47
 */
public class NumberInterpreter implements Interpreter {
 private int number;

 public NumberInterpreter(int number) {
 this.number = number;
 }

 public NumberInterpreter(String number) {
 this.number = Integer.parseInt(number);
 }

 @Override
 public int interpret() {
 return this.number;
 }
}

操作工具类

package top.fjy8018.designpattern.pattern.behavior.interpreter;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:47
 */
public class OperatorUtil {
 public static boolean isOperator(String symbol) {
 return (symbol.equals("+") || symbol.equals("*"));

 }

 public static Interpreter getExpressionObject(Interpreter firstExpression, Interpreter secondExpression, String symbol) {
 if (symbol.equals("+")) {
 return new AddInterpreter(firstExpression, secondExpression);
 } else if (symbol.equals("*")) {
 return new MultiInterpreter(firstExpression, secondExpression);
 }
 return null;
 }
}

自定义解释器实现

package top.fjy8018.designpattern.pattern.behavior.interpreter;

import lombok.extern.slf4j.Slf4j;

import java.util.Stack;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:48
 */
@Slf4j
public class CustomExpressionParser {
 private Stack<Interpreter> stack = new Stack<Interpreter>();

 public int parse(String str) {
 String[] strItemArray = str.split(" ");
 for (String symbol : strItemArray) {
 if (!OperatorUtil.isOperator(symbol)) {
 Interpreter numberExpression = new NumberInterpreter(symbol);
 stack.push(numberExpression);
 log.info(String.format("入栈: %d", numberExpression.interpret()));
 } else {
 //是运算符号，可以计算
 Interpreter firstExpression = stack.pop();
 Interpreter secondExpression = stack.pop();
 log.info(String.format("出栈: %d 和 %d",
 firstExpression.interpret(), secondExpression.interpret()));
 Interpreter operator = OperatorUtil.getExpressionObject(firstExpression, secondExpression, symbol);
 log.info(String.format("应用运算符: %s", operator));
 int result = operator.interpret();
 NumberInterpreter resultExpression = new NumberInterpreter(result);
 stack.push(resultExpression);
 log.info(String.format("阶段结果入栈: %d", resultExpression.interpret()));
 }
 }
 int result = stack.pop().interpret();
 return result;

 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.interpreter;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import org.springframework.expression.Expression;
import org.springframework.expression.spel.standard.SpelExpressionParser;

/**
 * @author F嘉阳
 * @date 2020/3/5 14:48
 */
@Slf4j
class InterpreterTest {

 @Test
 void interpret() {
 String myInterpreter = "6 100 11 + *";
 CustomExpressionParser expressionParser = new CustomExpressionParser();
 int result = expressionParser.parse(myInterpreter);
 log.info("解释器计算结果: " + result);
 }

 @Test
 void springTest() {
 org.springframework.expression.ExpressionParser parser = new SpelExpressionParser();
 Expression expression = parser.parseExpression("100 * 2 + 400 * 1 + 66");
 int result = (Integer) expression.getValue();
 log.info(String.valueOf(result));
 }
}

输出

14:52:28.086 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 6
14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 100
14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 11
14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 出栈: 11 和 100
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 应用运算符: +
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 阶段结果入栈: 111
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 出栈: 111 和 6
14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 应用运算符: *
14:52:28.091 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 阶段结果入栈: 666
14:52:28.091 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.InterpreterTest - 解释器计算结果: 666


14:52:28.550 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.InterpreterTest - 666

观察者模式

介绍

定义：定义了对象之间的一对多依赖，让多个观察者对象同时监听某一个主题对象，当主题对象发生变化时，它的所有依赖者（观察者）都会收到通知并更新

类型：行为型

适用场景

关联行为场景，建立一套触发机制

优点

观察者模式支持广播通信
观察者和被观察者之间建立一个抽象的耦合

缺点

使用要得当,要避免循环调用
观察者之间有过多的细节依赖、提高时间消耗及程序复杂度

代码样例——JDK观察者

类图

课程问题

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.AllArgsConstructor;
import lombok.Getter;

/**
 * 课程问题
 *
 * @author F嘉阳
 * @date 2020/3/5 15:23
 */
@Getter
@AllArgsConstructor
public class Question {

 private String username;

 private String content;

}

课程（被观察者）

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;

import java.util.EventListener;
import java.util.Observable;
import java.util.Vector;

/**
 * 课程
 * 被观察者类
 *
 * {@link Observable} 使用 {@link Vector} 保证线程安全
 *
 * JDK源码
 * {@link java.awt.Event} 事件监听器
 * {@link javax.servlet.ServletRequestListener} HTTP请求事件监听器
 * 继承 {@link EventListener}，有大量实现，例如{@link org.springframework.beans.factory.parsing.ReaderEventListener}
 *
 * @author F嘉阳
 * @date 2020/3/5 15:23
 */
@Slf4j
@Getter
@AllArgsConstructor
public class Course extends Observable {

 private String courseName;

 /**
 * 产生问题
 * @param course
 * @param question
 */
 public void produceQuestion(Course course,Question question){
 log.info("{}在{}提交了一个问题",question.getUsername(),course.getCourseName());
 // 状态变化
 setChanged();
 // 通知观察者
 notifyObservers(question);
 }
}

老师实体（观察者）

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;

import java.util.Observable;
import java.util.Observer;

/**
 * 老师实体
 *
 * 老师观察课程是否产生了新的问题
 * @author F嘉阳
 * @date 2020/3/5 15:24
 */
@Slf4j
@Getter
@AllArgsConstructor
public class Teacher implements Observer {

 private String teacherName;

 /**
 * This method is called whenever the observed object is changed. An
 * application calls an <tt>Observable</tt> object's
 * <code>notifyObservers</code> method to have all the object's
 * observers notified of the change.
 *
 * @param o the observable object.
 * @param arg an argument passed to the <code>notifyObservers</code>
 */
 @Override
 public void update(Observable o, Object arg) {
 Course course = (Course) o;
 Question question = (Question) arg;
 log.info("{}老师的{}课程接收到一个{}提交的问题，内容为：{}",
 teacherName,course.getCourseName(),question.getUsername(),question.getContent());
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.observer;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/3/5 15:36
 */
@Slf4j
class TeacherTest {

 @Test
 void update() {
 Course course = new Course("Java设计模式");
 Teacher teacher = new Teacher("fjy8018");

 course.addObserver(teacher);

 // 业务逻辑
 Question question = new Question("fjy","问题内容XXX");

 course.produceQuestion(course,question);
 }
}

输出

15:38:34.437 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.Course - fjy在Java设计模式提交了一个问题
15:38:34.444 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.Teacher - fjy8018老师的Java设计模式课程接收到一个fjy提交的问题，内容为：问题内容XXX

代码样例——Guava监听器

package top.fjy8018.designpattern.pattern.behavior.observer.guava;

import com.google.common.eventbus.Subscribe;
import lombok.extern.slf4j.Slf4j;

import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CopyOnWriteArraySet;

/**
 * Guava事件监听器
 * {@link EventBus} 使用内部对象 {@link SubscriberRegistry} 内部使用 {@link ConcurrentMap} 存储订阅者
 * 同时使用 {@link CopyOnWriteArraySet} 保证线程安全和订阅者唯一
 * 故{@link com.google.common.eventbus.Subscriber}
 * 重写了 {@link com.google.common.eventbus.Subscriber#equals(Object)}
 * 和 {@link com.google.common.eventbus.Subscriber#hashCode()} 方法
 *
 * @author F嘉阳
 * @date 2020/3/5 15:56
 */
@Slf4j
public class GuavaEvent {

 @Subscribe
 public void subscribe(String str) {
 // 业务逻辑
 log.info("执行订阅方法，传入参数是：{}", str);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.observer;

import com.google.common.eventbus.EventBus;
import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;
import top.fjy8018.designpattern.pattern.behavior.observer.guava.Course;
import top.fjy8018.designpattern.pattern.behavior.observer.guava.GuavaEvent;

/**
 * @author F嘉阳
 * @date 2020/3/5 15:58
 */
@Slf4j
class GuavaEventTest {

 @Test
 void subscribe() {
 EventBus eventBus = new EventBus();
 GuavaEvent event = new GuavaEvent();
 // 注册
 eventBus.register(event);
 eventBus.post("提交事件的内容");
 }
}

输出

16:24:27.716 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.GuavaEvent - 执行订阅方法，传入参数是：提交事件的内容

备忘录模式

介绍

定义：保存一个对象的某个状态，以便在适当的时候恢复对象。（类似历史记录存储）

“后悔药”

类型：行为型

适用场景

保存及恢复数据相关业务场景
后悔的时候，即想恢复到之前的状态

优点

为用户提供一种可恢复机制
存档信息的封装

缺点

资源占用

相关设计模式

备忘录模式和状态模式

代码样例

类图

文章

package top.fjy8018.designpattern.pattern.behavior.memento;

import lombok.AllArgsConstructor;
import lombok.Data;

/**
 * 文章，字段不宜过多，否则导致过大内存占用
 *
 * @author F嘉阳
 * @date 2020/3/6 17:00
 */
@Data
@AllArgsConstructor
public class Article {

 private String title;

 private String content;

 private String imgs;

 /**
 * 保存
 * @return
 */
 public ArticleMemento saveToMemento() {
 ArticleMemento articleMemento = new ArticleMemento(this.title,this.content,this.imgs);
 return articleMemento;
 }

 /**
 * 撤销
 * @param articleMemento
 */
 public void undoFromMemento(ArticleMemento articleMemento) {
 this.title = articleMemento.getTitle();
 this.content = articleMemento.getContent();
 this.imgs = articleMemento.getImgs();
 }
}

文章备忘录（快照）

package top.fjy8018.designpattern.pattern.behavior.memento;

import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.ToString;
import org.springframework.binding.message.StateManageableMessageContext;

/**
 * 文章备忘录（快照）
 * 快照不需要set方法，只能通过构造器注入
 *
 * Spring工作流中源码 {@link StateManageableMessageContext#createMessagesMemento()}
 *
 * @author F嘉阳
 * @date 2020/3/6 17:03
 */
@Getter
@ToString
@AllArgsConstructor
public class ArticleMemento {

 private String title;

 private String content;

 private String imgs;
}

备忘录核心，栈实现管理者

package top.fjy8018.designpattern.pattern.behavior.memento;

import java.util.Stack;

/**
 * 保存一个对象的某个状态，以便在适当的时候恢复对象
 *
 * @author F嘉阳
 * @date 2020/3/6 17:04
 */
public class ArticleMementoManager {

 /**
 * 使用栈特性实现动作撤销
 */
 private final Stack<ArticleMemento> ARTICLE_MEMENTO_STACK = new Stack<ArticleMemento>();

 public ArticleMemento getMemento() {
 ArticleMemento articleMemento= ARTICLE_MEMENTO_STACK.pop();
 return articleMemento;
 }

 public void addMemento(ArticleMemento articleMemento)
 {
 ARTICLE_MEMENTO_STACK.push(articleMemento);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.memento;

import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/3/6 17:07
 */
@Slf4j
class ArticleMementoTest {

 @Test
 void getTitle() {
 ArticleMementoManager articleMementoManager = new ArticleMementoManager();

 Article article= new Article("如影随行的设计模式A","手记内容A","手记图片A");
 // 存档
 ArticleMemento articleMemento = article.saveToMemento();

 articleMementoManager.addMemento(articleMemento);
 log.info("标题:"+article.getTitle()+" 内容:"+article.getContent()+" 图片:"+article.getImgs()+" 暂存成功");

 log.info("手记完整信息:"+article);
 
 log.info("修改手记start");

 article.setTitle("如影随行的设计模式B");
 article.setContent("手记内容B");
 article.setImgs("手记图片B");

 log.info("修改手记end");

 log.info("手记完整信息:"+article);

 articleMemento = article.saveToMemento();
 articleMementoManager.addMemento(articleMemento);
 
 article.setTitle("如影随行的设计模式C");
 article.setContent("手记内容C");
 article.setImgs("手记图片C");

 log.info("暂存回退start");
 // 取档
 log.info("回退出栈1次");
 articleMemento = articleMementoManager.getMemento();
 article.undoFromMemento(articleMemento);

 log.info("回退出栈2次");
 articleMemento = articleMementoManager.getMemento();
 article.undoFromMemento(articleMemento);

 log.info("暂存回退end");
 log.info("手记完整信息:"+article);
 }
}

输出

17:33:51.154 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 标题:如影随行的设计模式A 内容:手记内容A 图片:手记图片A 暂存成功
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式A, content=手记内容A, imgs=手记图片A)
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 修改手记start
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 修改手记end
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式B, content=手记内容B, imgs=手记图片B)
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 暂存回退start
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 回退出栈1次
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 回退出栈2次
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 暂存回退end
17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式A, content=手记内容A, imgs=手记图片A)

命令模式

介绍

定义：将请求封装成对象,以便使用不同的请求

特点：命令模式解决了应用程序中对象的职责以及它们之间的通信方式

类型：行为型

适用场景

请求调用者和请求接收者需要解耦,使得调用者和接收者不直接交互
需要抽象出等待执行的行为

优点

降低耦合
容易扩展新命令或者一组命令

缺点

命令的无限扩展会增加类的数量，提高系统实现复杂度

相关设计模式

命令模式和备忘录模式

代码样例

类图

命令抽象

package top.fjy8018.designpattern.pattern.behavior.command;

/**
 * 命令抽象
 * <p>
 * Java相关源码 {@link Runnable} 可理解为命令抽象
 * 其实现类为具体的命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
public interface Command {
 void execute();
}

操作视频

package top.fjy8018.designpattern.pattern.behavior.command;

import lombok.AllArgsConstructor;

/**
 * 关闭视频命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
@AllArgsConstructor
public class CloseCourseVideoCommand implements Command {
 private final CourseVideo courseVideo;

 @Override
 public void execute() {
 courseVideo.close();
 }
}

打开视频命令

package top.fjy8018.designpattern.pattern.behavior.command;

import lombok.AllArgsConstructor;

/**
 * 打开视频命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
@AllArgsConstructor
public class OpenCourseVideoCommand implements Command {
 private final CourseVideo courseVideo;

 @Override
 public void execute() {
 courseVideo.open();
 }
}

关闭视频命令

package top.fjy8018.designpattern.pattern.behavior.command;

import lombok.AllArgsConstructor;

/**
 * 关闭视频命令
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
@AllArgsConstructor
public class CloseCourseVideoCommand implements Command {
 private final CourseVideo courseVideo;

 @Override
 public void execute() {
 courseVideo.close();
 }
}

命令执行人

package top.fjy8018.designpattern.pattern.behavior.command;

import java.util.ArrayList;
import java.util.List;

/**
 * 命令执行人
 *
 * @author F嘉阳
 * @date 2020/7/21 15:58
 */
public class Staff {
 /**
 * 保证执行顺序
 */
 private final List<Command> commandList = new ArrayList<Command>();

 public void addCommand(Command command) {
 commandList.add(command);
 }

 public void executeCommands() {
 for (Command command : commandList) {
 command.execute();
 }
 commandList.clear();
 }

}

测试

package top.fjy8018.designpattern.pattern.behavior.command;

import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/7/21 16:02
 */
class CommandTest {

 @Test
 void doTest() {
 CourseVideo courseVideo = new CourseVideo("Java设计模式精讲");
 OpenCourseVideoCommand openCourseVideoCommand = new OpenCourseVideoCommand(courseVideo);
 CloseCourseVideoCommand closeCourseVideoCommand = new CloseCourseVideoCommand(courseVideo);

 Staff staff = new Staff();

 staff.addCommand(openCourseVideoCommand);
 staff.addCommand(closeCourseVideoCommand);

 staff.executeCommands();
 }
}

输出

Java设计模式精讲课程视频开放
Java设计模式精讲课程视频关闭

中介者模式

介绍

定义：定义一个封装一组对象如何交互的对象

特点：通过使对象明确地相互引用来促进松散耦合,并允许独立地改变它们的交互

适用场景

系统中对象之间存在复杂的引用关系，产生的相互依赖关系结构混乱且难以理解
交互的公共行为，如果需要改变行为则可以增加新的中介者类

优点

将一对多转化成了一对一、降低程序复杂度
类之间解耦

缺点

中介者过多,导致系统复杂

相关设计模式

中介者模式和观察者模式

代码样例

类图

学习群组（中介者）

package top.fjy8018.designpattern.pattern.behavior.mediator;

import java.util.Date;
import java.util.Timer;

/**
 * 学习群组（中介者）
 * <p>
 * jdk源码 {@link Timer#schedule(java.util.TimerTask, Date)} 多个方法重载
 * 都是调用{@link Timer#sched(java.util.TimerTask, long, long)} 实现的，由底层方法进行整体协调
 *
 * @author F嘉阳
 * @date 2020/7/21 16:20
 */
public class StudyGroup {

 public static void showMessage(User user, String message) {
 System.out.println(new Date().toString() + " [" + user.getName() + "] : " + message);
 }
}

用户

package top.fjy8018.designpattern.pattern.behavior.mediator;

import lombok.AllArgsConstructor;
import lombok.Data;

/**
 * @author F嘉阳
 * @date 2020/7/21 16:20
 */
@Data
@AllArgsConstructor
public class User {
 private String name;

 public void sendMessage(String message) {
 // 只和中介者交互
 StudyGroup.showMessage(this, message);
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.mediator;

import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/7/21 16:25
 */
class MediatorTest {

 @Test
 void doTest() {
 User fjy = new User("F嘉阳");
 User tom = new User("Tom");

 fjy.sendMessage(" Hey! Tom! Let's learn Design Pattern");
 tom.sendMessage("OK! F嘉阳");
 }
}

输出

Tue Jul 21 16:47:03 CST 2020 [F嘉阳] : Hey! Tom! Let's learn Design Pattern
Tue Jul 21 16:47:03 CST 2020 [Tom] : OK! F嘉阳

责任链模式

介绍

定义：为请求创建一个接收此次请求对象的链

类型：行为型

适用场景

一个请求的处理需要多个对象当中的一个或几个协作处理

优点

请求的发送者和接收者(请求的处理)解耦
责任链可以动态组合

缺点

责任链太长或者处理时间过长,影响性能
责任链有可能过多

相关设计模式

责任链模式和状态模式

代码样例

类图

模拟二级审批场景

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import javax.servlet.FilterChain;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;

/**
 * 模拟二级审批场景
 * <p>
 * 审批抽象
 * <p>
 * java 过滤器实现 {@link javax.servlet.Filter#doFilter(ServletRequest, ServletResponse, FilterChain)}
 * 实现 {@link FilterChain#doFilter(ServletRequest, ServletResponse)} 接口的组成一条过滤器链
 * 所有子类通过调用 chain.doFilter(request, response) 实现链式传递
 * <p>
 * SpringSecurity过滤器链实现也类似
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
public abstract class Approver {
 /**
 * 责任链核心：包含与自身相同的类型对象
 */
 protected Approver approver;

 /**
 * 设置下一个节点，组成链条
 *
 * @param approver
 */
 public void setNextApprover(Approver approver) {
 this.approver = approver;
 }

 /**
 * 交由子类实现
 *
 * @param course
 */
 public abstract void deploy(Course course);
}

手记审批

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import org.apache.commons.lang3.StringUtils;

/**
 * 手记审批
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
public class ArticleApprover extends Approver {
 @Override
 public void deploy(Course course) {
 if (StringUtils.isNotEmpty(course.getArticle())) {
 System.out.println(course.getName() + "含有手记,批准");
 // 是否为末尾
 if (approver != null) {
 // 执行下一个处理链
 approver.deploy(course);
 }
 } else {
 System.out.println(course.getName() + "不含有手记,不批准,流程结束");
 return;
 }
 }
}

视频审批

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import org.apache.commons.lang3.StringUtils;

/**
 * 视频审批
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
public class VideoApprover extends Approver {
 @Override
 public void deploy(Course course) {
 if (StringUtils.isNotEmpty(course.getVideo())) {
 System.out.println(course.getName() + "含有视频,批准");
 if (approver != null) {
 approver.deploy(course);
 }
 } else {
 System.out.println(course.getName() + "不含有视频,不批准,流程结束");
 return;
 }
 }
}

课程

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import lombok.Data;

/**
 * 课程
 *
 * @author F嘉阳
 * @date 2020/7/21 16:59
 */
@Data
public class Course {
 private String name;

 private String article;

 private String video;
}

测试

package top.fjy8018.designpattern.pattern.behavior.chainofresponsibility;

import org.junit.jupiter.api.Test;

/**
 * @author F嘉阳
 * @date 2020/7/21 15:06
 */
class ChainofResponsibilityTest {

 @Test
 void doTest() {
 Approver articleApprover = new ArticleApprover();
 Approver videoApprover = new VideoApprover();

 Course course = new Course();
 course.setName("Java设计模式");
 course.setArticle("Java设计模式的手记");
 course.setVideo("Java设计模式的视频");

 articleApprover.setNextApprover(videoApprover);

 articleApprover.deploy(course);
 }
}

输出

Java设计模式含有手记,批准
Java设计模式含有视频,批准

访问者模式

介绍

定义：封装作用于某数据结构(如List/set/Map等)中的各元素的操作

特性：可以在不改变各元素的类的前提下定义作用于这些元素的操作

类型：行为型

适用场景

一个数据结构如(List/set/Map等)包含很多类型对象
数据结构与数据操作分离

优点

增加新的操作很容易即增加一个新的访问者

缺点

增加新的数据结构困难
具体元素变更比较麻烦

相关设计模式

访问者模式和迭代器模式

代码样例

类图

课程抽象

package top.fjy8018.designpattern.pattern.behavior.visitor;

import lombok.Data;

/**
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
@Data
public abstract class Course {
 private String name;

 public abstract void accept(IVisitor visitor);

}

实战课

package top.fjy8018.designpattern.pattern.behavior.visitor;

import lombok.Data;
import lombok.EqualsAndHashCode;

/**
 * 实战课，包含价格
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
@Data
@EqualsAndHashCode(callSuper = true)
public class CodingCourse extends Course {
 private int price;

 @Override
 public void accept(IVisitor visitor) {
 visitor.visit(this);
 }
}

免费课

package top.fjy8018.designpattern.pattern.behavior.visitor;

/**
 * 免费课
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
public class FreeCourse extends Course {

 /**
 * 接受策略
 *
 * @param visitor
 */
 @Override
 public void accept(IVisitor visitor) {
 visitor.visit(this);
 }
}

访问者

package top.fjy8018.designpattern.pattern.behavior.visitor;

/**
 * 封装操作数据结构的方法
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
public interface IVisitor {

 void visit(FreeCourse freeCourse);

 void visit(CodingCourse codingCourse);
}

访问者核心实现

package top.fjy8018.designpattern.pattern.behavior.visitor;

/**
 * 访问者
 * 核心：对不同类型产生不同的操作行为
 * <p>
 * jdk源码 {@link java.nio.file.FileVisitor} 实现对文件的访问
 * spring源码 {@link org.springframework.beans.factory.config.BeanDefinitionVisitor} 遍历Bean具体属性并填充
 *
 * @author F嘉阳
 * @date 2020/7/22 10:20
 */
public class Visitor implements IVisitor {

 /**
 * 访问免费课程，打印所有免费课程名称
 *
 * @param freeCourse
 */
 @Override
 public void visit(FreeCourse freeCourse) {
 System.out.println("免费课程:" + freeCourse.getName());
 }

 /**
 * 访问实战课程，打印所有实战课程名称及价格
 *
 * @param codingCourse
 */
 @Override
 public void visit(CodingCourse codingCourse) {
 System.out.println("实战课程:" + codingCourse.getName() + " 价格:" + codingCourse.getPrice() + "元");
 }

}

测试

package top.fjy8018.designpattern.pattern.behavior.visitor;

import org.junit.jupiter.api.Test;

import java.util.ArrayList;
import java.util.List;

/**
 * @author F嘉阳
 * @date 2020/7/22 16:19
 */
class VisitorTest {

 @Test
 void visit() {
 List<Course> courseList = new ArrayList<>();

 FreeCourse freeCourse = new FreeCourse();
 freeCourse.setName("SpringMVC数据绑定");

 CodingCourse codingCourse = new CodingCourse();
 codingCourse.setName("Java设计模式精讲");
 codingCourse.setPrice(299);
 // 数组组装
 courseList.add(freeCourse);
 courseList.add(codingCourse);

 for (Course course : courseList) {
 course.accept(new Visitor());
 }
 }
}

输出

免费课程:SpringMVC数据绑定
实战课程:Java设计模式精讲 价格:299元

状态模式

介绍

定义：允许一个对象在其内部状态改变时改变它的行为

类型：行为型

适用场景

一个对象存在多个状态(不同状态下行为不同),且状态可相互转换

优点

将不同的状态隔离
把各种状态的转换逻辑,分布到 State的子类中,减少相互间依赖
增加新的状态非常简单

缺点

状态多的业务场景导致类数目增加,系统变复杂

相关设计模式

状态模式和享元模式

代码样例

类图

视频播放状态转换场景

package top.fjy8018.designpattern.pattern.behavior.state;

import javax.faces.lifecycle.Lifecycle;

/**
 * 视频播放状态转换场景
 * <p>
 * {@link Lifecycle#execute(javax.faces.context.FacesContext)} 该方法通过外部状态控制改变行为
 * {@link javax.faces.webapp.FacesServlet} 状态控制
 *
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public abstract class CourseVideoState {
 /**
 * 视频上下文
 */
 protected CourseVideoContext courseVideoContext;

 public void setCourseVideoContext(CourseVideoContext courseVideoContext) {
 this.courseVideoContext = courseVideoContext;
 }

 public abstract void play();

 public abstract void speed();

 public abstract void pause();

 public abstract void stop();

}

视频上下文

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * 视频上下文
 *
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class CourseVideoContext {

 /**
 * 存储当前状态
 */
 private CourseVideoState courseVideoState;

 /**
 * 结合享元模式设置为内存共享
 */
 public final static PlayState PLAY_STATE = new PlayState();
 public final static StopState STOP_STATE = new StopState();
 public final static PauseState PAUSE_STATE = new PauseState();
 public final static SpeedState SPEED_STATE = new SpeedState();

 public CourseVideoState getCourseVideoState() {
 return courseVideoState;
 }

 public void setCourseVideoState(CourseVideoState courseVideoState) {
 this.courseVideoState = courseVideoState;
 // 把自己本身设置到实例上下文中
 this.courseVideoState.setCourseVideoContext(this);
 }
 public void play(){
 this.courseVideoState.play();
 }

 public void speed(){
 this.courseVideoState.speed();
 }

 public void stop(){
 this.courseVideoState.stop();
 }

 public void pause(){
 this.courseVideoState.pause();
 }
}

各状态实现

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class PlayState extends CourseVideoState {

 @Override
 public void play() {
 System.out.println("正常播放课程视频状态");
 }

 @Override
 public void speed() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.SPEED_STATE);
 }

 @Override
 public void pause() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PAUSE_STATE);
 }

 @Override
 public void stop() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE);
 }
}

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class PauseState extends CourseVideoState {

 @Override
 public void play() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE);
 }

 @Override
 public void speed() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.SPEED_STATE);
 }

 @Override
 public void pause() {
 System.out.println("暂停播放课程视频状态");
 }

 @Override
 public void stop() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE);
 }
}

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class SpeedState extends CourseVideoState {
 @Override
 public void play() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE);
 }

 @Override
 public void speed() {
 System.out.println("快进播放课程视频状态");
 }

 @Override
 public void pause() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PAUSE_STATE);
 }

 @Override
 public void stop() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE);
 }
}

package top.fjy8018.designpattern.pattern.behavior.state;

/**
 * @author F嘉阳
 * @date 2020/7/22 15:20
 */
public class StopState extends CourseVideoState {
 @Override
 public void play() {
 super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE);
 }

 @Override
 public void speed() {
 System.out.println("ERROR 停止状态不能快进!!");
 }

 @Override
 public void pause() {
 System.out.println("ERROR 停止状态不能暂停!!");
 }

 @Override
 public void stop() {
 System.out.println("停止播放课程视频状态");
 }
}

测试

package top.fjy8018.designpattern.pattern.behavior.state;

import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.*;

/**
 * @author F嘉阳
 * @date 2020/7/22 16:19
 */
class PlayStateTest {

 @Test
 void play() {
 CourseVideoContext courseVideoContext = new CourseVideoContext();
 courseVideoContext.setCourseVideoState(new PlayState());

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());
 courseVideoContext.pause();

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());

 courseVideoContext.speed();

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());

 courseVideoContext.stop();

 System.out.println("当前状态:"+courseVideoContext.getCourseVideoState().getClass().getSimpleName());

 courseVideoContext.speed();
 }
}

输出

当前状态:PlayState
当前状态:PauseState
当前状态:SpeedState
当前状态:StopState
ERROR 停止状态不能快进!!