[{"content":"背景 在过去的DevOps实践中，跨环境的基础设施部署和迁移经常存在遗漏，导致环境差异，部署一个新环境往往需要几天。 传统CICD流水线采用“推”模式，将变更推送到不同的环境，推送后也不对目标环境状态进行管理和监控，容易导致推送后部署失败。 随着云原生容器化已经成为行业事实标准，GitOps成为一种更可靠的应用部署和基础设置运维的范式。\n技术选型 ArgoCD遵循GitOps的核心原则，已经成为落地GitOps的默认技术选型，因此此处不探讨其他选型方案。\n原理 Git 作为唯一事实来源（GitOps思想） 自动化和持续同步：Argo CD会持续监控 Git 仓库的状态，并自动将集群的实际状态与 Git 中声明的目标状态进行同步。 利用K8s的Kustomize能力完成对集群apply操作 实践 环境 2C 8G K3s 安装ArgoCD $ kubectl create namespace argocd $ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml 检查安装结果\n查看argo cd登录密码\nkubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\u0026#34;{.data.password}\u0026#34; | base64 -d; echo 设计代码库结构 落地ArgoCD通常有以下几种仓库设计范式\n单仓库模式 多仓库模式 App of Apps模式 本次采用App of Apps的单仓库模式，仓库结构如下\n. ├── apps │ ├── app-one │ ├── app-two ├── cluster-infra │ ├── argocd │ │ ├── base │ │ └── overlays │ ├── ingress │ │ ├── base │ │ └── overlays │ │ └── dev │ └── prometheus-stack │ ├── base │ └── overlays │ └── dev └── environments └── dev └── root.yaml 该结构特点如下\n基础设施和应用部署分离（apps和cluster-infra） 环境基础配置和环境定制配置分离（base和overlays） 环境根节点配置（environments控制git仓库配置、应用apps注册） 本次将该代码库存放到阿里云代码仓库中。\n环境配置迁移 目前的k3s环境中已经手动运维安装了argocd，现在需要将已安装的配置迁移到git库中\n将官方安装配置https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml写入到cluster-infra/argocd/base/argocd.yaml中\n配置``cluster-infra/argocd/base/kustomization.yaml`文件，注册同目录的安装文件\nresources: - argocd.yaml 本次依赖阿里云代码仓库，通过ssh连接，需要覆盖默认的argocd配置，编写cluster-infra/argocd/overlays/with-known-hosts/ssh-known-hosts.patch.yaml\napiVersion: v1 kind: ConfigMap metadata: name: argocd-ssh-known-hosts-cm data: ssh_known_hosts: |- # BEGIN \u0026gt;\u0026gt;\u0026gt; 自定义条目（全局生效） codeup.aliyun.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6WbValczSDKC4ktjRzDNC1qINDAViw/oGA8ud7qH5F9XFZ4QAWhZmFBsp+cppBIOM63exjc4O9vbzMrd7AUItXigymGfTqrGt5VykIiaNhjttxSwbZf3N9eDcAvX4gGm3fcm6D1ACvPxvxMsCwOpdz5tDHqDxGOLPAXN+FSAvjdcN4uMz0MXghGh9Q3zSpy8z1TLkAKT2QBnr4q2tsdZKwWON8bBHIPwP3RSAdSUL6bAyotyncMDAvG53kwRKJ3wJ5ZezXqZNNV7EzHds+pvwThtuWtWsBuuRpL/oZiviWlXy6hILoWbkRfgONPwT0faaOn1GLzjHLuqHHfALGOen # END \u0026lt;\u0026lt;\u0026lt; 自定义条目 # This file was automatically generated by hack/update-ssh-known-hosts.sh. DO NOT EDIT [ssh.github.com]:443 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg= [ssh.github.com]:443 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl [ssh.github.com]:443 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk= bitbucket.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPIQmuzMBuKdWeF4+a2sjSSpBK0iqitSQ+5BM9KhpexuGt20JpTVM7u5BDZngncgrqDMbWdxMWWOGtZ9UgbqgZE= bitbucket.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQeJzhupRu0u0cdegZIa8e86EG2qOCsIsD1Xw0xSeiPDlCr7kq97NLmMbpKTX6Esc30NuoqEEHCuc7yWtwp8dI76EEEB1VqY9QJq6vk+aySyboD5QF61I/1WeTwu+deCbgKMGbUijeXhtfbxSxm6JwGrXrhBdofTsbKRUsrN1WoNgUa8uqN1Vx6WAJw1JHPhglEGGHea6QICwJOAr/6mrui/oB7pkaWKHj3z7d1IC4KWLtY47elvjbaTlkN04Kc/5LFEirorGYVbt15kAUlqGM65pk6ZBxtaO3+30LVlORZkxOh+LKL/BvbZ/iRNhItLqNyieoQj/uh/7Iv4uyH/cV/0b4WDSd3DptigWq84lJubb9t/DnZlrJazxyDCulTmKdOR7vs9gMTo+uoIrPSb8ScTtvw65+odKAlBj59dhnVp9zd7QUojOpXlL62Aw56U4oO+FALuevvMjiWeavKhJqlR7i5n9srYcrNV7ttmDw7kf/97P5zauIhxcjX+xHv4M= github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg= github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk= gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY= gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9 ssh.dev.azure.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H vs-ssh.visualstudio.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H 配置``cluster-infra/argocd/overlays/with-known-hosts/kustomization.yaml`文件，注册hosts patch文件\nresources: - ../../base patchesStrategicMerge: - ssh-known-hosts.patch.yaml 按这样配置后，argocd会先执行base的安装文件apply基础环境，再使用ssh-known-hosts.patch.yaml覆盖ssh的配置，保证环境与git配置一致。\n接下来，继续在apps根目录注册应用，配置environments/dev/kustomization.yaml\nresources: - ../../cluster-infra/argocd/overlays/with-known-hosts 编写根目录git配置\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: dev-root namespace: argocd spec: project: default source: repoURL: git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git targetRevision: master path: environments/dev kustomize: enableHelm: true destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: automated: prune: true selfHeal: true 到此，所有配置迁移完成，将git仓库提交到远程后，在master节点添加私钥\nargocd repo add git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git --ssh-private-key-path /root/.ssh/gitops_ed25519 同步dev根应用\nkubectl apply -f dev-root.yaml 本地使用kubectl暴露临时端口，再使用argocd cli连接集群查看状态\nkubectl port-forward svc/argocd-server -n argocd --address 0.0.0.0 30443:443 argocd login xxxx:30443 --username admin argocd app list 部署ingress 一直使用临时端口+命令行查看应用状态也不方便，官方提供webui界面查看部署状态，考虑使用ingress暴露固定域名访问\ningress的部署也遵循相同步骤\n配置官方安装文件 配置环境个性化 端口暴露 根应用添加配置 将官方https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml安装文件配置到cluster-infra/ingress/base/ingress-nginx.yaml\n配置端口暴露cluster-infra/ingress/overlays/dev/ingress-nginx-controller-nodeport.yaml\napiVersion: v1 kind: Service metadata: name: ingress-nginx-controller namespace: ingress-nginx spec: type: NodePort ports: - name: http port: 80 targetPort: http protocol: TCP nodePort: 30080 - name: https port: 443 targetPort: https protocol: TCP nodePort: 30443 根应用添加配置\nresources: - ../../cluster-infra/ingress/overlays/dev - ../../cluster-infra/argocd/overlays/with-known-hosts 在argocd环境目录下添加ingress域名配置\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: argocd-ingress namespace: argocd annotations: nginx.ingress.kubernetes.io/backend-protocol: \u0026#34;HTTPS\u0026#34; spec: ingressClassName: nginx rules: - host: xxxx http: paths: - path: / pathType: Prefix backend: service: name: argocd-server port: number: 443 git提交后即可自动同步并创建ingress\n访问域名查看argocd webui 部署Prometheus+Grafana 为了完善集群监控，argocd 支持通过helm charts部署应用\n配置cluster-infra/prometheus-stack/base/prometheus-stack.yaml\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: prometheus-stack namespace: argocd annotations: argocd.argoproj.io/sync-options: ServerSideApply=true spec: project: default source: repoURL: https://prometheus-community.github.io/helm-charts chart: kube-prometheus-stack targetRevision: 75.16.1 helm: valueFiles: - values.yaml destination: server: https://kubernetes.default.svc namespace: monitoring syncPolicy: automated: prune: true selfHeal: true 注意，由于Helm仓库部署prometheus时伴随大量CRD资源文件apply，需要开启ServerSideApply模式避免通过接口apply导致字符超长报错。 参考：https://argocd.devops.gold/user-guide/sync-options/#_9\n使用helm模式安装的应用无法使用patchesStrategicMerge策略覆盖，目前覆盖个性化配置只能将所有内容写到一个资源文件内，因此dev环境的个性化配置values如下\nprometheus: prometheusSpec: retention: 3d retentionSize: \u0026#34;2Gi\u0026#34; resources: requests: memory: 128Mi limits: memory: 1Gi storageSpec: volumeClaimTemplate: spec: resources: requests: storage: 5Gi ingress: enabled: true ingressClassName: nginx annotations: {} hosts: - prometheus.fjy8018.top paths: - / tls: - hosts: - xxx.xxx secretName: prometheus-tls prometheusSpec: externalUrl: https://xxx.xxx routePrefix: / grafana: adminPassword: \u0026#34;2EvwJJ0NOLzhYa\u0026#34; resources: requests: memory: 64Mi limits: memory: 512Mi ingress: enabled: true ingressClassName: nginx annotations: {} hosts: - xxx.xxx tls: - hosts: - xxx.xxx secretName: grafana-tls 需要在dev中覆盖base的资源文件，通过multi repo模式合并helm配置\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: prometheus-stack namespace: argocd annotations: argocd.argoproj.io/sync-options: ServerSideApply=true spec: project: default sources: - repoURL: https://prometheus-community.github.io/helm-charts chart: kube-prometheus-stack targetRevision: 75.16.1 helm: valueFiles: - $values/cluster-infra/prometheus-stack/overlays/dev/values-dev.yaml - repoURL: \u0026#39;git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git\u0026#39; targetRevision: master ref: values destination: server: https://kubernetes.default.svc namespace: monitoring syncPolicy: automated: prune: true selfHeal: true prometheus依赖CRD提前部署的namespace需要注册为独立应用提前配置好\napiVersion: v1 kind: Namespace metadata: name: monitoring 命名空间的配置和普通应用一致\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: common-ns namespace: argocd spec: project: default source: repoURL: git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git targetRevision: master path: cluster-infra/common-ns destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: automated: prune: true selfHeal: true 最终根目录应用配置如下\nresources: - ../../cluster-infra/ingress/overlays/dev - ../../cluster-infra/argocd/overlays/with-known-hosts - ../../cluster-infra/prometheus-stack/overlays/dev - common-ns-app.yaml 同步后效果如下 查看监控部署拓扑 通过域名访问grafana\n","date":"2025-08-23T22:49:50+08:00","permalink":"/archives/292/","title":"ArgoCD落地GitOps"},{"content":"参考资料 jingyaogong/minimind\n环境准备 本地环境配置\nCPU 12700 内存 32G DDR4 3200 GPU RTX 2060Super 系统 Win11 CUDA 12.6 Python 3.12 由于需要本地编译，windows环境需要安装Microsoft Visual C++ 编译器，tiktoken编译需要rust环境，安装地址：\nBuild Tools for Visual Studio rustup.rs - The Rust toolchain installer 克隆并安装环境\ngit clone https://github.com/jingyaogong/minimind.git cd minimind pip install -r requirements.txt 注意：此处不建议使用清华源、也不建议使用conda虚拟环境，会导致pytorch包版本匹配失败\n依赖安装检查，执行以下代码\nimport torch print(torch.__version__) print(torch.version.cuda) print(torch.cuda.is_available()) 输入如下即成功\n(base) PS F:\\java_workspace\\minimind\u0026gt; python .\\test_env.py 2.6.0+cu126 12.6 True 按照文档指引下载默认的预训练集和监督训练集\nminimind_dataset · 数据集 训练过程 预训练 执行预训练\npython train_pretrain.py 在我的环境下需要3小时才能完成预训练\n执行预训练，得到pretrain_*.pth作为预训练的输出权重（其中为模型的dimension，默认为512） 监督微调 python train_full_sft.py 执行监督全参数微调，在我的环境下需要2.5小时 效果测试 测试全参微调\npython eval_model.py --model_mode 1 从结果来看，能进行对话，但受限于参数量，很多知识不完整，对话回答质量比较一般。\n模型优化 受本地环境限制，只能完成监督微调步骤，后续步骤执行会爆显存，耗时显著增加。 后续学习改为使用阿里云PAI-DSW环境\n","date":"2025-03-30T11:44:00+08:00","permalink":"/archives/291/","title":"本地LLM训练"},{"content":"背景 由于官方DeepSeek频繁不可用，第三方API站点部署的全量模型无法联网，因此有需要联网推理的场景只能使用本地部署解决。\n环境 CPU：12700 内存：32G DDR4 3200 显卡：RTX 2060 Super\n模型准备 受显存限制，使用ollama只能下载8b和14b模型。\n8b模型可以完全加载到显存中，使用cuda推理。 14b模型需要利用CPU+GPU混合推理。 本地下载开源docker镜像open-webui\ndocker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main 该镜像下载困难可以手动传到阿里云镜像仓库做一次代理，本次使用我同步的代理镜像启动\ndocker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --rm registry.cn-guangzhou.aliyuncs.com/fjy8018_proxy/open-webui:main 首次启动耗时较长，耐心等待，显示如下画面表示启动完成 联网API 进入http://localhost:3000/admin/settings 点击联网搜索配置搜索引擎 目前有3种免费的联网搜索API，各有优劣\nDuckduckgo 优点：免费，无需API配置 缺点：网络无法直接访问。\nBing Search 优点：免费，国内网络可直连。 缺点：需要微软账户+信用卡认证。\nGoogle Search 优点：免费，搜索内容可靠。 缺点：需要配置，网络无法直连。\n此处为保障最好的搜索质量，使用Google搜索API接入。\n申请API和密钥 进入谷歌可编程搜索引擎官网：Programmable Search Engine | Google for Developers 创建搜索引擎 点击控制面板 创建搜索引擎，名称任意 创建后得到搜索引擎ID 选择程序化访问 获取密钥 配置联网搜索 选择google pse，填写密钥和搜索ID 验证 8b + 联网搜索 使用8b+联网搜索验证 容器日志可以看到搜索到的文本内容 8b联网+推理显存沾满，显卡满载，但不需要CPU参与推理，32token/s，速度较快。 14b+联网搜索 改成14b后，token速度下降到7token/s，勉强可用。\n由于显存不足，需要CPU+GPU混合推理，显卡cuda性能无法完全利用，CPU占用率高。 ","date":"2025-02-09T17:28:00+08:00","permalink":"/archives/289/","title":"DeepSeek 本地部署开启联网搜索"},{"content":"基本概念 参考oracle jdk 21中对ClassLoader定义\nA class loader is an object that is responsible for loading classes. The class ClassLoader is an abstract class. Given the binary name of a class, a class loader should attempt to locate or generate data that constitutes a definition for the class. A typical strategy is to transform the name into a file name and then read a \u0026ldquo;class file\u0026rdquo; of that name from a file system.\n在Java中，ClassLoader负责加载类，当给定一个类的二进制名称，ClassLoader会尝试定位或生成构成该类定义的数据。\n层次结构 ClassLoader遵循双亲委派模型，当请求查找类或资源时， ClassLoader 实例通常会将类或资源的搜索委托给其父类加载器，然后再尝试查找类或资源本身\n由于JDK9中模块系统（Jigsaw）的引入，jdk8和jdk9类加载器层次结构有区别，ExtClassLoader变成PlatformClassLoader负责加载--module-path（或MODPATH环境变量）指定的模块\ngraph TD Bootstrap[Bootstrap ClassLoader] Extension[Extension ClassLoader] Application[Application ClassLoader] Bootstrap --\u003e Extension Extension --\u003e Application subgraph \"JDK 8\" Bootstrap:::primary Extension:::secondary Application:::tertiary end graph TD Bootstrap[Bootstrap ClassLoader] Platform[Platform ClassLoader] App[Application ClassLoader] Bootstrap --\u003e Platform Platform --\u003e App subgraph \"JDK 9\" Bootstrap:::primary Platform:::secondary App:::tertiary end 生命周期 类加载的过程可以分为几个阶段及常见异常\n加载（Loading）\nClassNotFoundException（找不到目标类）\n验证（Verification）\nClassFormatError（类文件格式不正确）\nNoClassDefFoundError（已加载的类无法初始化）\n准备（Preparation）\n解析（Resolution）\nLinkageError（类加载器无法找到或者无法链接到其他类或接口）\nIncompatibleClassChangeError\nAbstractMethodError\nNoSuchMethodError\n\u0026hellip;\n初始化（Initialization）\n核心方法 ClassLoader类提供了几个关键的方法来支持类的加载过程，其中loadClass和findClass最常使用\nloadClass protected Class\u0026lt;?\u0026gt; loadClass(String name, boolean resolve) throws ClassNotFoundException Loads the class with the specified binary name. The default implementation of this method searches for classes in the following order:\nInvoke findLoadedClass(String) to check if the class has already been loaded. Invoke the loadClass method on the parent class loader. If the parent is null the class loader built into the virtual machine is used, instead. Invoke the findClass(String) method to find the class. If the class was found using the above steps, and the resolve flag is true, this method will then invoke the resolveClass(Class) method on the resulting Class object.\nSubclasses of ClassLoader are encouraged to override findClass(String), rather than this method.\nUnless overridden, this method synchronizes on the result of getClassLoadingLock method during the entire class loading process.\nParameters:\nname - The binary name of the class\nresolve - If true then resolve the class\nReturns:\nThe resulting Class object\nThrows:\nClassNotFoundException - If the class could not be found\n官方文档给出了方法设计介绍，loadClass用于实现双亲委派职责，当父类无法加载目标class时，调用子类的findClass加载。\n当需要打破双亲委派模型时，则直接覆盖loadClass\nfindClass protected Class\u0026lt;?\u0026gt; findClass(String name) throws ClassNotFoundException Finds the class with the specified binary name. This method should be overridden by class loader implementations that follow the delegation model for loading classes, and will be invoked by the loadClass method after checking the parent class loader for the requested class.\nImplementation Requirements:\nThe default implementation throws ClassNotFoundException.\nParameters:\nname - The binary name of the class\nReturns:\nThe resulting Class object\nThrows:\nClassNotFoundException - If the class could not be found\nSince:\n1.2\n在findClass用于实现自定义类加载逻辑\n以常用的URLClassLoader为例，findClass将传入的类名转换成路径格式匹配在URLClassPath类路径下实际存在的文件结构，如果找到了对应的资源，则继续处理；否则返回null。\npublic class URLClassLoader extends SecureClassLoader implements Closeable { protected Class\u0026lt;?\u0026gt; findClass(final String name) throws ClassNotFoundException { final Class\u0026lt;?\u0026gt; result; try { result = AccessController.doPrivileged( new PrivilegedExceptionAction\u0026lt;Class\u0026lt;?\u0026gt;\u0026gt;() { public Class\u0026lt;?\u0026gt; run() throws ClassNotFoundException { String path = name.replace(\u0026#39;.\u0026#39;, \u0026#39;/\u0026#39;).concat(\u0026#34;.class\u0026#34;); Resource res = ucp.getResource(path, false); if (res != null) { try { return defineClass(name, res); } catch (IOException e) { throw new ClassNotFoundException(name, e); } } else { return null; } } }, acc); } catch (java.security.PrivilegedActionException pae) { throw (ClassNotFoundException) pae.getException(); } if (result == null) { throw new ClassNotFoundException(name); } return result; } } 应用 tomcat 为了支持应用隔离和模块化部署，tomcat扩展了已有的双亲委派模型\n以tomcat9为例，其层次结构如下\ngraph TD Bootstrap[Bootstrap ClassLoader] System[System ClassLoader] Common[Common ClassLoader] Webapp1[Webapp1 ClassLoader] Webapp2[Webapp2 ClassLoader] Webapp3[Webapp3 ClassLoader] Bootstrap --\u003e System System --\u003e Common Common --\u003e Webapp1 Common --\u003e Webapp2 Common --\u003e Webapp3 Bootstrap（所有应用可见，JVM核心依赖）\n$JAVA_HOME/jre/lib/ext\nSystem（所有应用可见，tomcat核心依赖）\n$CATALINA_HOME/bin\nCommon（需要对所有应用可见的扩展类）\nWebapp（每个应用隔离，打破双亲委派，本地优先）\nAs mentioned above, the web application class loader diverges from the default Java delegation model (in accordance with the recommendations in the Servlet Specification, version 2.4, section 9.7.2 Web Application Classloader). When a request to load a class from the web application\u0026rsquo;s WebappX class loader is processed, this class loader will look in the local repositories first, instead of delegating before looking.\n重点在于webapp classloader，其打破双亲委派机制，当从 Web 应用程序的 WebappX 类加载器加载类的请求被处理时，这个类加载器会首先查看本地存储库，而不是先进行委托。\n在应用程序视角，类查找顺序如下\nBootstrap ClassLoader\nWebApp ClassLoader\n/WEB-INF/classes\n/WEB-INF/lib/*.jar\nSystem ClassLoader\nCommon ClassLoader\n当应用配置\u0026lt;Loader delegate=\u0026quot;true\u0026quot;/\u0026gt;时，可切换为标准双亲委派模型\nOSGi OSGi是一个 Java 模块化系统和服务平台，它提供了一种动态管理模块（Bundle）的方法，支持不停止应用程序的情况下动态更新、添加或删除模块。\n以Apache Felix实现的OSGi为例，其查找逻辑如下\n循环检测：防止在查找过程中出现无限递归。 委托给父类加载器：如果目标类或资源属于需要委托给父类加载器处理的包，则先尝试使用父类加载器进行查找。 从导入列表中查找：在当前修订版本的导入列表中搜索类或资源。 从本地类路径查找：如果前面步骤未能找到，则尝试从当前修订版本自己的类路径中查找。 从动态导入中查找：最后，如果还没有找到，则会尝试从动态导入中查找。 private Object findClassOrResourceByDelegation(String name, boolean isClass) throws ClassNotFoundException, ResourceNotFoundException { // 初始化结果为null，表示尚未找到类或资源 Object result = null; // 用于循环检测的集合，避免在查找过程中出现无限递归的情况 Set requestSet = (Set) m_cycleCheck.get(); if (requestSet == null) { // 如果当前线程中没有设置循环检测集合，则创建一个新的并设置到当前线程 requestSet = new HashSet(); m_cycleCheck.set(requestSet); } // 将要查找的名字添加到循环检测集合中，如果已经存在说明出现了循环依赖 if (requestSet.add(name)) { try { // 根据是查找类还是资源来获取目标的包名 String pkgName = (isClass) ? Util.getClassPackage(name) : Util.getResourcePackage(name); // 检查是否应该将该包委托给父类加载器 if (shouldBootDelegate(pkgName)) { try { // 获取适当的类加载器进行委托 ClassLoader bdcl = getBootDelegationClassLoader(); // 根据是要查找类还是资源，调用相应的方法 result = (isClass) ? (Object) bdcl.loadClass(name) : (Object) bdcl.getResource(name); // 如果是java.*包下的内容或者已经找到了结果，则直接返回 if (pkgName.startsWith(\u0026#34;java.\u0026#34;) || (result != null)) { return result; } } catch (ClassNotFoundException ex) { // 如果是java.*包下的类且未找到，则抛出异常 if (pkgName.startsWith(\u0026#34;java.\u0026#34;)) { throw ex; } } } // 在导入列表中查找 result = searchImports(pkgName, name, isClass); // 如果在导入列表中未找到，则尝试从修订版本自己的类路径中查找 if (result == null) { if (isClass) { // 获取内部类加载器，并尝试加载类 ClassLoader cl = getClassLoaderInternal(); if (cl == null) { // 如果类加载器无效，则抛出异常 throw new ClassNotFoundException( \u0026#34;Unable to load class \u0026#39;\u0026#34; + name + \u0026#34;\u0026#39; because the bundle wiring for \u0026#34; + m_revision.getSymbolicName() + \u0026#34; is no longer valid.\u0026#34;); } result = ((BundleClassLoader) cl).findClass(name); } else { // 如果不是查找类而是资源，则使用修订版本的方法来查找资源 result = m_revision.getResourceLocal(name); } // 如果还是没找到，则尝试从动态导入中查找 if (result == null) { result = searchDynamicImports(pkgName, name, isClass); } } } finally { // 清理：从循环检测集合中移除当前查找的名字 requestSet.remove(name); } } else { // 如果检测到循环依赖，则返回null以打破循环 return null; } // 如果最终结果仍为null，根据查找类型抛出相应的异常 if (result == null) { if (isClass) { throw new ClassNotFoundException( name + \u0026#34; not found by \u0026#34; + this.getBundle()); } else { throw new ResourceNotFoundException( name + \u0026#34; not found by \u0026#34; + this.getBundle()); } } // 返回最终找到的结果 return result; } SofaArk SofaArk是蚂蚁集团开源轻量类隔离框架\n其定义3个概念：Ark Container，Ark Plugin，Ark Biz。\nsofaark通过maven插件在打包时将插件启动器、导出类、导入类、导出资源、导入资源、优先级等元信息保存在 Ark Plugin 中的 META-INF/MANIFEST.MF 中，每个ark plugin都有独立的类加载器读取插件元信息和类资源导入导出关系。插件之间可以互相委托加载类。 ark biz只能加载自己模块的类信息，也可以委托plugin加载所需的类。 graph TD Bootstrap[BootstrapClassLoader] Extension[ExtensionClassLoader] Application[ApplicationClassLoader] URLClassLoader[URLClassLoader] Container[ContainerClassLoader] Agent[AgentClassLoader] Biz[BizClassLoader] Plugin[PluginClassLoader] Bootstrap --\u003e Extension Extension --\u003e Application Application --\u003e URLClassLoader URLClassLoader --\u003e Container URLClassLoader --\u003e Biz URLClassLoader --\u003e Plugin URLClassLoader --\u003e Agent 源码 AbstractClasspathClassLoader提供了类加载查找模式的抽象实现，loadClass会从以下顺序查找\n本地类加载缓存 子类实现的loadClassInternal方法 BizClassLoader实现如下\npackage com.alipay.sofa.ark.container.service.classloader; public class BizClassLoader extends AbstractClasspathClassLoader { @Override protected Class\u0026lt;?\u0026gt; loadClassInternal(String name, boolean resolve) throws ArkLoaderException { Class\u0026lt;?\u0026gt; clazz = null; // 0. sun reflect related class throw exception directly if (classloaderService.isSunReflectClass(name)) { throw new ArkLoaderException( String .format( \u0026#34;[ArkBiz Loader] %s : can not load class: %s, this class can only be loaded by sun.reflect.DelegatingClassLoader\u0026#34;, bizIdentity, name)); } // 1. findLoadedClass if (clazz == null) { clazz = findLoadedClass(name); } // 2. JDK related class if (clazz == null) { clazz = resolveJDKClass(name); } // 3. Ark Spi class if (clazz == null) { clazz = resolveArkClass(name); } // 4. pre find class if (clazz == null) { clazz = preLoadClass(name); } // 5. Plugin Export class if (clazz == null) { clazz = resolveExportClass(name); } // 6. Biz classpath class if (clazz == null) { clazz = resolveLocalClass(name); } // 7. Java Agent ClassLoader for agent problem if (clazz == null) { clazz = resolveJavaAgentClass(name); } // 8. post find class if (clazz == null) { clazz = postLoadClass(name); } if (clazz != null) { if (resolve) { super.resolveClass(clazz); } return clazz; } throw new ArkLoaderException(String.format(\u0026#34;[ArkBiz Loader] %s : can not load class: %s\u0026#34;, bizIdentity, name)); } } PluginClassLoader实现如下\npackage com.alipay.sofa.ark.container.service.classloader; public class PluginClassLoader extends AbstractClasspathClassLoader { @Override protected Class\u0026lt;?\u0026gt; loadClassInternal(String name, boolean resolve) throws ArkLoaderException { Class\u0026lt;?\u0026gt; clazz = null; // 0. sun reflect related class throw exception directly if (classloaderService.isSunReflectClass(name)) { throw new ArkLoaderException( String .format( \u0026#34;[ArkPlugin Loader] %s : can not load class: %s, this class can only be loaded by sun.reflect.DelegatingClassLoader\u0026#34;, pluginName, name)); } // 1. findLoadedClass if (clazz == null) { clazz = findLoadedClass(name); } // 2. JDK related class if (clazz == null) { clazz = resolveJDKClass(name); } // 3. Ark Spi class if (clazz == null) { clazz = resolveArkClass(name); } // 4. pre find class if (clazz == null) { clazz = preLoadClass(name); } // 5. Import class export by other plugins if (clazz == null) { clazz = resolveExportClass(name); } // 6. Plugin classpath class if (clazz == null) { clazz = resolveLocalClass(name); } // 7. Java Agent ClassLoader for agent problem if (clazz == null) { clazz = resolveJavaAgentClass(name); } // 8. Post find class if (clazz == null) { clazz = postLoadClass(name); } if (clazz != null) { if (resolve) { super.resolveClass(clazz); } return clazz; } throw new ArkLoaderException(String.format( \u0026#34;[ArkPlugin Loader] %s : can not load class: %s\u0026#34;, pluginName, name)); } } 流程基本一致，主要在于plugin和biz读取到的import类元信息不同 参考资料 ClassLoader (Java SE 21 \u0026amp; JDK 21) (oracle.com) Apache Tomcat 9 (9.0.94) - Class Loader How-To apache/felix-atomos: Apache Felix Atomos (github.com) org.apache.felix.framework/src/main/java/org/apache/felix/framework/BundleWiringImpl.java at master · snefru/org.apache.felix.framework (github.com) https://www.sofastack.tech/projects/sofa-boot/sofa-ark-readme/ ","date":"2024-11-13T21:56:49+08:00","permalink":"/archives/287/","title":"ClassLoader类隔离"},{"content":"背景 ChatGLM2-6B 是开源中英双语对话模型 ChatGLM-6B的第二代版本，提升了性能和推理能力，可结合Langchain-Chatchat和Embedding 可实现基于本地知识库的推理能力\n准备环境 由于需要构建强大的知识库而非通过prompt组装问答方式，故需要更大的显存和内存，此处直接使用阿里云提供的A10显卡，可免费试用3个月\n申请机器 创建工作空间 创建实例 启动机器后进入工作台\n部署ChatGLM2-6B 下载仓库 参考官方仓库的部署指南\n$ git clone https://github.com/THUDM/ChatGLM2-6B $ cd ChatGLM2-6B 然后使用 pip 安装依赖：\n$ pip install -r requirements.txt 安装git lfs 阿里云的git源直接安装lfs会存在问题，需要安装前置依赖\n$ apt -y install build-essential nghttp2 libnghttp2-dev libssl-dev $ git lfs install --skip-smudge 下载模型 $ git clone https://huggingface.co/THUDM/chatglm2-6b /mnt/workspace/model/chatglm2-6b $ git lfs fetch $ git lfs checkout 修改启动配置 此处只使用webUI模式验证，修改 web_demo.py\ntokenizer= AutoTokenizer.from_pretrained(\u0026#34;/mnt/workspace/model/chatglm2-6b\u0026#34;, trust_remote_code=True) model= AutoModel.from_pretrained(\u0026#34;/mnt/workspace/model/chatglm2-6b\u0026#34;, trust_remote_code=True).half().cuda() # 要暴露在公网上 demo.queue().launch(share=True, inbrowser=True, server_name=\u0026#39;0.0.0.0\u0026#39;, server_port=7860) 安装启动依赖\n$ pip install gradio 启动WebUI $ python web_demo.py 启动成功\n使用非量化模型，显存占用12G\n问答验证 基于A10显卡推理，运算速度非常快，对问题的理解能力也比之前体验的chatglm-6b更加准确，但依然有不少文字错误\n部署Langchain知识库应用 介绍 Langchain-Chatchat是一种利用 langchain 思想实现的基于本地知识库的问答应用，目标期望建立一套对中文场景与开源模型支持友好、可离线运行的知识库问答解决方案。\n原理示意图\n更多介绍参考官方仓库\n下载仓库 Langchain依赖的python包有些许差别，需要先升级python依赖版本\n$ pip3 install --upgrade pip $ pip uninstall accelerate $ pip install accelerate\u0026gt;=0.20.3 # 拉取仓库 $ git clone https://github.com/chatchat-space/Langchain-Chatchat.git # 进入目录 $ cd Langchain-Chatchat # 安装全部依赖 $ pip install -r requirements.txt 下载Embedding模型 此处使用官方推荐的m3e-base模型为例\n$ git clone https://huggingface.co/moka-ai/m3e-base $ git lfs fetch $ git lfs checkout 修改配置 修改Langchain模型引用配置\n$ cd Langchain-Chatchat 复制样例文件，创建新配置configs/model_config.py 和 configs/server_config.py\n本地的 LLM 模型本地存储路径写在 llm_model_dict 对应模型的 local_model_path 属性中\nllm_model_dict={ \u0026#34;chatglm2-6b\u0026#34;: { \u0026#34;local_model_path\u0026#34;: \u0026#34;/mnt/workspace/model/chatglm2-6b\u0026#34;, \u0026#34;api_base_url\u0026#34;: \u0026#34;http://127.0.0.1:9090/v1\u0026#34;, # \u0026#34;name\u0026#34;修改为 FastChat 服务中的\u0026#34;api_base_url\u0026#34; \u0026#34;api_key\u0026#34;: \u0026#34;EMPTY\u0026#34; }, } 本地的 Embedding 模型本地存储路径写在 embedding_model_dict 对应模型位置\nembedding_model_dict = { \u0026#34;m3e-base\u0026#34;: \u0026#34;/mnt/workspace/model/m3e-base\u0026#34;, } 重建知识库 首次使用模型需要重建知识库\n$ python init_database.py --recreate-vs 启动知识库 一键启动所有 Fastchat 服务、API 服务、WebUI 服务\n$ python startup.py -a 问答验证 LLM模型验证，连接本地ChatGLM2-6B成功\n工程自带样例知识库，可以使用内置知识库验证，此处跳过，直接使用个人知识库验证\n构建知识库 基金研报知识库 从天天基金拉取研报：中信证券：把握宏观政策接连发力下电商板块估值修复机会_天天基金网 (eastmoney.com)\n将内容存储为txt上传到知识库\n基于基金研报知识库问答，总结准确，速度非常快\n推理过程日志，显存占用14G\n个人面试知识库 将Java并发知识点转为txt类型知识库上传并创建新的知识库空间\n不使用知识库情况下回答\n使用知识库情况下回答\n显然使用知识库回答更加准确\n","date":"2023-09-06T22:43:00+08:00","permalink":"/archives/286/","title":"基于 ChatGLM2-6B 搭建个人知识库"},{"content":"背景 在ChatGPT引爆大语言模型热潮下，国内清华大学开源了包含62亿参数的GLM-6B，由于使用的参数较少，因此能支持在消费级显卡上私有化部署体验。\n硬件 CPU：12700\n内存：32G\nGPU：RTX 2060 SUPER 8G\n实践-ChatGLM-6B官方版本 介绍 GitHub - THUDM/ChatGLM-6B: ChatGLM-6B: An Open Bilingual Dialogue Language Model | 开源双语对话语言模型\nChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于 General Language Model (GLM) 架构，具有 62 亿参数。结合模型量化技术，用户可以在消费级的显卡上进行本地部署（INT4 量化级别下最低只需 6GB 显存）。 ChatGLM-6B 使用了和 ChatGPT 相似的技术，针对中文问答和对话进行了优化。经过约 1T 标识符的中英双语训练，辅以监督微调、反馈自助、人类反馈强化学习等技术的加持，62 亿参数的 ChatGLM-6B 已经能生成相当符合人类偏好的回答，更多信息请参考我们的博客。\n部署 采用秋叶一键部署包，使用8G显存版本\n启动成功，访问web-ui\n对话 对话流畅度与chatgpt类似，精准度不如gpt4\n由于显存限制，咨询过大的问题会直接导致爆显存无法回答\n实践-闻达版本 介绍 GitHub - wenda-LLM/wenda: 闻达：一个LLM调用平台。为小模型外挂知识库查找和设计自动执行动作，实现不亚于于大模型的生成能力\n闻达基于GLM-6B扩展，支持了本地知识库和联网知识库，提供了近似new bing的体验\n部署 使用官方一键包，默认使用6g显存模型\n启动成功\n对话-离线 只使用已有模型数据，询问\n回答内容文不对题\n主动提供正确数据内容后再询问，此时回答正确\n对话-使用在线知识库 打开联网知识库，清理历史后重新询问，不需要提供本地知识也可以自行通过网络检索后回答正确\n系统设计问题\n文档解析 直接读取文档 直接使用读取文档并对话的方式很容易脱离文档内容，导致答非所问\n尽管使用的是6g显存的版本，读取较大的文档时也会爆显存\n手动模拟文档读取 GLM文档分析实际上是将文档识别后转换为特定prompt进行问答对话\n参考内嵌prompt\n手动复制文档内容进行模拟对话，内容回答正确\n类AutoGPT功能 使用wenda提供的“根据标题写论文功能”可自动触发AutoGPT模式\n其会自己列提纲，并逐个扩展，无需手工操作\n总结 在消费级甜品卡上已经能实现接近chatgpt3的能力，ChatGLM-130B更是拥有1300亿参数的中英双语稠密模型，在能力上更进一步。\n未来GLM企业级私有化部署可能会被包装为简单易用的产品向政府、企业推广，满足数据保密要求。\n","date":"2023-05-28T19:50:00+08:00","permalink":"/archives/285/","title":"ChatGLM-6B 本地化部署体验"},{"content":"介绍 GraphQL订阅模式支持服务端主动向客户端推送数据通知，避免客户端轮训\nDGS Subscriptions 新增依赖 此处使用的是SpringBoot Web组件，故引入websockets依赖\n如果使用的是WebFlux则要引入对应的dgs-webflux\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.netflix.graphql.dgs\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;graphql-dgs-subscriptions-websockets-autoconfigure\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; 异步通知解析 此处定义为当新增一个Actor就给所有订阅者发异步通知\n@Slf4j @DgsComponent @RequiredArgsConstructor public class ActorDataFetcher { private final ActorAssembler actorAssembler; private FluxSink\u0026lt;Actor\u0026gt; actorStream; private ConnectableFlux\u0026lt;Actor\u0026gt; actorPublisher; @PostConstruct private void createActorPublisher() { Flux\u0026lt;Actor\u0026gt; publisher = Flux.create(emitter -\u0026gt; { actorStream = emitter; }); actorPublisher = publisher.publish(); actorPublisher.connect(); } @DgsMutation public Actor addActor(@InputArgument SubmitActor actor) { Actor actorEntity = actorAssembler.convert(actor); actorEntity.setActorId(10); actorEntity.setLastUpdate(DateUtil.format(new Date(), DatePattern.NORM_DATETIME_PATTERN)); actorStream.next(actorEntity); log.info(\u0026#34;服务端创建演员：{}\u0026#34;, actor); return actorEntity; } /** * 异步通知 * * @return */ @DgsSubscription public Publisher\u0026lt;Actor\u0026gt; actorAdded() { return actorPublisher; } } 单元测试 由于异步通知无法用浏览器自带的控制台测试，只能通过GraphQL客户端订阅后接收异步通知，此处参考官方最佳实践，使用单元测试模拟\n@Slf4j @SpringBootTest class ActorDataFetcherTest { @Autowired private DgsQueryExecutor dgsQueryExecutor; @Test void addActor() { GraphQLQueryRequest graphQLQueryRequest = new GraphQLQueryRequest( AddActorGraphQLQuery.newRequest() .actor( SubmitActor.newBuilder() .firstName(\u0026#34;fu\u0026#34;) .lastName(\u0026#34;jiayang\u0026#34;) .build()) .build(), new AddActorProjectionRoot() .actorId() .firstName() .lastUpdate()); dgsQueryExecutor.execute(graphQLQueryRequest.serialize()); } @Test void actorAdded() { ExecutionResult executionResult = dgsQueryExecutor.execute(\u0026#34;\u0026#34;\u0026#34; subscription { actorAdded { actorId firstName lastName lastUpdate } } \u0026#34;\u0026#34;\u0026#34;); Publisher\u0026lt;ExecutionResult\u0026gt; reviewPublisher = executionResult.getData(); List\u0026lt;Actor\u0026gt; actors = new CopyOnWriteArrayList\u0026lt;\u0026gt;(); reviewPublisher.subscribe(new Subscriber\u0026lt;\u0026gt;() { @Override public void onSubscribe(Subscription s) { s.request(2); } @Override public void onNext(ExecutionResult executionResult) { if (executionResult.getErrors().size() \u0026gt; 0) { executionResult.getErrors().forEach(error -\u0026gt; log.error(error.toString())); } Map\u0026lt;String, Object\u0026gt; actorResult = executionResult.getData(); Actor actor = new ObjectMapper().convertValue(actorResult.get(\u0026#34;actorAdded\u0026#34;), Actor.class); log.info(\u0026#34;客户端监听到演员新增：{}\u0026#34;,actor); actors.add(actor); } @Override public void onError(Throwable t) { t.printStackTrace(); } @Override public void onComplete() { } }); addActor(); addActor(); assertThat(actors.size()).isEqualTo(2); } } 从日志输出可以看到客户端成功接收异步通知\n总结 订阅模式属于GraphQL特色功能，类似消息队列实现方式，减少客户端对普通查询接口轮训性能开销\n","date":"2022-10-28T21:59:23+08:00","permalink":"/archives/283/","title":"GraphQL实践10——Netflix Dgs Graphql异步订阅"},{"content":"功能介绍 graphql-java提供了Mutations类型支持数据更新\nNGS Mutations 类型定义 type Mutation { addActor(actor: SubmitActor!): Actor } type Subscription { actorAdded: Actor } input SubmitActor { firstName: String! lastName: String! } SubmitActor类型可以交给代码生成器自动生成\n类型转换器 @Mapper(componentModel = SPRING) public interface ActorAssembler { Actor convert(SubmitActor submit); } 定义解析器 @DgsMutation public Actor addActor(@InputArgument SubmitActor actor) { Actor actorEntity = actorAssembler.convert(actor); actorEntity.setActorId(10); actorEntity.setLastUpdate(new Date()); return actorEntity; } 测试 访问http://localhost:8080/graphiql即可看到在线查询页面\n","date":"2022-10-28T21:58:54+08:00","permalink":"/archives/282/","title":"GraphQL实践9——Netflix Dgs Graphql数据更新"},{"content":"功能介绍 DGS官方开发了识别schema配置生成对应类型、测试类的代码生成器，而Maven插件则由社区维护\nMaven代码生成插件 新增依赖 \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.netflix.graphql.dgs.codegen\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;graphql-dgs-codegen-client-core\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;5.1.17\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; 还需要新增构建插件\n\u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;io.github.deweyjose\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;graphqlcodegen-maven-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.24\u0026lt;/version\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;generate\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;addGeneratedAnnotation\u0026gt;true\u0026lt;/addGeneratedAnnotation\u0026gt; \u0026lt;packageName\u0026gt;top.fjy8018.graphsqldemo.generated\u0026lt;/packageName\u0026gt; \u0026lt;schemaPaths\u0026gt; \u0026lt;param\u0026gt;src/main/resources/graphql/actor.graphqls\u0026lt;/param\u0026gt; \u0026lt;/schemaPaths\u0026gt; \u0026lt;!-- 生成测试客户端 --\u0026gt; \u0026lt;generateClient\u0026gt;true\u0026lt;/generateClient\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;/plugin\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.codehaus.mojo\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;build-helper-maven-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;phase\u0026gt;generate-sources\u0026lt;/phase\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;add-source\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;sources\u0026gt; \u0026lt;source\u0026gt;${project.build.directory}/generated-sources\u0026lt;/source\u0026gt; \u0026lt;/sources\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; 插件配置指引：deweyjose/graphqlcodegen: Maven port of the Netflix Gradle code generation plugin for graphql.\n编译 执行\nmvn clean package 即可生成对应的类\n存在问题 IDEA默认不识别target目录下的资源文件，生成后会导致无法引用\n解决方法 需要手动将对应的生成目录配置为Source Root即可\n总结 DGS官方推荐开发者优先使用代码生成器生成标准类库后再开发，可以减少开发量\n","date":"2022-10-28T21:57:00+08:00","permalink":"/archives/281/","title":"GraphQL实践8——Netflix Dgs Graphql代码生成"},{"content":"问题背景 上一篇文章提到，使用List查询每次都返回全量数据，而实际场景更多使用分页查询，graphql-java提供Connection实现游标分页，在Dgs也有对应功能扩展Relay Pagination\n集成Relay Pagination 新增依赖 \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.netflix.graphql.dgs\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;graphql-dgs-pagination\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; Mybatis Plus 配置分页插件 @Configuration @MapperScan(\u0026#34;top.fjy8018.graphsqldemo.mapper\u0026#34;) public class MybatisPlusConfig { /** * 分页插件配置 * * @return */ @Bean public MybatisPlusInterceptor mybatisPlusInterceptor() { MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor(); PaginationInnerInterceptor pageConfig = new PaginationInnerInterceptor(DbType.MYSQL); interceptor.addInnerInterceptor(pageConfig); return interceptor; } } Relay Pagination配置 type Query { actorList: [Actor] findOneActor(id : Int!): Actor filmList(current: Int!,size: Int!): FilmConnection findOneFilm(id : Int!): Film } type Actor { actorId: Int! firstName: String! lastName: String! lastUpdate: String } type Film @connection { filmId: Int! title: String! description: String releaseYear: Int languageId: Int! originalLanguageId: Int rentalDuration: Int rentalRate: Float length: Int replacementCost: Float rating: String specialFeatures: String lastUpdate: String actors: [Actor] } @connection和XXXConection已经由graphql-java底层框架自动生成，无需手动定义\n类型转换器 public interface ConnectionAssembler { /** * mybatis plus分页对象转为Connection * * @param page 分页对象 * @param \u0026lt;T\u0026gt; 泛型 * @return Connection */ static \u0026lt;T\u0026gt; Connection\u0026lt;T\u0026gt; convert(IPage\u0026lt;T\u0026gt; page) { AtomicInteger index = new AtomicInteger(); List\u0026lt;Edge\u0026lt;T\u0026gt;\u0026gt; defaultEdges = Stream.ofNullable(page.getRecords()) .flatMap(Collection::stream) .map(e -\u0026gt; (Edge\u0026lt;T\u0026gt;) new DefaultEdge\u0026lt;\u0026gt;(e, new DefaultConnectionCursor(String.valueOf(index.incrementAndGet())))) .toList(); return new DefaultConnection\u0026lt;\u0026gt;(defaultEdges, new DefaultPageInfo(new DefaultConnectionCursor(String.valueOf(page.getCurrent())), new DefaultConnectionCursor(String.valueOf(page.getPages())), page.getCurrent() \u0026gt; 1, page.getCurrent() \u0026lt; page.getPages())); } } Graphql解析配置 @DgsQuery public Connection\u0026lt;Film\u0026gt; filmList(@InputArgument(\u0026#34;current\u0026#34;) Integer current, @InputArgument(\u0026#34;size\u0026#34;) Integer size, DgsDataFetchingEnvironment dfe) { IPage\u0026lt;Film\u0026gt; page = new PageDTO\u0026lt;\u0026gt;(current, size); IPage\u0026lt;Film\u0026gt; pageResult = filmRepository.page(page); return ConnectionAssembler.convert(pageResult); } 将List查询改为返回Connection即可\n测试 访问http://localhost:8080/graphiql即可看到在线查询页面\n查看查询数量，确认是按照分页查询\n总结 到此，DSG核心功能尝试基本完成，基本实现常用的Rest功能，同时集成方便，而且官方文档清晰，但目前开源社区还没有太多的基于DSG的系统集成样例，还需要在实践中评估摸索。\nthoughtworks也从2016年开始关注graphql，目前给出的建议也是“评估”，可以进行小规模尝试并在合适的时候推广\nWe\u0026rsquo;ve seen many successful GraphQL implementations on our projects. We\u0026rsquo;ve seen some interesting patterns of use too, including GraphQL for server-side resource aggregation. That said, we\u0026rsquo;ve concerns about misuse of this framework and some of the problems that can occur. Examples include performance gotchas around N+1 queries and lots of boilerplate code needed when adding new models, leading to complexity. There are workarounds to these gotchas such as query caching. Even though it\u0026rsquo;s not a silver bullet, we still think it\u0026rsquo;s worth assessing as part of your architecture.\n","date":"2022-10-28T16:06:31+08:00","permalink":"/archives/280/","title":"GraphQL实践7——Netflix Dgs Graphql分页查询"},{"content":"N+1问题介绍 对于上一篇文章样例，如果要获取每个电影的演员名单，要执行如下动作\n查询所有电影清单 遍历N个电影，查询对应电影的演员名单 总查询开销为N+1次查询，代价非常大，效率低\n优化方案 DataLoader 对于一对一表关联的情况且每个关联对象只有一个值，可以直接使用BatchLoader\n但实际上，对于大部分表关联情况，通常为一对多或者多对多，不保证每个关联都有值，此时需要使用MappedBatchLoader维护关联关系\n维护多对多实体关系Map public interface ActorService { Map\u0026lt;Integer, List\u0026lt;Actor\u0026gt;\u0026gt; listByFilmId(Collection\u0026lt;Integer\u0026gt; filmIds); } @Slf4j @Service @RequiredArgsConstructor public class ActorServiceImpl implements ActorService { private final ActorRepository actorRepository; private final FilmActorRepository filmActorRepository; @Override public Map\u0026lt;Integer, List\u0026lt;Actor\u0026gt;\u0026gt; listByFilmId(Collection\u0026lt;Integer\u0026gt; filmIds) { List\u0026lt;FilmActor\u0026gt; filmActors = filmActorRepository.listByFilmId(filmIds); if (CollectionUtils.isEmpty(filmActors)) { return Collections.emptyMap(); } Map\u0026lt;Integer, List\u0026lt;FilmActor\u0026gt;\u0026gt; filmActorMap = filmActors.stream().collect(Collectors.groupingBy(FilmActor::getFilmId)); List\u0026lt;Integer\u0026gt; actorIds = filmActors.stream().map(FilmActor::getActorId).distinct().toList(); List\u0026lt;Actor\u0026gt; actors = actorRepository.listByIds(actorIds); Map\u0026lt;Integer, Actor\u0026gt; actorsMap = Stream.ofNullable(actors).flatMap(Collection::stream).collect(Collectors.toMap(Actor::getActorId, Function.identity(), (e1, e2) -\u0026gt; e1)); Map\u0026lt;Integer, List\u0026lt;Actor\u0026gt;\u0026gt; result = Maps.newHashMapWithExpectedSize(filmIds.size()); filmActorMap.forEach((k, v) -\u0026gt; { List\u0026lt;Actor\u0026gt; actorList = Stream.ofNullable(filmActorMap.get(k)).flatMap(Collection::stream).map(e -\u0026gt; actorsMap.get(e.getActorId())).toList(); result.put(k, actorList); }); return result; } } Dgs MappedBatchLoader 配置 @DgsDataLoader(name = \u0026#34;actors\u0026#34;) @RequiredArgsConstructor public class FilmActorsDataLoader implements MappedBatchLoader\u0026lt;Integer, List\u0026lt;Actor\u0026gt;\u0026gt; { private final ActorService actorService; @Override public CompletionStage\u0026lt;Map\u0026lt;Integer, List\u0026lt;Actor\u0026gt;\u0026gt;\u0026gt; load(Set\u0026lt;Integer\u0026gt; keys) { return CompletableFuture.supplyAsync(() -\u0026gt; actorService.listByFilmId(keys)); } } 测试 访问http://localhost:8080/graphiql即可看到在线查询页面\n此时执行嵌套查询只会查询2次，一次查电影列表，一次查所有电影的Actor列表，非常快速\n总结 使用BatchLoader实现对N+1问题优化，但还有一个潜在问题，即大数据分页，该样例中，有1000个电影，对应5000+演员，查询一次获取全量数据，对数据库压力较大，也不符合实际场景，还需要进行分页优化\n","date":"2022-10-28T16:05:51+08:00","permalink":"/archives/279/","title":"GraphQL实践6——Netflix Dgs Graphql N+1问题"},{"content":"问题背景 在graphql中，需要查询的字段由调用方传递，如果此时存在关联表才能获取的字段，但前端不需要该字段的时候，默认查询所有字段会带来较大开销\n例如\ntype Actor { actorId: Int! firstName: String! lastName: String! lastUpdate: String } type Film { filmId: Int! title: String! description: String actors: [Actor] } type Query { filmList: [Film] } 解决方案——懒加载 DGS框架中支持配置调用方传递了指定字段才读取对应的数据\n配置懒加载 增加实体定义，模拟嵌套实体\nMybatis Plus配置 扫描配置 @Configuration @MapperScan(\u0026#34;top.fjy8018.graphsqldemo.mapper\u0026#34;) public class MybatisPlusConfig { } 实体 表结构还是使用sakila样例数据库的表\n@Data public class Actor implements Serializable { @Serial private static final long serialVersionUID = 1L; /** * 联系人ID */ @TableId(type = IdType.AUTO) private Integer actorId; private String firstName; private String lastName; private Date lastUpdate; } @Data public class Film implements Serializable { @Serial private static final long serialVersionUID = 1L; /** * 联系人ID */ @TableId(type = IdType.AUTO) private Integer filmId; private String title; private String description; private Integer releaseYear; private Integer languageId; private Integer originalLanguageId; private Integer rentalDuration; private BigDecimal rentalRate; private Integer length; private BigDecimal replacementCost; private String rating; private String specialFeatures; private Date lastUpdate; } @Data public class FilmActor implements Serializable { @Serial private static final long serialVersionUID = 1L; /** * 联系人ID */ @TableId(type = IdType.AUTO) private Integer actorId; private Integer filmId; private Date lastUpdate; } Mapper public interface ActorMapper extends BaseMapper\u0026lt;Actor\u0026gt; {} public interface FilmActorMapper extends BaseMapper\u0026lt;FilmActor\u0026gt; {} public interface FilmMapper extends BaseMapper\u0026lt;Film\u0026gt; {} DAO public interface ActorRepository extends IService\u0026lt;Actor\u0026gt; { } public interface FilmActorRepository extends IService\u0026lt;FilmActor\u0026gt; { List\u0026lt;FilmActor\u0026gt; listByFilmId(@Size.List({@Size(min = 1, message = \u0026#34;至少要有一个元素\u0026#34;)}) List\u0026lt;Integer\u0026gt; filmIds); } public interface FilmRepository extends IService\u0026lt;Film\u0026gt; { } @Validated @Repository @RequiredArgsConstructor public class ActorRepositoryImpl extends ServiceImpl\u0026lt;ActorMapper, Actor\u0026gt; implements ActorRepository { } @Validated @Repository @RequiredArgsConstructor public class FilmActorRepositoryImpl extends ServiceImpl\u0026lt;FilmActorMapper, FilmActor\u0026gt; implements FilmActorRepository { @Override public List\u0026lt;FilmActor\u0026gt; listByFilmId(List\u0026lt;Integer\u0026gt; filmIds) { return lambdaQuery().in(FilmActor::getFilmId, filmIds).list(); } } @Validated @Repository @RequiredArgsConstructor public class FilmRepositoryImpl extends ServiceImpl\u0026lt;FilmMapper, Film\u0026gt; implements FilmRepository { } Service 多表查询聚合逻辑\npublic interface ActorService { List\u0026lt;Actor\u0026gt; listByFilmId(Integer filmId); } @Slf4j @Service @RequiredArgsConstructor public class ActorServiceImpl implements ActorService { private final ActorRepository actorRepository; private final FilmActorRepository filmActorRepository; @Override public List\u0026lt;Actor\u0026gt; listByFilmId(Integer filmId) { List\u0026lt;FilmActor\u0026gt; filmActors = filmActorRepository.listByFilmId(List.of(filmId)); if (CollectionUtils.isEmpty(filmActors)) { return Collections.emptyList(); } List\u0026lt;Integer\u0026gt; actorIds = filmActors.stream().map(FilmActor::getActorId).toList(); return actorRepository.listByIds(actorIds); } } Graphql 查询解析器 @DgsComponent @RequiredArgsConstructor public class FilmDataFetcher { private final FilmRepository filmRepository; private final ActorService actorService; @DgsQuery public Collection\u0026lt;Film\u0026gt; filmList() { return filmRepository.list(); } @DgsData(parentType = \u0026#34;Film\u0026#34;) public List\u0026lt;Actor\u0026gt; actors(DgsDataFetchingEnvironment dfe) { Film film = dfe.getSource(); return actorService.listByFilmId(film.getFilmId()); } @DgsQuery public Film findOneFilm(@InputArgument Integer id) { return filmRepository.getById(id); } } 增加DgsDataFetchingEnvironment入参，DGS会自动获取请求上下文参数\n此处定义即为传入了actors参数才执行该方法获取对应的Actor集合，实现懒加载\n测试 访问http://localhost:8080/graphiql即可看到在线查询页面\n不访问Actor 此时可以看到日志输出，没有查询Actor表\n访问Actor 此时DSG会执行N+1查询Actor表，性能损耗非常大\n总结 对于表关联，开销较大字段，都可以使用DSG懒加载实现性能优化，但此时还是存在N+1查询性能问题，将在后面使用DataLoader优化\n","date":"2022-10-28T16:04:43+08:00","permalink":"/archives/278/","title":"GraphQL实践5——Netflix Dgs Graphql懒加载"},{"content":"介绍 通过JPA或者Mybatis Plus集成Graphql框架已经很简便，但其实使用graphql-java集成过于底层，很多扩展内容实现并不方便，例如：\n分页查询 资源懒加载 N+1问题 Netflix 基于长期的Graphql实践，2020年开源了DGS Framework，让解决上述问题更加简单，大量开源系统集成Graphql也是通过集成DGS组件\nThe DGS framework project started at Netflix in 2019 as internal teams began developing multiple GraphQL services. As 2020 wrapped up, Netflix decided to open source the framework and build a community around it.\n集成Netflix DGS 官网：Home - DGS Framework (netflix.github.io)\n引入依赖 技术栈：\nSpringBoot 2.7.1 Mybatis Plus JDK 17 Dgs SpringBoot 本次和上一篇文章一样，选择原因不在赘述\n\u0026lt;dependencyManagement\u0026gt; \u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.netflix.graphql.dgs\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;graphql-dgs-platform-dependencies\u0026lt;/artifactId\u0026gt; \u0026lt;!-- The DGS BOM/platform dependency. This is the only place you set version of DGS --\u0026gt; \u0026lt;version\u0026gt;4.9.16\u0026lt;/version\u0026gt; \u0026lt;type\u0026gt;pom\u0026lt;/type\u0026gt; \u0026lt;scope\u0026gt;import\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; \u0026lt;/dependencyManagement\u0026gt; \u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-web\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-validation\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.netflix.graphql.dgs\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;graphql-dgs-spring-boot-starter\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.projectlombok\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;lombok\u0026lt;/artifactId\u0026gt; \u0026lt;optional\u0026gt;true\u0026lt;/optional\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;mysql\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;mysql-connector-java\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;runtime\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- mybatis-plus-boot-starter --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.baomidou\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;mybatis-plus-boot-starter\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;3.5.2\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-test\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;test\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; 配置文件 spring: datasource: driver-class-name: com.mysql.cj.jdbc.Driver username: root password: **** url: jdbc:mysql://localhost:3306/sakila?useSSL=false\u0026amp;useUnicode=true\u0026amp;characterEncoding=UTF-8\u0026amp;serverTimezone=Asia/Shanghai server: port: 8080 logging: level: top.fjy8018.graphsqldemo.mapper: DEBUG dgs: graphql: schema-locations: - classpath*:graphql/*.graphqls 由于DGS默认扫描路径为classpath*:schema/**/*.graphql*，本次基于上一篇文章的工程配置，需要手动指定一下扫描文件目录\nMybatis Plus配置 扫描配置 @Configuration @MapperScan(\u0026#34;top.fjy8018.graphsqldemo.mapper\u0026#34;) public class MybatisPlusConfig { } 实体 表结构还是使用sakila样例数据库的表\n@Data public class Actor implements Serializable { @Serial private static final long serialVersionUID = 1L; /** * 联系人ID */ @TableId(type = IdType.AUTO) private Integer actorId; private String firstName; private String lastName; private Date lastUpdate; } Mapper public interface ActorMapper extends BaseMapper\u0026lt;Actor\u0026gt; { } DAO 从分层模型定义上，MP自带的CRUD接口本质上还是一个DAO，故定义为Repository更加合适，不与业务逻辑Serice冲突\n同时也建议QueryWrapper仅在对应的Repository使用，防止查询条件散落在各处\npublic interface ActorRepository extends IService\u0026lt;Actor\u0026gt; { } @Validated @Repository @RequiredArgsConstructor public class ActorRepositoryImpl extends ServiceImpl\u0026lt;ActorMapper, Actor\u0026gt; implements ActorRepository { } Graphql 查询解析器 @DgsComponent @RequiredArgsConstructor public class ActorDataFetcher { private final ActorRepository actorRepository; @DgsQuery public Collection\u0026lt;Actor\u0026gt; actorList() { return actorRepository.list(); } @DgsQuery public Actor findOneActor(@InputArgument Integer id) { return actorRepository.getById(id); } } type Query { actorList: [Actor] findOneActor(id : Int!): Actor } type Actor { actorId: Int! firstName: String! lastName: String! lastUpdate: String } 此处集成和graphql-java-kickstart类似，也是十分简便，不指定别名默认将方法名映射到文件方法名\n测试 访问http://localhost:8080/graphiql即可看到在线查询页面\n总结 此处还看不出DGS与其他graphql-java组件的区别，都是集成简便，可以和任何数据源、ORM框架集成，后续将会逐步实践DGS特性，包括懒加载、N+1问题、分页组件等\n","date":"2022-10-28T16:03:00+08:00","permalink":"/archives/277/","title":"GraphQL实践4——Netflix Dgs Graphql集成Mybatis Plus"},{"content":"介绍 考虑到国内大部分场景对接的是Mybatis或者Mybatis Plus生态，而不是JPA，同时Graphql版本也进一步迭代，故计划使用Mybatis Plus对接Graphql\n集成过程 引入依赖 由于JDK已经发展到19，目前最新的LTS为17，我所在的公司也将生产升级到了JDK 17，未来更多新应用会直接基于17构建，故此处直接基于JDK17对接 SpringBoot版本为2.7.1，目前已知3.0.0 RC1还不兼容第三方graphql 此处不使用spring官方的starter，改为使用graphql-java-kickstart的starter，编写更加简单 \u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-web\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.projectlombok\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;lombok\u0026lt;/artifactId\u0026gt; \u0026lt;optional\u0026gt;true\u0026lt;/optional\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- graphql-spring-boot-starter --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.graphql-java-kickstart\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;graphql-spring-boot-starter\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;12.0.0\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;mysql\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;mysql-connector-java\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;runtime\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- mybatis-plus-boot-starter --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.baomidou\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;mybatis-plus-boot-starter\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;3.5.2\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- querydsl-core --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.querydsl\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;querydsl-core\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-test\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;test\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; 配置文件 spring: datasource: driver-class-name: com.mysql.cj.jdbc.Driver username: root password: **** url: jdbc:mysql://localhost:3306/sakila?useSSL=false\u0026amp;useUnicode=true\u0026amp;characterEncoding=UTF-8\u0026amp;serverTimezone=Asia/Shanghai server: port: 8080 graphql: tools: schema-location-pattern: graphql/*.graphqls graphiql: # 开启可交互页面，用于调试 enabled: true logging: level: top.fjy8018.graphsqldemo.mapper: DEBUG Mybatis Plus配置 扫描配置 @Configuration @MapperScan(\u0026#34;top.fjy8018.graphsqldemo.mapper\u0026#34;) public class MybatisPlusConfig { } 实体 @Data public class Actor implements Serializable { @Serial private static final long serialVersionUID = 1L; /** * 联系人ID */ @TableId(type = IdType.AUTO) private Integer actorId; private String firstName; private String lastName; private Date lastUpdate; } Mapper public interface ActorMapper extends BaseMapper\u0026lt;Actor\u0026gt; { } DAO public interface ActorRepository extends IService\u0026lt;Actor\u0026gt; { } @Validated @Repository @RequiredArgsConstructor public class ActorRepositoryImpl extends ServiceImpl\u0026lt;ActorMapper, Actor\u0026gt; implements ActorRepository { } Graphql 查询解析器 @Component @AllArgsConstructor public class ActorGraphQLQueryResolver implements GraphQLQueryResolver { private final ActorRepository actorRepository; public Collection\u0026lt;Actor\u0026gt; actorList() { return actorRepository.list(); } public Actor findOneActor(Integer id) { return actorRepository.getById(id); } } 查询文件定义在resources/graphql/actor.graphqls\ntype Query { actorList: [ActorEntity] findOneActor(id : ID!): ActorEntity } type ActorEntity { actorId: ID! firstName: String! lastName: String! lastUpdate: String } 测试 访问http://localhost:8080/graphiql即可看到在线查询页面\n总结 依托Mybatis Plus强大的自带CRUD方法，可以实现和JPA类似的快速对接\n","date":"2022-10-27T15:37:00+08:00","permalink":"/archives/276/","title":"GraphQL实践3——Spring-Graphql集成Mybatis Plus"},{"content":"背景 版本发布 2021年3月16日 Java 17 作为继Java 11的下一个LTS版本正式发布，JDK 17 作为下一代 LTS 将提供至少到 2026 年的支持。\n同业动态 2022年ThoughtWorks 26期技术雷达也提到，建议各个组织将Java 17纳入评估。\n评估 82. Java 17\n我们通常不会专门介绍某一语言的新版本，但我们还是想关注一下 Java 新的长期支持（LTS）版本——Java 17。尽管出现了前景不错的新特性，比如模式匹配等预览特性，但其实向新 LTS 版本升级的方案更应该吸引各个组织的兴趣。我们建议各个组织在 Java 有新的发布时对其进行评估，确保能够恰当地适配这些新特性和版本。尽管定期更新有利于简化开发并且方便管理，但许多组织却意外地并不经常更新语言的版本。希望通过 LTS 版本的升级以及开发团队对语言的定期更新，能够使生产软件免于因为“更新成本太高”而一直困在 Java 的过时版本上。\n框架发展情况 Spring Framework 6和SpringBoot 3.0 将在2022年底正式发布，将基于Java 17构建。\nSpring Boot 3.0.0 M1 Release Notes · spring-projects/spring-boot Wiki (github.com)\nMinimum Requirements Changes Spring Boot 3.0 makes the following changes to its minimum supported versions:\nGradle 7.3 Jakarta EE 9 Java 17 Kotlin 1.6 Spring Framework 6 其他团队落地情况 在此之前，我所在的团队使用的Java版本一直停留在Java8，今年经过其他开发团队的尝试，我们团队在4月份也正式决定在生产环境使用Java 17。\n实践情况 开发组件版本 本次计划在内部运营管理系统重构中使用Java 17作为基础版本，其他核心组件版本如下\nSpringBoot 2.6.6 mapstruct 1.4.2.Final mybatis-plus 3.5.1 jdk 17.0.1 开发时SpringBoot 3.0还处于M2里程碑阶段，暂时不适合带入生产环境，等未来3.0 GA后再考虑升级\n基础镜像 原本使用的基于jdk8 的Docker image也需要替换成基于jdk 17的image\n带来的问题：\n原本基于jdk8构建并注入到镜像中的opentelemetry-javaagent暂时不可用，等待后续基于jdk17 构建后才能重新启用\nJVM参数 由于从Java11开始G1替代CMS作为默认的垃圾收集器，Java14开始移除了CMS GC。\n故团队内定义的系统框架配置的GC调优参数基本废弃，并且可以使用Java 14新引入的JVM参数，例如-XX:+ShowCodeDetailsInExceptionMessages\n开启前 Exception in thread \u0026#34;main\u0026#34; java.lang.NullPointerException at Book.main(Book.java:5) 开启后 Exception in thread \u0026#34;main\u0026#34; java.lang.NullPointerException: Cannot assign field \u0026#34;book\u0026#34; because \u0026#34;shoopingcart.buy\u0026#34; is null at Book.main(Book.java:5) 语法特性 从Java8到Java17期间新增的语法特性很多，此处提及几个比较常用的\n快速创建不可变集合 不可变对象具有许多优点，包括：\n可安全使用不受信任的库。 线程安全。 不需要支持变化判断，所有不可变的集合实现都比其可变的同级实现更节省内存。 Java9开始新增List、Map、Set等of方法快速创建不可变集合\nList.of(prodStrategy, instStrategy) 实现源码\nstatic \u0026lt;E\u0026gt; List\u0026lt;E\u0026gt; of(E e1, E e2) { return new ImmutableCollections.List12\u0026lt;\u0026gt;(e1, e2); } Java16开始，Stream类中也新增toList()方法，快速将流转成不可变集合\nList\u0026lt;Long\u0026gt; ids = idList.stream().map(ContactId::id).toList(); 实现源码\ndefault List\u0026lt;T\u0026gt; toList() { return (List\u0026lt;T\u0026gt;) Collections.unmodifiableList(new ArrayList\u0026lt;\u0026gt;(Arrays.asList(this.toArray()))); } 注意：如果需要将流转成可变集合，还是需要使用collect(Collectors.toList())方法\nrecord类型 Java14开始新增record类型，用于表示不可变class，实践中可以方便的表示值对象，无需使用@Value注解\npublic record ContactId(@NotNull(message = \u0026#34;ID不能为空\u0026#34;) Long id) { } instanceof模式匹配 Java14开始支持模式匹配，无需做强制类型转换\nprivate void configMessageConverters(List\u0026lt;HttpMessageConverter\u0026lt;?\u0026gt;\u0026gt; converters) { converters.removeIf(x -\u0026gt; x instanceof StringHttpMessageConverter || x instanceof MappingJackson2HttpMessageConverter); converters.add(new StringHttpMessageConverter(Charsets.UTF_8)); converters.add(new MappingJackson2HttpMessageConverter(objectMapper)); } 创建类型安全的流 Java9 开始，新增ofNullable方法创建类型安全的流对象，避免在调用stream方法时报NPE\n// query.getDescColumns()可能为null List\u0026lt;OrderItem\u0026gt; descItems = Stream.ofNullable(query.getDescColumns()) .flatMap(Collection::stream) .map(OrderItem::desc) .toList(); 性能和安全提升 从同一段功能逻辑中测试发现，Java8和Java17在并发场景下性能差距达到2倍以上\n并且同样的逻辑（使用了并发安全的工具）在并发量达到50000的时候在Java8中依然有线程安全问题，而Java17中不会有线程安全问题\n外部测评结果\n从规划调度引擎 OptaPlanner 项目（原文 (opens new window)）对 JDK 17和 JDK 11 的性能基准测试进行了对比来看：\n对于 G1GC（默认），Java 17 比 Java 11 快 8.66%； 对于 ParallelGC，Java 17 比 Java 11 快 6.54%； Parallel GC 整体比 G1 GC 快 16.39% 简而言之，JDK17 更快，高吞吐量垃圾回收器比低延迟垃圾回收器更快。\n","date":"2022-05-06T11:04:00+08:00","permalink":"/archives/284/","title":"Java 17 生产实践"},{"content":"介绍 上一篇介绍到用第三方库集成GraphQL，目前spring-graphql项目已经出到1.0.0-M3版本，属于内部预览版，此处尝鲜验证\n相关文章\n集成过程 引入依赖 由于SpringBoot 2.6.0还未发布，因此需要引入较多依赖\n\u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-web\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.projectlombok\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;lombok\u0026lt;/artifactId\u0026gt; \u0026lt;optional\u0026gt;true\u0026lt;/optional\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.experimental\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;graphql-spring-boot-starter\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.0.0-M3\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!--升级到Spring Boot 2.6后可以移除--\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.data\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-data-commons\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;2.6.0-RC1\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.data\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-data-jpa\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;2.6.0-RC1\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- https://mvnrepository.com/artifact/org.hibernate.javax.persistence/hibernate-jpa-2.1-api --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.hibernate.javax.persistence\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;hibernate-jpa-2.1-api\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.0.2.Final\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- https://mvnrepository.com/artifact/com.querydsl/querydsl-jpa --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.querydsl\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;querydsl-jpa\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;4.4.0\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- https://mvnrepository.com/artifact/com.querydsl/querydsl-apt --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.querydsl\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;querydsl-apt\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;4.4.0\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- https://mvnrepository.com/artifact/javax.annotation/javax.annotation-api --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;javax.annotation\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;javax.annotation-api\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.querydsl\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;querydsl-core\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.querydsl\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;querydsl-jpa\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;mysql\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;mysql-connector-java\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;runtime\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-data-jpa\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-test\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;test\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; 引入里程碑仓库 由于还没有GA，所以要引入里程碑仓库\n\u0026lt;repositories\u0026gt; \u0026lt;repository\u0026gt; \u0026lt;id\u0026gt;spring-milestones\u0026lt;/id\u0026gt; \u0026lt;name\u0026gt;Spring Milestones\u0026lt;/name\u0026gt; \u0026lt;url\u0026gt;https://repo.spring.io/milestone\u0026lt;/url\u0026gt; \u0026lt;/repository\u0026gt; \u0026lt;repository\u0026gt; \u0026lt;id\u0026gt;spring-snapshots\u0026lt;/id\u0026gt; \u0026lt;name\u0026gt;Spring Snapshots\u0026lt;/name\u0026gt; \u0026lt;url\u0026gt;https://repo.spring.io/snapshot\u0026lt;/url\u0026gt; \u0026lt;snapshots\u0026gt; \u0026lt;enabled\u0026gt;true\u0026lt;/enabled\u0026gt; \u0026lt;/snapshots\u0026gt; \u0026lt;/repository\u0026gt; \u0026lt;/repositories\u0026gt; 引入插件 由于使用了dsl动态生成，还要引入注解处理器\n\u0026lt;build\u0026gt; \u0026lt;plugins\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-maven-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;excludes\u0026gt; \u0026lt;exclude\u0026gt; \u0026lt;groupId\u0026gt;org.projectlombok\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;lombok\u0026lt;/artifactId\u0026gt; \u0026lt;/exclude\u0026gt; \u0026lt;/excludes\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;/plugin\u0026gt; \u0026lt;!-- added for QueryDsl--\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;com.mysema.maven\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;apt-maven-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.1.3\u0026lt;/version\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;process\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;outputDirectory\u0026gt;target/generated-sources/java\u0026lt;/outputDirectory\u0026gt; \u0026lt;processor\u0026gt;com.querydsl.apt.jpa.JPAAnnotationProcessor\u0026lt;/processor\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; \u0026lt;/plugins\u0026gt; \u0026lt;/build\u0026gt; SpringBoot 配置 编写配置文件\nspring: datasource: driver-class-name: com.mysql.cj.jdbc.Driver username: root password: \u0026#34;****\u0026#34; url: jdbc:mysql://localhost:3306/sakila?useSSL=false\u0026amp;useUnicode=true\u0026amp;characterEncoding=UTF-8\u0026amp;serverTimezone=Asia/Shanghai jpa: hibernate: ddl-auto: none show-sql: true database-platform: org.hibernate.dialect.MySQL55Dialect properties: hibernate: format_sql: true use_sql_comments: true graphql: path: /graphql schema: locations: classpath:graphql/ fileExtensions: .graphqls, .gqls printer: enabled: true server: port: 8080 实体类 package top.fjy8018.graphsqldemo.entity; import lombok.Data; import javax.persistence.*; import java.sql.Timestamp; import java.util.Objects; /** * 演员表实体类 * * @author F嘉阳 * @date 2021/11/5 10:34 */ @Data @Entity @Table(name = \u0026#34;actor\u0026#34;, schema = \u0026#34;sakila\u0026#34;) public class ActorEntity { @Id @Column(name = \u0026#34;actor_id\u0026#34;, nullable = false) private Integer actorId; @Column(name = \u0026#34;first_name\u0026#34;, nullable = false, length = 45) private String firstName; @Column(name = \u0026#34;last_name\u0026#34;, nullable = false, length = 45) private String lastName; @Column(name = \u0026#34;last_update\u0026#34;, nullable = false) private Date lastUpdate; } DAO DAO直接继承dsl相关处理器，可自动配对增删改查方法，相比第三方库更加简便\npackage top.fjy8018.graphsqldemo.repository; import org.springframework.data.querydsl.QuerydslPredicateExecutor; import org.springframework.data.repository.CrudRepository; import org.springframework.graphql.data.GraphQlRepository; import top.fjy8018.graphsqldemo.entity.ActorEntity; /** * 演员表DAO * * @author F嘉阳 * @date 2021/11/5 10:35 */ @GraphQlRepository public interface ActorEntityRepository extends CrudRepository\u0026lt;ActorEntity, Integer\u0026gt;, QuerydslPredicateExecutor\u0026lt;ActorEntity\u0026gt; { } Graphql资源定义 在resources/graphql下定义接口和实体文件\n接口声明和实体定义schema.graphqls\ntype Query { findOneActor(actorId : ID!): ActorEntity actorList: [ActorEntity] } type ActorEntity { actorId: ID! firstName: String! lastName: String! lastUpdate: String } 接口定义acotrEntity.graphql\nquery findOneActor($id: ID!) { findOneActor(actorId: $id) { actorId firstName lastName lastUpdate } } query actorList { actorList{ actorId } } 启动测试 Spring-Graphql默认不包含可视化界面，此处使用postman进行测试\n唯一查询 列表查询 总结 使用Spring官方组件好处在于和Spring生态集成度很高，如果本身就采用Jpa方式进行业务开发，迁移更加方便，需要开发的代码也很少。\n样例源码地址 FJiayang/graphql-demo\n","date":"2021-11-15T15:38:00+08:00","permalink":"/archives/275/","title":"GraphQL实践2——Spring-GraphQL集成JPA与MySQL"},{"content":"介绍 官方定义：\nGraphQL 既是一种用于 API 的查询语言也是一个满足你数据查询的运行时。 GraphQL 对你的 API 中的数据提供了一套易于理解的完整描述，使得客户端能够准确地获得它需要的数据，而且没有任何冗余，也让 API 更容易地随着时间推移而演进，还能用于构建强大的开发者工具。\n中文官网：\nGraphQL — A query language for your API\n集成方案选择 GraphQL经过近几年的发展，SpringBoot官方、第三方都提供了starter\nSpringBoot官方提供的starter\nSpring GraphQL\n第三方提供的starter\ngraphql-spring-boot\n两者活跃度都还不错，但考虑到SpringBoot官方的依赖库还未GA，而且使用起来与Spring家族耦合过大，最终决定采用第三方的starter\n集成过程 数据库配置 此处采用MySQL数据库，数据内容采用MySQL官方样例sakila数据，表结构和数据地址如下\nsakila database下载地址\n导入后如下\n工程配置 引入最新的SpringBoot框架，依赖如下\n\u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-web\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.projectlombok\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;lombok\u0026lt;/artifactId\u0026gt; \u0026lt;optional\u0026gt;true\u0026lt;/optional\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- https://mvnrepository.com/artifact/com.graphql-java-kickstart/graphql-spring-boot-starter --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;com.graphql-java-kickstart\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;graphql-spring-boot-starter\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;12.0.0\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;mysql\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;mysql-connector-java\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;runtime\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-data-jpa\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-test\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;test\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; 增加数据库配置\nspring: datasource: driver-class-name: com.mysql.cj.jdbc.Driver username: root password: \u0026#34;****\u0026#34; url: jdbc:mysql://localhost:3306/sakila?useSSL=false\u0026amp;useUnicode=true\u0026amp;characterEncoding=UTF-8\u0026amp;serverTimezone=Asia/Shanghai jpa: hibernate: ddl-auto: update show-sql: true database-platform: org.hibernate.dialect.MySQL55Dialect properties: hibernate: format_sql: true use_sql_comments: true server: port: 8080 graphql: tools: schema-location-pattern: graphql/*.graphqls graphiql: # 开启可交互页面，用于调试 enabled: true 实体映射 新增实体映射，此处只映射Actor表作为样例\npackage top.fjy8018.graphsqldemo.entity; import lombok.Data; import javax.persistence.*; import java.sql.Timestamp; import java.util.Objects; /** * 演员表实体映射 * * @author F嘉阳 * @date 2021/11/5 10:34 */ @Data @Entity @Table(name = \u0026#34;actor\u0026#34;, schema = \u0026#34;sakila\u0026#34;) public class ActorEntity { @Id @Column(name = \u0026#34;actor_id\u0026#34;, nullable = false) private Integer actorId; @Column(name = \u0026#34;first_name\u0026#34;, nullable = false, length = 45) private String firstName; @Column(name = \u0026#34;last_name\u0026#34;, nullable = false, length = 45) private String lastName; @Column(name = \u0026#34;last_update\u0026#34;, nullable = false) private Timestamp lastUpdate; } 新增DAO package top.fjy8018.graphsqldemo.repository; import org.springframework.data.jpa.repository.JpaRepository; import top.fjy8018.graphsqldemo.entity.ActorEntity; /** * 演员表DAO * * @author F嘉阳 * @date 2021/11/5 10:35 */ public interface ActorEntityRepository extends JpaRepository\u0026lt;ActorEntity, Integer\u0026gt; { } 新增查询接口配置 在resources/graphql下新增文件actor.graphqls\ntype Query { actorList: [ActorEntity] findOneActor(id : ID!): ActorEntity } type ActorEntity { actorId: ID! firstName: String! lastName: String! lastUpdate: String } 新增查询解析器\npackage top.fjy8018.graphsqldemo.resolver; import graphql.kickstart.tools.GraphQLQueryResolver; import lombok.AllArgsConstructor; import org.springframework.stereotype.Component; import top.fjy8018.graphsqldemo.entity.ActorEntity; import top.fjy8018.graphsqldemo.repository.ActorEntityRepository; import java.util.Collection; /** * 演员表GraphQL查询解析器 * * @author F嘉阳 * @date 2021/11/5 16:27 */ @Component @AllArgsConstructor public class ActorGraphQLQueryResolver implements GraphQLQueryResolver { private final ActorEntityRepository actorRepository; public Collection\u0026lt;ActorEntity\u0026gt; actorList() { return actorRepository.findAll(); } public ActorEntity findOneActor(Integer id) { return actorRepository.findById(id).orElse(null); } } 启动验证 启动工程后，浏览器打开http://localhost:8080/graphiql\n列表查询 由于采用交互式页面，输入也会有语法提示，比较方便\n唯一查询 优缺点分析 优点 接口定义清晰，配置即接口 前端按需获取字段，减少数据传输量 接口整合、演进更加便利 缺点 减少字段个数对后端实现没有变化，并不能提升数据库性能 样例源码地址 FJiayang/graphql-demo\n","date":"2021-11-05T17:41:00+08:00","permalink":"/archives/274/","title":"GraphQL实践1——集成JPA与MySQL"},{"content":"创建企业空间 创建一个企业空间和一个帐户 (project-admin)，必须邀请该帐户至该企业空间并赋予 workspace-self-provisioner 角色。\n进入企业空间，切换到DevOps工程，点击创建\n创建工程\n安装helm $ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash 找不到环境变量报错 安装helm遇到报错\n$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash % Total % Received % Xferd Average Speed Time Time Time Current unset -f pathmunge Dload Upload Total Spent Left Speed 100 11248 100 11248 0 0 61543 0 --:--:-- --:--:-- --:--:-- 61802 Helm v3.6.0 is available. Changing from version v3.2.1. Downloading https://get.helm.sh/helm-v3.6.0-linux-amd64.tar.gz Verifying checksum... Done. Preparing to install helm into /usr/local/bin helm installed into /usr/local/bin/helm helm not found. Is /usr/local/bin on your $PATH? Failed to install helm For support, go to https://github.com/helm/helm. 则需要配置环境变量\n$ vi /etc/profile # 添加配置 export PATH=/usr/local/bin:$PATH $ source /etc/profile 验证\n$ helm version version.BuildInfo{Version:\u0026#34;v3.6.0\u0026#34;, GitCommit:\u0026#34;7f2df6467771a75f5646b7f12afb408590ed1755\u0026#34;, GitTreeState:\u0026#34;clean\u0026#34;, GoVersion:\u0026#34;go1.16.3\u0026#34;} 安装SonarQube 执行以下命令安装 SonarQube 服务器\n$ helm upgrade --install sonarqube sonarqube --repo https://charts.kubesphere.io/main -n kubesphere-devops-system --create-namespace --set service.type=NodePort 输出\n执行以下命令以获取 SonarQube NodePort\n$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath=\u0026#34;{.spec.ports[0].nodePort}\u0026#34; services sonarqube-sonarqube) $ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath=\u0026#34;{.items[0].status.addresses[0].address}\u0026#34;) $ echo http://$NODE_IP:$NODE_PORT 输出\n此时可在kubesphere控制台中看到sonar服务\n配置 SonarQube 服务器 访问地址，点击右上角的 Log in，然后使用默认帐户 admin/admin 登录。\n创建 SonarQube 管理员令牌 (Token) 点击右上角字母 A，然后从菜单中选择 My Account 以转到 Profile 页面。\n生成token\n创建 Webhook 服务器 执行以下命令获取 SonarQube Webhook 的地址。\n$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath=\u0026#34;{.spec.ports[0].nodePort}\u0026#34; services ks-jenkins) $ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath=\u0026#34;{.items[0].status.addresses[0].address}\u0026#34;) $ echo http://$NODE_IP:$NODE_PORT/sonarqube-webhook/ 输出\n依次点击 Administration、Configuration 和 Webhooks 创建一个 Webhook。\n点击 Create\n由于webhook都通过内部网络调用，故可以使用内部网络地址\n将 SonarQube 配置添加到 ks-installer 执行以下命令编辑 ks-installer\n$ kubectl edit cc -n kubesphere-system ks-installer 搜寻至 devops。添加字段 sonarqube 并在其下方指定 externalSonarUrl 和 externalSonarToken。\ndevops: enabled: true jenkinsJavaOpts_MaxRAM: 2g jenkinsJavaOpts_Xms: 512m jenkinsJavaOpts_Xmx: 512m jenkinsMemoryLim: 2Gi jenkinsMemoryReq: 1500Mi jenkinsVolumeSize: 8Gi sonarqube: externalSonarUrl: http://10.170.0.5:32108 externalSonarToken: c6a6e2e15a9d52e30b744df0eef4c639bbea88a1 完成操作后保存此文件。\n将 SonarQube 服务器添加至 Jenkins 执行以下命令获取 Jenkins 的地址。\n$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath=\u0026#34;{.spec.ports[0].nodePort}\u0026#34; services ks-jenkins) $ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath=\u0026#34;{.items[0].status.addresses[0].address}\u0026#34;) $ echo http://$NODE_IP:$NODE_PORT 输出\n使用地址 http://{$Public IP}:30180 访问 Jenkins。安装 KubeSphere 时，默认情况下也会安装 Jenkins 仪表板。此外，Jenkins 还配置有 KubeSphere LDAP，这意味着您可以直接使用 KubeSphere 帐户（例如 admin/P@88w0rd）登录 Jenkins。\n使用kubesphere admin账户登录\n前往系统管理下的 Manage Credentials 并点击 Stores scoped to Jenkins 下的 Jenkins，再点击全局凭据 (unrestricted)\n点击左侧导航栏的添加凭据，用 SonarQube 管理员令牌添加凭证\n点击左侧的系统管理，向下翻页找到并点击系统配置。\n搜寻到 SonarQube servers，然后点击 Add SonarQube。\n添加sonar服务器地址和名称，点击应用完成操作。\n将 sonarqubeURL 添加到 KubeSphere 控制台 指定 sonarqubeURL，以便可以直接从 KubeSphere 控制台访问 SonarQube。\n执行以下命令：\n$ kubectl edit cm -n kubesphere-system ks-console-config 搜寻到 client，添加 devops 字段并指定 sonarqubeURL。\nclient: version: kubesphere: v3.1.0 kubernetes: v1.20.4 openpitrix: v0.3.5 enableKubeConfig: true devops: sonarqubeURL: http://34.92.90.181:32108 结果\n重启服务 执行以下命令。\n$ kubectl -n kubesphere-system rollout restart deploy ks-apiserver $ kubectl -n kubesphere-system rollout restart deploy ks-console 为新工程创建 SonarQube Token 创建一个 SonarQube 令牌，以便您的流水线可以在运行时与 SonarQube 通信。\n在 SonarQube 控制台上，点击 Create new project。\n设置工程名称\n生成token\n创建令牌后，点击 Continue。\n分别选择 Java 和 Maven。复制下图所示绿色框中的序列号，如果要在流水线中使用，则需要在凭证中添加此序列号。\n创建流水线工程 创建凭证 登录 KubeSphere 控制台。转到 DevOps 工程，在工程管理下的凭证页面创建以下凭证。\n凭证 ID 类型 用途 dockerhub-id 帐户凭证 Docker Hub demo-kubeconfig kubeconfig Kubernetes demo-kubeconfig\ndockerhub-id\n创建一个凭证 ID (sonar-token)\n列表中看到已创建的三个凭证。\n创建流水线 在流水线页面点击创建，为了使用图形化流水线搭建，不在此处引用代码仓库，直接点击下一步\n保持默认配置，点击创建\n创建成功\n进入流水线，点击编辑\n此时选择CICD模板\n可视化编辑流水线\n编辑后运行\n流水线支持交互式操作\n在 KubeSphere 控制台查看结果 流水线执行成功后，即可查看到代码质量标签\nsonar控制台也可以查看对应结果\njenkins视图\nblue ocean\n","date":"2021-06-19T22:16:00+08:00","permalink":"/archives/273/","title":"Kubesphere落地DevOps"},{"content":"平台准备 创建工作空间\n启用计算引擎API\n准备虚拟机 作为演示环境，此处使用Google Cloud Platform创建3台虚拟机，配置如下\nCPU 内存 磁盘 4 16G 40G 4 16G 40G 4 16G 40G 创建实例模板 创建需要的实例模板\n配置操作系统、资源等，注意放行防火墙\n创建实例 基于实例模板创建3台虚拟机，选择地区为香港\n此处用于演示用途，可选择无状态机器，机器删除后不会保留磁盘内容\n注意关闭自动扩缩容\n等待实例创建\n由于资源调度问题，最终只成功分配两台虚拟机，不过对于演示环境安装过程一样。\n配置连接秘钥 添加本地生成好的rsa秘钥即可\n本地SSH工具配置私钥即可连接\n前置环境准备 kubesphere建议前置项 节点要求 所有节点必须都能通过 SSH 访问。 所有节点时间同步。 所有节点都应使用 sudo/curl/openssl。 依赖项要求 KubeKey 可以一同安装 Kubernetes 和 KubeSphere。根据要安装的 Kubernetes 版本，需要安装的依赖项可能会不同。您可以参考下表，查看是否需要提前在节点上安装相关依赖项。\n依赖项 Kubernetes 版本 ≥ 1.18 Kubernetes 版本 \u0026lt; 1.18 socat 必须 可选，但建议安装 conntrack 必须 可选，但建议安装 ebtables 可选，但建议安装 可选，但建议安装 ipset 可选，但建议安装 可选，但建议安装 以下内容所有节点都需要执行\n$ yum install -y socat curl openssl conntrack ebtables ipset 主节点下载kubekey，由于使用google虚拟机，能正常访问google和github，故直接下载即可\n$ curl -sfL https://get-kk.kubesphere.io | VERSION=v1.1.0 sh - $ chmod +x kk 放行VPC防火墙\n创建集群 服务器免密登陆 主节点生成免密登陆配置\n$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory \u0026#39;/root/.ssh\u0026#39;. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:yxln4RFEXhsvIMcL0/EzF8CLUpo/hMlkM8KBDcNEjbU root@kubesphere-instance-group-1-bx5d The key\u0026#39;s randomart image is: +---[RSA 2048]----+ | +=Oo..=Bo+.. | | +.=.B++=.+ . | | E= @+o*.o | | B.+o.= | | S++ | | . *o | | + . | | | | | +----[SHA256]-----+ 所有主机允许公钥免密登陆\n$ vi /etc/ssh/sshd_config # 将下面配置改为yes并去掉注释 PubkeyAuthentication yes PermitRootLogin yes $ service sshd restart 控制台添加主节点公钥，则所有新建主机都能被主节点访问\n创建示例配置文件 $./kk create config --with-kubernetes v1.20.4 --with-kubesphere v3.1.0 -f gcp-config 按默认配置只安装控制台和基础服务\napiVersion: kubekey.kubesphere.io/v1alpha1 kind: Cluster metadata: name: sample spec: hosts: - {name: node1, address: 34.92.90.181, internalAddress: 10.170.0.5, privateKeyPath: \u0026#34;/root/.ssh/id_rsa\u0026#34;} - {name: node2, address: 35.220.138.230, internalAddress: 10.170.0.6, privateKeyPath: \u0026#34;/root/.ssh/id_rsa\u0026#34;} roleGroups: etcd: - node1 master: - node1 worker: - node1 - node2 controlPlaneEndpoint: domain: lb.kubesphere.local address: \u0026#34;\u0026#34; port: 6443 kubernetes: version: v1.20.4 imageRepo: kubesphere clusterName: cluster.local network: plugin: calico kubePodsCIDR: 10.233.64.0/18 kubeServiceCIDR: 10.233.0.0/18 registry: registryMirrors: [] insecureRegistries: [] addons: [] --- apiVersion: installer.kubesphere.io/v1alpha1 kind: ClusterConfiguration metadata: name: ks-installer namespace: kubesphere-system labels: version: v3.1.0 spec: persistence: storageClass: \u0026#34;\u0026#34; authentication: jwtSecret: \u0026#34;\u0026#34; zone: \u0026#34;\u0026#34; local_registry: \u0026#34;\u0026#34; etcd: monitoring: false endpointIps: localhost port: 2379 tlsEnable: true common: redis: enabled: false redisVolumSize: 2Gi openldap: enabled: false openldapVolumeSize: 2Gi minioVolumeSize: 20Gi monitoring: endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090 es: elasticsearchMasterVolumeSize: 4Gi elasticsearchDataVolumeSize: 20Gi logMaxAge: 7 elkPrefix: logstash basicAuth: enabled: false username: \u0026#34;\u0026#34; password: \u0026#34;\u0026#34; externalElasticsearchUrl: \u0026#34;\u0026#34; externalElasticsearchPort: \u0026#34;\u0026#34; console: enableMultiLogin: true port: 30880 alerting: enabled: false # thanosruler: # replicas: 1 # resources: {} auditing: enabled: false devops: enabled: false jenkinsMemoryLim: 2Gi jenkinsMemoryReq: 1500Mi jenkinsVolumeSize: 8Gi jenkinsJavaOpts_Xms: 512m jenkinsJavaOpts_Xmx: 512m jenkinsJavaOpts_MaxRAM: 2g events: enabled: false ruler: enabled: true replicas: 2 logging: enabled: false logsidecar: enabled: true replicas: 2 metrics_server: enabled: false monitoring: storageClass: \u0026#34;\u0026#34; prometheusMemoryRequest: 400Mi prometheusVolumeSize: 20Gi multicluster: clusterRole: none network: networkpolicy: enabled: false ippool: type: none topology: type: none notification: enabled: false openpitrix: store: enabled: false servicemesh: enabled: false kubeedge: enabled: false cloudCore: nodeSelector: {\u0026#34;node-role.kubernetes.io/worker\u0026#34;: \u0026#34;\u0026#34;} tolerations: [] cloudhubPort: \u0026#34;10000\u0026#34; cloudhubQuicPort: \u0026#34;10001\u0026#34; cloudhubHttpsPort: \u0026#34;10002\u0026#34; cloudstreamPort: \u0026#34;10003\u0026#34; tunnelPort: \u0026#34;10004\u0026#34; cloudHub: advertiseAddress: - \u0026#34;\u0026#34; nodeLimit: \u0026#34;100\u0026#34; service: cloudhubNodePort: \u0026#34;30000\u0026#34; cloudhubQuicNodePort: \u0026#34;30001\u0026#34; cloudhubHttpsNodePort: \u0026#34;30002\u0026#34; cloudstreamNodePort: \u0026#34;30003\u0026#34; tunnelNodePort: \u0026#34;30004\u0026#34; edgeWatcher: nodeSelector: {\u0026#34;node-role.kubernetes.io/worker\u0026#34;: \u0026#34;\u0026#34;} tolerations: [] edgeWatcherAgent: nodeSelector: {\u0026#34;node-role.kubernetes.io/worker\u0026#34;: \u0026#34;\u0026#34;} tolerations: [] 安装集群 此时，在虚拟机安装代理即可查看虚拟机监控\n创建完成\n访问成功\n集群正常，不安装组件时内存占用率并不高\n安装kubectl\n$ yum install -y kubectl 安装可插拔组件 点击自定义资源 CRD，然后在搜索栏中输入 clusterconfiguration，点击搜索结果进入其详情页面。\n按照自身需要启用可插拔组件，此处打开所有组件，即修改为如下配置\napiVersion: installer.kubesphere.io/v1alpha1 kind: ClusterConfiguration metadata: labels: version: v3.1.0 name: ks-installer namespace: kubesphere-system spec: alerting: enabled: true auditing: enabled: true authentication: jwtSecret: \u0026#39;\u0026#39; common: es: basicAuth: enabled: false password: \u0026#39;\u0026#39; username: \u0026#39;\u0026#39; elasticsearchDataVolumeSize: 20Gi elasticsearchMasterVolumeSize: 4Gi elkPrefix: logstash externalElasticsearchPort: \u0026#39;\u0026#39; externalElasticsearchUrl: \u0026#39;\u0026#39; logMaxAge: 7 minioVolumeSize: 2Gi monitoring: endpoint: \u0026#39;http://prometheus-operated.kubesphere-monitoring-system.svc:9090\u0026#39; openldap: enabled: true openldapVolumeSize: 2Gi redis: enabled: true redisVolumSize: 2Gi console: enableMultiLogin: true port: 30880 devops: enabled: true jenkinsJavaOpts_MaxRAM: 2g jenkinsJavaOpts_Xms: 512m jenkinsJavaOpts_Xmx: 512m jenkinsMemoryLim: 2Gi jenkinsMemoryReq: 1500Mi jenkinsVolumeSize: 8Gi etcd: endpointIps: 10.170.0.5 monitoring: true port: 2379 tlsEnable: true events: enabled: true ruler: enabled: true replicas: 2 kubeedge: cloudCore: cloudHub: advertiseAddress: - \u0026#39;\u0026#39; nodeLimit: \u0026#39;100\u0026#39; cloudhubHttpsPort: \u0026#39;10002\u0026#39; cloudhubPort: \u0026#39;10000\u0026#39; cloudhubQuicPort: \u0026#39;10001\u0026#39; cloudstreamPort: \u0026#39;10003\u0026#39; nodeSelector: node-role.kubernetes.io/worker: \u0026#39;\u0026#39; service: cloudhubHttpsNodePort: \u0026#39;30002\u0026#39; cloudhubNodePort: \u0026#39;30000\u0026#39; cloudhubQuicNodePort: \u0026#39;30001\u0026#39; cloudstreamNodePort: \u0026#39;30003\u0026#39; tunnelNodePort: \u0026#39;30004\u0026#39; tolerations: [] tunnelPort: \u0026#39;10004\u0026#39; edgeWatcher: edgeWatcherAgent: nodeSelector: node-role.kubernetes.io/worker: \u0026#39;\u0026#39; tolerations: [] nodeSelector: node-role.kubernetes.io/worker: \u0026#39;\u0026#39; tolerations: [] enabled: true logging: enabled: true logsidecar: enabled: true replicas: 2 metrics_server: enabled: true monitoring: prometheusMemoryRequest: 400Mi prometheusVolumeSize: 5Gi storageClass: \u0026#39;\u0026#39; multicluster: clusterRole: none network: ippool: type: calico networkpolicy: enabled: true topology: type: weave-scope openpitrix: store: enabled: true persistence: storageClass: \u0026#39;\u0026#39; servicemesh: enabled: true 通过在线控制台，查看安装进度\n执行以下命令\n$ kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath=\u0026#39;{.items[0].metadata.name}\u0026#39;) -f 安装组件过程中，对CPU负载较大\n所有组件安装完成\n资源占用情况\n","date":"2021-06-19T22:15:00+08:00","permalink":"/archives/272/","title":"Kubesphere安装——以GCP为例"},{"content":"背景 使用原生Kubernetes-dashboard和Grafana+Prometheus管理较大规模集群比较困难，有如下缺陷\n每次执行命令都要登陆服务器 容器日志采集方案、监控方案搭建比较繁琐 添加节点不方便 管理存储插件、秘钥、配置不方便 解决方案 目前国内流行的k8s管理平台主要有rancher、kubesphere、kubeboard\nrancher主打原生集群管理，侧重运维人员 kubesphere主打开发运维工作流，侧重研发人员 kubeboard与kubesphere类似，侧重微服务管理，但功能上较少，对多节点管理需要收费 经过评估，最终决定采用kubesphere进行集群管理\nRancher — 多云混合云多集群Kubernetes管理平台\n[post url=\u0026ldquo;https://kubesphere.io/zh/\u0026rdquo; title=\u0026ldquo;Kubesphere\u0026rdquo; intro=\u0026ldquo;面向云原生应用的 容器混合云\u0026rdquo; cover=\u0026ldquo;https://kubesphere.io/images/logo.svg\u0026rdquo; /]\nKuboard — 快速在 Kubernetes 上落地微服务\n安装步骤 目标是面对小规模集群管理，节点数最多为5个，故不考虑高可用集群配置，采用单主节点+4个worker节点配置\n采用纯净CentOS 7 Linux系统，需要做以下前置准备\n依赖包安装 安装官方要求的依赖包，由于计划安装NFS存储插件，还需要安装nfs工具包\n$ yum install -y socat conntrack ebtables ipset nfs-utils 配置DNS解析 将各节点DNS配置为私有DNS服务器\n$ tee /etc/resolv.conf \u0026lt;\u0026lt;-\u0026#39;EOF\u0026#39; # Generated by NetworkManager nameserver 192.168.1.13 EOF 私有DNS搭建可以参考文章\n相关文章\n[[利用Docker自建Dns服务器]]\n按照官方教程安装多节点kubernetes 强烈建议先用KubeKey安装纯净k8s集群+控制面板，关闭所有插件。等安装完毕再一个个插件打开，防止因为内存不够而无法打开kubesphere\n安装步骤参考官方文档一键安装方式\n多节点安装 — Kubesphere\n启动成功\n启动成功后按照官方指南打开插件\n启用可插拔组件 — Kubesphere\n所有插件打开后如下\n使用 节点状态\n存储卷管理\n监控告警\n日志采集\n日志查询\n操作审计\n工具箱\n企业空间权限控制\n内置helm应用商店，支持上线审批，在线编辑等\n支持关联外部仓库\n应用流量治理\n链路追踪\n灰度发布\n费用统计\n更多的使用场景包括：DevOps、边缘节点、多集群管理还在探索中\n","date":"2021-05-23T22:58:00+08:00","permalink":"/archives/271/","title":"Kubesphere运维Kubernetes集群"},{"content":"背景 当Linux集群中机器不断增多，集群内通过配置/hosts实现域名内网访问。\n目标\n相同域名在集群之外通过公有DNS访问外网地址 相同域名在集群内通过DNS解析访问内网地址 步骤 搭建DNS服务器 DNS服务器可以通过jpillora/dnsmasq容器实现\njpillora/dnsmasq — DockerHub\n新增配置文件 dnsmasq.conf\n# 完整配置文件参考以下地址 # http://oss.segetech.com/intra/srv/dnsmasq.conf # 记录所有DNS查询请求 log-queries # 不使用本地配置的域名解析器 no-resolv # 不使用本地hosts no-hosts # 配置上游DNS服务器地址，用于解析国内域名 server=114.114.114.114 # 配置国际DNS服务器地址 server=8.8.8.8 strict-order # 配置内网解析 address=/gitlab.fjy8018.top/192.168.1.11 配置docker-compose文件，DNS服务器默认通过53 udp端口解析，不建议修改53端口的外部地址\nversion: \u0026#34;3.5\u0026#34; services: dnsmasq: image: jpillora/dnsmasq:1.1.0 networks: - dns-net ports: - 53:53/udp - 5380:8080 volumes: - ./dnsmasq.conf:/etc/dnsmasq.conf environment: - \u0026#34;HTTP_USER=admin\u0026#34; - \u0026#34;HTTP_PASS=admin\u0026#34; restart: always networks: dns-net: 启动服务即可\n后续需要修改DNS解析只需要访问5380的web UI界面即可动态调整\n服务器指向新DNS服务器 将所有服务器/etc/resolv.conf文件指向新DNS服务器地址\n$ tee /etc/resolv.conf \u0026lt;\u0026lt;-\u0026#39;EOF\u0026#39; # Generated by NetworkManager nameserver 192.168.1.13 EOF 补充 对于kubernetes集群，由于集群内使用了coredns和nodelocaldns缓存服务，还需要重启两个dns服务组件\n$ kubectl -n kube-system rollout restart daemonset nodelocaldns $ kubectl -n kube-system rollout restart deploy coredns ","date":"2021-05-09T18:14:12+08:00","permalink":"/archives/270/","title":"利用Docker自建DNS服务器"},{"content":"背景 SpringCloud 3.1版本之后，@EnableBinding、@Output等StreamApi注解都标记为废弃\n/** * Indicates that an output binding target will be created by the framework. * * @author Dave Syer * @author Marius Bogoevici * @author Artem Bilan * * @deprecated as of 3.1 in favor of functional programming model */ @Qualifier @Target({ ElementType.FIELD, ElementType.METHOD, ElementType.ANNOTATION_TYPE, ElementType.PARAMETER }) @Retention(RetentionPolicy.RUNTIME) @Inherited @Documented @Deprecated public @interface Output { /** * Specify the binding target name; used as a bean name for binding target and as a * destination name by default. * @return the binding target name */ String value() default \u0026#34;\u0026#34;; } /** * Enables the binding of targets annotated with {@link Input} and {@link Output} to a * broker, according to the list of interfaces passed as value to the annotation. * * @author Dave Syer * @author Marius Bogoevici * @author David Turanski * @author Soby Chacko * * @deprecated as of 3.1 in favor of functional programming model */ @Target({ ElementType.TYPE, ElementType.ANNOTATION_TYPE }) @Retention(RetentionPolicy.RUNTIME) @Documented @Inherited @Configuration @Import({ BindingBeansRegistrar.class, BinderFactoryAutoConfiguration.class }) @EnableIntegration @Deprecated public @interface EnableBinding { /** * A list of interfaces having methods annotated with {@link Input} and/or * {@link Output} to indicate binding targets. * @return list of interfaces */ Class\u0026lt;?\u0026gt;[] value() default {}; } 相关文章\nProducing and Consuming Messages — SpringCloud 官方文档\n@EnableBinding @deprecated as of 3.1 in favor of functional programming model — stackoverflow\nThe type EnableBinding is deprecated — stackoverflow\n因此，所有采用命令式编程的API，在升级到新版本时，官方推荐使用更加简单的函数式编程API\n完整代码样例\nFJiayang/spring-cloud-stream-rabbit-example — GitHub项目地址\n升级方式 命令式编程模型 以简单多Topic消息为例，原本的编程方式如下\n生产者 @SpringBootApplication public class ProducerApplication { public static void main(String[] args) { SpringApplication.run(ProducerApplication.class, args); } } /** * @author F嘉阳 * @date 2018-10-08 17:57 */ @RestController @EnableBinding(MySource.class) public class Producer { @Autowired private MySource channel; @RequestMapping(\u0026#34;/send\u0026#34;) public String send() { channel.output().send(MessageBuilder.withPayload(new Date()).build()); return \u0026#34;success\u0026#34;; } } /** * @author F嘉阳 * @date 2018-10-08 18:01 */ public interface MySource { String OUTPUT = \u0026#34;output\u0026#34;; @Output(MySource.OUTPUT) MessageChannel output(); } 配置文件\nspring: rabbitmq: host: 192.168.163.128 username: cms password: cms-mq-admin cloud: stream: bindings: output: destination: my-test-channel server: port: 8082 消费者 @SpringBootApplication public class ConsumerApplication { public static void main(String[] args) { SpringApplication.run(ConsumerApplication.class, args); } } /** * @author F嘉阳 * @date 2018-10-08 18:10 */ @EnableBinding(MySink.class) public class Consumer { @StreamListener(MySink.INPUT) public void receive(Message\u0026lt;String\u0026gt; message) { System.out.println(\u0026#34;接收到MQ消息:\u0026#34; + message.getPayload()); } } /** * @author F嘉阳 * @date 2018-10-08 18:07 */ public interface MySink { String INPUT = \u0026#34;input\u0026#34;; @Input(MySink.INPUT) SubscribableChannel input(); } 配置文件\nspring: rabbitmq: host: 192.168.163.128 username: cms password: cms-mq-admin cloud: stream: bindings: input: destination: my-test-channel server: port: 8081 函数式编程模型 从命令式升级到函数式编程，首先需要升级SpringCloud版本至3.1以上，此处升级到最新版2020.0.1\n\u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;UTF-8\u0026#34;?\u0026gt; \u0026lt;project xmlns=\u0026#34;http://maven.apache.org/POM/4.0.0\u0026#34; xmlns:xsi=\u0026#34;http://www.w3.org/2001/XMLSchema-instance\u0026#34; xsi:schemaLocation=\u0026#34;http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd\u0026#34;\u0026gt; \u0026lt;modelVersion\u0026gt;4.0.0\u0026lt;/modelVersion\u0026gt; \u0026lt;groupId\u0026gt;top.fjy8018\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;cloud-stream\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.0.1-SNAPSHOT\u0026lt;/version\u0026gt; \u0026lt;packaging\u0026gt;pom\u0026lt;/packaging\u0026gt; \u0026lt;name\u0026gt;cloud-stream\u0026lt;/name\u0026gt; \u0026lt;description\u0026gt;Demo project for Spring Boot\u0026lt;/description\u0026gt; \u0026lt;modules\u0026gt; \u0026lt;module\u0026gt;producer\u0026lt;/module\u0026gt; \u0026lt;module\u0026gt;consumer\u0026lt;/module\u0026gt; \u0026lt;/modules\u0026gt; \u0026lt;parent\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-parent\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;2.4.2\u0026lt;/version\u0026gt; \u0026lt;relativePath/\u0026gt; \u0026lt;!-- lookup parent from repository --\u0026gt; \u0026lt;/parent\u0026gt; \u0026lt;properties\u0026gt; \u0026lt;spring-cloud.version\u0026gt;2020.0.1\u0026lt;/spring-cloud.version\u0026gt; \u0026lt;/properties\u0026gt; \u0026lt;dependencyManagement\u0026gt; \u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.cloud\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-cloud-dependencies\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;${spring-cloud.version}\u0026lt;/version\u0026gt; \u0026lt;type\u0026gt;pom\u0026lt;/type\u0026gt; \u0026lt;scope\u0026gt;import\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; \u0026lt;/dependencyManagement\u0026gt; \u0026lt;/project\u0026gt; 目前国内有清晰的升级指引的还比较少，目前只能参考官方文档材料\nProgramming Model — SpringCloud官方文档\n模型1——系统定时驱动的消息队列 官方文档描述\nSuppliers (Sources) Function and Consumer are pretty straightforward when it comes to how their invocation is triggered. They are triggered based on data (events) sent to the destination they are bound to. In other words, they are classic event-driven components.\nHowever, Supplier is in its own category when it comes to triggering. Since it is, by definition, the source (the origin) of the data, it does not subscribe to any in-bound destination and, therefore, has to be triggered by some other mechanism(s). There is also a question of Supplier implementation, which could be imperative or reactive and which directly relates to the triggering of such suppliers.\nConsider the following sample:\n@SpringBootApplication public static class SupplierConfiguration { @Bean public Supplier\u0026lt;String\u0026gt; stringSupplier() { return () -\u0026gt; \u0026#34;Hello from Supplier\u0026#34;; } } The preceding Supplier bean produces a string whenever its get() method is invoked. However, who invokes this method and how often? The framework provides a default polling mechanism (answering the question of \u0026ldquo;Who?\u0026rdquo;) that will trigger the invocation of the supplier and by default it will do so every second (answering the question of \u0026ldquo;How often?\u0026rdquo;). In other words, the above configuration produces a single message every second and each message is sent to an output destination that is exposed by the binder.\n其大意为，若采用默认的函数式编程模型，会每秒钟发送一条消息到消费者，定时发送参数可以配置\n按照文档升级后如下\n生产者 @SpringBootApplication public class ProducerApplication { public static void main(String[] args) { SpringApplication.run(ProducerApplication.class, args); } @Bean public Supplier\u0026lt;Date\u0026gt; source1() { return () -\u0026gt; new Date(); } } 函数式编程不需要在代码中定义监听队列，只需要按照约定写在配置文件中即可\n官方文档描述\nFunctional binding names Unlike the explicit naming required by annotation-based support (legacy) used in the previous versions of spring-cloud-stream, the functional programming model defaults to a simple convention when it comes to binding names, thus greatly simplifying application configuration. Let’s look at the first example:\n@SpringBootApplication public class SampleApplication { @Bean public Function\u0026lt;String, String\u0026gt; uppercase() { return value -\u0026gt; value.toUpperCase(); } } In the preceding example we have an application with a single function which acts as message handler. As a Function it has an input and output. The naming convention used to name input and output bindings is as follows:\ninput - \u0026lt;functionName\u0026gt; + -in- + \u0026lt;index\u0026gt; output - \u0026lt;functionName\u0026gt; + -out- + \u0026lt;index\u0026gt; The in and out corresponds to the type of binding (such as input or output). The index is the index of the input or output binding. It is always 0 for typical single input/output function, so it’s only relevant for Functions with multiple input and output arguments.\nSo if for example you would want to map the input of this function to a remote destination (e.g., topic, queue etc) called \u0026ldquo;my-topic\u0026rdquo; you would do so with the following property:\n--spring.cloud.stream.bindings.uppercase-in-0.destination=my-topic Note how uppercase-in-0 is used as a segment in property name. The same goes for uppercase-out-0.\n配置文件\nspring: rabbitmq: host: 192.168.163.128 username: cms password: cms-mq-admin cloud: stream: bindings: source1-out-0: destination: test1 function: definition: source1 server: port: 8083 由此可见，整体配置简化了很多，多为规约大于配置的方式进行定义，特别注意function配置的位置，是在stream配置的同级\n消费者 @SpringBootApplication public class ConsumerApplication { public static void main(String[] args) { SpringApplication.run(ConsumerApplication.class, args); } @Bean public Consumer\u0026lt;Date\u0026gt; sink1() { return System.out::println; } } 配置文件同理\nspring: rabbitmq: host: 192.168.163.128 username: cms password: cms-mq-admin cloud: stream: bindings: sink1-in-0: destination: test1 function: definition: sink1 server: port: 8081 运行结果 生产者每隔一秒发送一次消息\n模型2——StreamBridge 但实际生产中，更多是由业务场景触发，故不能使用该模式进行配置，SpringCloud还提供了另外一种StreamBridge模式\n官方文档介绍\nSending arbitrary data to an output (e.g. Foreign event-driven sources) There are cases where the actual source of data may be coming from the external (foreign) system that is not a binder. For example, the source of the data may be a classic REST endpoint. How do we bridge such source with the functional mechanism used by spring-cloud-stream?\nSpring Cloud Stream provides two mechanisms, so let’s look at them in more details\nHere, for both samples we’ll use a standard MVC endpoint method called delegateToSupplier bound to the root web context, delegating incoming requests to stream via two different mechanisms - imperative (via StreamBridge) and reactive (via EmitterProcessor).\n大意是，实际的数据源可以是外部事件驱动，例如通过Rest接口\n官方样例\nUsing StreamBridge @SpringBootApplication @Controller public class WebSourceApplication { public static void main(String[] args) { SpringApplication.run(WebSourceApplication.class, \u0026#34;--spring.cloud.stream.source=toStream\u0026#34;); } @Autowired private StreamBridge streamBridge; @RequestMapping @ResponseStatus(HttpStatus.ACCEPTED) public void delegateToSupplier(@RequestBody String body) { System.out.println(\u0026#34;Sending \u0026#34; + body); streamBridge.send(\u0026#34;toStream-out-0\u0026#34;, body); } } Here we autowire a StreamBridge bean which allows us to send data to an output binding effectively bridging non-stream application with spring-cloud-stream. Note that preceding example does not have any source functions defined (e.g., Supplier bean) leaving the framework with no trigger to create source bindings, which would be typical for cases where configuration contains function beans. So to trigger the creation of source binding we use spring.cloud.stream.source property where you can declare the name of your sources. The provided name will be used as a trigger to create a source binding. So in the preceding example the name of the output binding will be toStream-out-0 which is consistent with the binding naming convention used by functions (see Binding and Binding names). You can use ; to signify multiple sources (e.g., --spring.cloud.stream.source=foo;bar)\n参考官方文档改造后\n生产者 这里直接贴出多主题的消费模式，实际生产中多数也是订阅多个主题\n@SpringBootApplication public class ProducerApplication { public static void main(String[] args) { SpringApplication.run(ProducerApplication.class, args); } } /** * @author F嘉阳 * @date 2018-10-08 17:57 */ @RestController public class Producer { @Autowired private StreamBridge streamBridge; @RequestMapping(\u0026#34;/send1\u0026#34;) public String send1() { streamBridge.send(\u0026#34;source1-out-0\u0026#34;, new Date()); return \u0026#34;success1\u0026#34;; } @RequestMapping(\u0026#34;/send2\u0026#34;) public String send2() { streamBridge.send(\u0026#34;source2-out-0\u0026#34;, LocalDateTime.now().format(DateTimeFormatter.ISO_LOCAL_DATE_TIME)); return \u0026#34;success2\u0026#34;; } } 配置文件\nspring: rabbitmq: host: 192.168.133.128 username: dev-user password: devpassword cloud: stream: bindings: source1-out-0: destination: test2 source2-out-0: destination: test3 function: definition: source1;source2 server: port: 8083 消费者 @SpringBootApplication public class ConsumerApplication { public static void main(String[] args) { SpringApplication.run(ConsumerApplication.class, args); } @Bean public Consumer\u0026lt;Date\u0026gt; sink1() { return System.out::println; } @Bean public Consumer\u0026lt;String\u0026gt; sink2() { return System.out::println; } } 配置文件\nspring: rabbitmq: host: 192.168.133.128 username: dev-user password: devpassword cloud: stream: bindings: sink1-in-0: destination: test2 sink2-in-0: destination: test3 function: definition: sink1;sink2 server: port: 8081 运行结果 触发/send1\n触发/send2\n可见消息被正确的路由\n总结 整体来看，函数式编程比命令式编程更加简洁，而且加上SpringCloud规约大于配置的特点，可以明显减少配置的代码量，是未来发展的趋势。\n","date":"2021-02-21T12:22:00+08:00","permalink":"/archives/269/","title":"SpringCloudStream函数式编程模型"},{"content":"背景 众所周知，Spring框架提供了著名的类拷贝工具BeanUtils，其使用简单、方便。\n但是其内部实现基于运行时反射进行同名属性拷贝，对于需要进行字段映射和高性能运行时场景都无法满足要求，因此，开源社区衍生出了以下的类拷贝框架\nMapStruct (编译期生成 Mapper 实现) Selma (编译期生成 Mapper 实现) yangtu222 - BeanUtils (第一次生成 copy 实现字节码) mica (第一次生成 copy 实现字节码) hutool (反射) 使用 目前第二代密码管理平台采用的是mica重写后的BeanUtil类拷贝方法，整体来说性能可以满足要求，但为了进一步提升查询时类转换性能，同时考虑到应用中本身就包含了swagger依赖，而swagger中本身就引入了mapstruct依赖包，故无需引入新依赖。\n因此，最终考虑将业务场景下的类拷贝改成Mapstruct实现，由此开始mapstruct爬坑之路。。\n软件包版本 SpringBoot:2.4.2\nSpringCloud:2020.0.1\nlombok: 1.18.16\nmapstruct: 1.3.1.Final\n新增依赖 由于需要由mapstruct根据定义的接口自动生成类转换实现，故需要引入mapstruct接口处理包，有两种方式，一种通过maven插件的形式，一种通过maven依赖的形式\nmaven插件的形式（官方推荐）\n\u0026lt;build\u0026gt; \u0026lt;plugins\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.apache.maven.plugins\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;maven-compiler-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;3.5.1\u0026lt;/version\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;source\u0026gt;1.8\u0026lt;/source\u0026gt; \u0026lt;target\u0026gt;1.8\u0026lt;/target\u0026gt; \u0026lt;annotationProcessorPaths\u0026gt; \u0026lt;path\u0026gt; \u0026lt;groupId\u0026gt;org.mapstruct\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;mapstruct-processor\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;${org.mapstruct.version}\u0026lt;/version\u0026gt; \u0026lt;/path\u0026gt; \u0026lt;/annotationProcessorPaths\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;/plugin\u0026gt; \u0026lt;/plugins\u0026gt; \u0026lt;/build\u0026gt; maven依赖的形式\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.mapstruct\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;mapstruct-processor\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;${mapstruct.version}\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; 考虑到平台的maven模块数量较大，互相依赖关系比较复杂，遗漏插件依赖容易导致难以排查的bug，最终采用第二种方式，减少编码量。\n新增映射类 /** * @author F嘉阳 * @date 2021/2/18 12:06 */ @Mapper(componentModel = \u0026#34;spring\u0026#34;, unmappedTargetPolicy = ReportingPolicy.IGNORE) public interface OssEntityMapper { OssVO ossToOssVO(Oss oss); OssDTO ossToOssDto(Oss oss); /** * bladeFile转Oss * * @param bladeFile * @return */ @Mapping(target = \u0026#34;fileName\u0026#34;, source = \u0026#34;name\u0026#34;) Oss bladeFileToOss(BladeFile bladeFile); } 新增单元测试 由于首次使用mapstruct，编写单元测试检测代码是否成功\n/** * @author F嘉阳 * @date 2021/2/18 14:23 */ @Slf4j @ExtendWith(BladeSpringExtension.class) @SpringBootTest(classes = ResourceApplication.class) @BladeBootTest(appName = AppConstant.APPLICATION_RESOURCE_NAME, profile = \u0026#34;test\u0026#34;, enableLoader = true) class OssEntityMapperTest { @Autowired private OssEntityMapper mapper; @Test void ossToOssVO() { Oss oss = new Oss(); oss.setFileName(\u0026#34;filename\u0026#34;); oss.setLink(\u0026#34;link\u0026#34;); oss.setOriginalName(\u0026#34;origin\u0026#34;); oss.setOssType(OssType.QI_NIU_OSS); oss.setIsDeleted(BladeConstant.DB_NOT_DELETED); OssVO ossVO = mapper.ossToOssVO(oss); assertEquals(\u0026#34;filename\u0026#34;, ossVO.getFileName()); assertEquals(\u0026#34;link\u0026#34;, ossVO.getLink()); assertEquals(\u0026#34;origin\u0026#34;, ossVO.getOriginalName()); } @Test void ossToOssDto() { Oss oss = new Oss(); oss.setFileName(\u0026#34;filename\u0026#34;); oss.setLink(\u0026#34;link\u0026#34;); oss.setOriginalName(\u0026#34;origin\u0026#34;); oss.setOssType(OssType.QI_NIU_OSS); oss.setIsDeleted(BladeConstant.DB_NOT_DELETED); OssDTO ossDTO = mapper.ossToOssDto(oss); assertEquals(\u0026#34;filename\u0026#34;, ossDTO.getFileName()); assertEquals(\u0026#34;link\u0026#34;, ossDTO.getLink()); assertEquals(\u0026#34;origin\u0026#34;, ossDTO.getOriginalName()); } @Test void bladeFileToOss() { BladeFile file = new BladeFile(); file.setName(\u0026#34;file\u0026#34;); file.setLink(\u0026#34;link\u0026#34;); file.setOriginalName(\u0026#34;origin\u0026#34;); Oss oss = mapper.bladeFileToOss(file); assertEquals(\u0026#34;file\u0026#34;, oss.getFileName()); assertEquals(\u0026#34;link\u0026#34;, oss.getLink()); assertEquals(\u0026#34;origin\u0026#34;, oss.getOriginalName()); } } 编译结果 编译结果，报错\n这就很奇怪了，这是按官网最简单的方式写的，而且IDEA也安装了mapstruct插件，也没报错，能正确识别属性，怎么编译就不通过了\n查看生成的代码，确实没有属性映射\n爬坑之路 由于编译报错，考虑以下方法解决\n移除spring容器依赖，采用类引用的方式编写。——无效\n移除属性映射，尝试编译，可以编译通过，但生成的代码没有同名属性映射，只会返回空值。——无效\n@Override public Oss bladeFileToOss(BladeFile bladeFile) { if ( bladeFile == null ) { return null; } Oss oss = new Oss(); return oss; } 更换类名后测试通过，多次测试仍然有不通过的情况。——可能和编译环境有关？\n相同代码放在单独的工程（非maven多模块）测试通过。——可能和maven多模块依赖有关？\n根据网上检索的结果，普遍反映的lombok版本太低导致的，经过检查，当前的lombok版本应该是满足最低版本要求的。——无效\n考虑是否和spring某些组件冲突，单独新开maven多模块工程测试，依然测试不通过。——果然和maven有关？\n解决方法 最终通过给官方仓库提交issue咨询得到答案\n原来是lombok版本太高导致的。。\n最终降低lombok版本问题解决。\n","date":"2021-02-20T22:45:11+08:00","permalink":"/archives/267/","title":"Mapstruct爬坑记录"},{"content":"背景 Sentinel默认只能对限流规则存储到内存中\n在生产环境中使用 Sentinel — Sentinel中文文档\n官方文档对于数据持久化描述如下\n我们提供了 ZooKeeper, Apollo, Nacos 等的动态数据源实现。以 ZooKeeper 为例子，如果要使用第三方的配置中心作为配置管理，您需要做下面的几件事情:\n实现一个公共的 ZooKeeper 客户端用于推送规则，在 Sentinel 控制台配置项中需要指定 ZooKeeper 的地址，启动时即创建 ZooKeeper Client。 我们需要针对每个应用（appName），每种规则设置不同的 path（可随时修改）；或者约定大于配置（如 path 的模式统一为 /sentinel_rules/{appName}/{ruleType}，e.g. sentinel_rules/appA/flowRule）。 规则配置页需要进行相应的改造，直接针对应用维度进行规则配置；修改同个应用多个资源的规则时可以批量进行推送，也可以分别推送。Sentinel 控制台将规则缓存在内存中（如 InMemFlowRuleStore），可以对其进行改造使其支持应用维度的规则缓存（key 为 appName），每次添加/修改/删除规则都先更新内存中的规则缓存，然后需要推送的时候从规则缓存中获取全量规则，然后通过上面实现的 Client 将规则推送到 ZooKeeper 即可。 应用客户端需要注册对应的读数据源以监听变更，可以参考 相关文档。 由于alibaba/Sentinel官方并没有实现对具体数据源的写入支持，只提供的实现的接口，故无法做到开箱即用\n解决方法 已经有贡献者提供了对三种动态数据源的实现PR，但还没合并进主版本中\n添加持久化支持，可选持久化选项 — Fixes Issues #1696 添加持久化支持，可选持久化选项：nacos、apollo、zookeeper ; 修改 README.md 说明文档\n进一步完善 本次基于jiajiangnan/Sentinelfork出自己的工程FJiayang/Sentinel，实现docker镜像打包，并基于GitHub Actions完成自动化构建和上传制品。\njiajiangnan/Sentinel — jiajiangnan的Sentinel仓库源码\nFJiayang/Sentinel — FJiayang的Sentinel仓库源码\n添加Dockerfile 基于最小jdk镜像制作\nFROM openjdk:8-jdk-alpine3.7 MAINTAINER fjy8018@gmail.com ENV VERSION 1.8.0 RUN mkdir -p /sentinel WORKDIR /sentinel ADD ./sentinel-dashboard/target/sentinel-dashboard.jar ./app-${VERSION}.jar # 设置时区，默认为UTC RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime RUN echo \u0026#39;Asia/Shanghai\u0026#39; \u0026gt; /etc/timezone CMD java ${PARAM} -jar app-${VERSION}.jar 添加GitHub流水线配置文件\nname: Publish Docker image on: [push] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up Maven Central Repository uses: actions/setup-java@v1 with: java-version: 1.8 - name: Publish package run: mvn --batch-mode clean package -Dmaven.test.skip=true - name: Push to Docker Hub uses: docker/build-push-action@v1 with: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_PASSWORD }} repository: fjy8018/sentinel tag_with_ref: true 具体的流水线编写可参考文章\n相关文章\n使用方法 具体参数文档\n项 类型 默认值 最小值 描述 sentinel.dashboard.auth.username String sentinel 无 登录控制台的用户名，默认为 sentinel sentinel.dashboard.auth.password String sentinel 无 登录控制台的密码，默认为 sentinel sentinel.dashboard.app.hideAppNoMachineMillis Integer 0 60000 是否隐藏无健康节点的应用，距离最近一次主机心跳时间的毫秒数，默认关闭 sentinel.dashboard.removeAppNoMachineMillis Integer 0 120000 是否自动删除无健康节点的应用，距离最近一次其下节点的心跳时间毫秒数，默认关闭 sentinel.dashboard.unhealthyMachineMillis Integer 60000 30000 主机失联判定，不可关闭 sentinel.dashboard.autoRemoveMachineMillis Integer 0 300000 距离最近心跳时间超过指定时间是否自动删除失联节点，默认关闭 datasource.provider String memory 无 默认为 memory, 可选持久化配置 nacos、apollo、zookeeper datasource.provider.nacos.server-addr String localhost:8848 无 nacos 注册中心地址 datasource.provider.nacos.username String 无 nacos 用户名，默认为空 datasource.provider.nacos.password String 无 nacos 密码，默认为空 datasource.provider.nacos.namespace String 无 nacos 名臣空间，默认为空 datasource.provider.nacos.group-id String SENTINEL_GROUP 无 nacos 分组，默认为 SENTINEL_GROUP datasource.provider.apollo.server-addr String http://localhost:10034 无 apollo 注册中心地址，必须有前缀 http:// 或 https:// datasource.provider.apollo.token String token 无 apollo 登录 token，默认为 token datasource.provider.zookeeper.server-addr String localhost:2181 无 zookeeper 注册中心地址 datasource.provider.zookeeper.session-timeout Integer 60000 0 zookeeper session超时时间，默认 60000 datasource.provider.zookeeper.connection-timeout Integer 15000 0 zookeeper connection超时时间，默认 15000 datasource.provider.zookeeper.retry.max-retries Integer 3 0 zookeeper 最大重试次数， 默认 3 datasource.provider.zookeeper.retry.base-sleep-time Integer 1000 1000 zookeeper 重试间隔最小时长，默认 1000 datasource.provider.zookeeper.retry.max-sleep-time Integer 2147483647 0 zookeeper 重试间隔最大时长，默认 2147483647 对于单条Docker命令启动\n$ docker run --rm -e PARAM=\u0026#34;-Dserver.port=8858 -Dcsp.sentinel.dashboard.server=localhost:8858 -Dproject.name=sentinel-dashboard -Ddatasource.provider=nacos -Ddatasource.provider.nacos.server-addr=192.168.133.128:8848 -Ddatasource.provider.nacos.username=nacos -Ddatasource.provider.nacos.password=nacos -Ddatasource.provider.nacos.group-id=SENTINEL_GROUP\u0026#34; fjy8018/sentinel:1.8.0 对于docker-compose启动，此处结合nacos与mysql基础设施给出示例，mysql需要提前写入nacos相关初始化表结构\nversion: \u0026#39;3.7\u0026#39; services: mysql57: image: mysql:5.7.29 restart: always command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci ports: - 3306:3306 networks: - dev volumes: - ./mysql57/data:/var/lib/mysql - ./mysql57/custom-master.cnf:/etc/mysql/conf.d/custom.cnf - ./mysql57/log:/var/log/mysql environment: MYSQL_ROOT_PASSWORD: root nacos-standalone: image: nacos/nacos-server:1.1.4 networks: - dev environment: - PREFER_HOST_MODE=hostname - MODE=standalone - SPRING_DATASOURCE_PLATFORM=mysql - MYSQL_MASTER_SERVICE_HOST=mysql57 - MYSQL_MASTER_SERVICE_DB_NAME=nacos - MYSQL_MASTER_SERVICE_PORT=3306 - MYSQL_SLAVE_SERVICE_HOST=mysql57 - MYSQL_SLAVE_SERVICE_PORT=3306 - MYSQL_MASTER_SERVICE_USER=root - MYSQL_MASTER_SERVICE_PASSWORD=root ports: - 8848:8848 depends_on: - mysql57 volumes: - ./standalone-logs/:/home/nacos/logs sentinel: image: fjy8018/sentinel:1.8.0 environment: - PARAM=-Dserver.port=8858 -Dcsp.sentinel.dashboard.server=localhost:8858 -Dproject.name=sentinel-dashboard -Ddatasource.provider=nacos -Ddatasource.provider.nacos.server-addr=nacos-standalone:8848 -Ddatasource.provider.nacos.username=nacos -Ddatasource.provider.nacos.password=nacos -Ddatasource.provider.nacos.group-id=SENTINEL_GROUP networks: - dev ports: - 8858:8858 restart: on-failure networks: dev: 启动成功后可以看到nacos上成功注册了sentinel-dashboard\n","date":"2021-02-07T12:06:58+08:00","permalink":"/archives/266/","title":"Sentinel持久化存储和镜像构建"},{"content":"工程准备 此处使用fork自alibaba/Sentinel的FJiayang/Sentinel项目实践\n工程地址\nSentinel — Sentinel仓库源码\n准备Dockerfile 该工程为标准Java项目，故准备好的Dockerfile文件如下\nFROM anapsix/alpine-java:8_server-jre_unlimited MAINTAINER fjy8018@gmail.com ENV PARAM \u0026#34;\u0026#34; ENV VERSION 1.8.0 RUN mkdir -p /sentinel WORKDIR /sentinel ADD ./sentinel-dashboard/target/sentinel-dashboard.jar ./app-${VERSION}.jar # 设置时区，默认为UTC RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime RUN echo \u0026#39;Asia/Shanghai\u0026#39; \u0026gt; /etc/timezone ENTRYPOINT [\u0026#34;java\u0026#34;, \u0026#34;${PARAM}\u0026#34;, \u0026#34;-jar\u0026#34;, \u0026#34;app-${VERSION}.jar\u0026#34;] 编写GitHub工作流 GitHub DevOps工作流又叫GitHub Actions 是2019年末推出的自动化CICD持续集成工作流，有详细的样例文档\n此处参考以下文档编写\n使用 Maven 构建和测试 Java — GitHub Actions 指南\n安全配置 由于需要建构建后的制品推送到DockerHub中，但又不能在工作流中直接使用用户名和密码登陆，故还需要配置环境变量\n在DockerHub中新增秘钥\n在构建工程的设置中新增用户名和密码变量\n编写后的工作流如下\nname: Publish Docker image on: [push] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up Maven Central Repository uses: actions/setup-java@v1 with: java-version: 1.8 - name: Publish package run: mvn --batch-mode clean package -Dmaven.test.skip=true - name: Push to Docker Hub uses: docker/build-push-action@v1 with: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_PASSWORD }} repository: fjy8018/sentinel tag_with_ref: true 触发构建 当前的触发方式配置为PUSH，修改好文件后会自动触发流水线工作\nMaven打包，速度很快\n打包完成\n检查结果 登录到DockerHub中检查推送结果，可见成功推送\n总结 整体来看，GitHub工作流效率高，文档齐全，编码上手速度快，相比GitLab CICD和DockerHub CICD来说，GitHub CICD配置更加简单、友好和高效。\n","date":"2021-02-07T11:58:43+08:00","permalink":"/archives/265/","title":"GitHub DevOps工作流构建Java项目并上传到DockerHub"},{"content":"背景 上一篇DevOps实践心得编写于2019年08月16日\n相关文章\n文章最后提到未来探索方向是\n全面容器化服务治理方向 国内对容器化有非常多的实践 kubernetes作为容器调度成为大多数服务治理的选择 容器隔离、迁移和伸缩带来的巨大优势 容器与传统应用部署结合方式 以半全自动方式落地DevOps，采用Mesos+marathon方式同时兼容传统服务部署和容器化部署 兼容分布式服务治理 国外有大量的实践案例 日志收集体系完备 经过一年半的时间，完成对完整DevOps的探索\nMesos+Marathon的DevOps分布式环境探索 参考文章\n相关文章\n该架构更加适合还未进行容器化改造的项目，其支持较为通用，但各组件都是由开源组件协调而成，故搭建较为繁琐，而且后续版本升级并不方便\n全面容器化服务探索 探索历程\n相关文章\n相关文章\n相关文章\n相关文章\n最终完成了DevOps容器化自动部署实践\n相关文章\n采用kuberneties方案，其统一性更加一致，依赖的基础设施也多支持容器化部署，机器利用率较高，版本升级、维护较为方便，适用于云原生项目部署\n遇到的坑 最终生产环境采用kuberneties集群方案，5个节点，但由于实施经验不足，依然遇到不少坑\n相关文章\n后续发展展望 云原生技术还在不断发展中，不少公司也研发并开源关于kuberneties集群管理方案，例如kubesphere\nkubesphere — The Kubernetes platform tailored for hybrid multicloud\n经过测试，其2.x版本资源占用较高，还处于刚起步阶段，暂时不适用于小规模集群\n","date":"2020-12-27T12:25:47+08:00","permalink":"/archives/263/","title":"DevOps实践心得二"},{"content":"问题背景 生产环境kuberneties集群搭建于2019年11月7日，使用kubeadm结合国内镜像搭建，参考文章\n相关文章\n2020年11月8日突然发现生产环境集群服务全都失效并且不可用，进行紧急排查处理\n处理过程 1. 排查服务是否启动 [root@k8smaster1 ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 65dff767f422 b95b1efa0436 \u0026#34;kube-controller-m...\u0026#34; 20 hours ago Up 20 hours k8s_kube-controller-manager_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_11 1b26acecbcea registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 \u0026#34;/pause\u0026#34; 20 hours ago Up 20 hours k8s_POD_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_1 9c6449df0a18 00638a24688b \u0026#34;kube-scheduler --...\u0026#34; 20 hours ago Up 20 hours k8s_kube-scheduler_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_10 028a1177cc9b registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 \u0026#34;/pause\u0026#34; 20 hours ago Up 20 hours k8s_POD_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_1 6c6bdc2c2644 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 \u0026#34;/pause\u0026#34; 20 hours ago Up 20 hours k8s_POD_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1 fe4a12daa7d7 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 \u0026#34;/pause\u0026#34; 20 hours ago Up 20 hours k8s_POD_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1 容器服务正常，但kubelet服务有报错信息，报连接不上apiserver\n[root@k8smaster1 ~]$ systemctl status kubelet -l ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since 日 2020-11-08 22:16:07 CST; 20h ago Docs: https://kubernetes.io/docs/ Main PID: 15313 (kubelet) Tasks: 16 Memory: 68.4M CGroup: /system.slice/kubelet.service └─15313 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 11月 09 18:43:06 k8smaster1 kubelet[15313]: E1109 18:43:06.967416 15313 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://192.168.1.14:6443/api/v1/services?limit=500\u0026amp;resourceVersion=0: dial tcp 192.168.1.14:6443: connect: connection refused 11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.009411 15313 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.109665 15313 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 11月 09 18:43:07 k8smaster1 kubelet[15313]: I1109 18:43:07.165755 15313 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach 11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.167106 15313 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSIDriver: Get https://192.168.1.14:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500\u0026amp;resourceVersion=0: dial tcp 192.168.1.14:6443: connect: connection refused 11月 09 18:43:07 k8smaster1 kubelet[15313]: I1109 18:43:07.169000 15313 kubelet_node_status.go:72] Attempting to register node k8smaster1 11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.209831 15313 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.310013 15313 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.362049 15313 kubelet_node_status.go:94] Unable to register node \u0026#34;k8smaster1\u0026#34; with API server: Post https://192.168.1.14:6443/api/v1/nodes: dial tcp 192.168.1.14:6443: connect: connection refused 11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.410174 15313 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 2. 检查APIServer apiserver 端口连接正常\n[root@k8smaster1 ~]$ telnet 192.168.1.14 6443 Trying 192.168.1.14... telnet: connect to address 192.168.1.14: Connection refused 重设防火墙，并重新检查，连接正常\n[root@k8smaster1 ~]$ iptables -F \u0026amp;\u0026amp; iptables -X \u0026amp;\u0026amp; iptables -F -t nat \u0026amp;\u0026amp; iptables -X -t nat \u0026amp;\u0026amp; iptables -P FORWARD ACCEPT [root@k8smaster1 ~]$ telnet 192.168.1.14 6443 Trying 192.168.1.14... Connected to 192.168.1.14. 但服务仍然无法访问\n3. 重启kubelet [root@k8smaster1 ~]$ systemctl restart kubelet [root@k8smaster1 ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4383926ef6c9 ecf910f40d6e \u0026#34;kube-apiserver --...\u0026#34; 3 seconds ago Up 2 seconds k8s_kube-apiserver_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1362 88eb6134fe3c 2c4adeb21b4f \u0026#34;etcd --advertise-...\u0026#34; 3 seconds ago Up 2 seconds k8s_etcd_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1768 65dff767f422 b95b1efa0436 \u0026#34;kube-controller-m...\u0026#34; 20 hours ago Up 20 hours k8s_kube-controller-manager_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_11 1b26acecbcea registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 \u0026#34;/pause\u0026#34; 20 hours ago Up 20 hours k8s_POD_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_1 9c6449df0a18 00638a24688b \u0026#34;kube-scheduler --...\u0026#34; 20 hours ago Up 20 hours k8s_kube-scheduler_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_10 028a1177cc9b registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 \u0026#34;/pause\u0026#34; 20 hours ago Up 20 hours k8s_POD_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_1 6c6bdc2c2644 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 \u0026#34;/pause\u0026#34; 20 hours ago Up 20 hours k8s_POD_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1 fe4a12daa7d7 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 \u0026#34;/pause\u0026#34; 20 hours ago Up 20 hours k8s_POD_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1 检查kubelet状态，仍然报错\n[root@k8smaster1 ~]$ systemctl status kubelet -l ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since 一 2020-11-09 18:52:24 CST; 24s ago Docs: https://kubernetes.io/docs/ Main PID: 31759 (kubelet) Tasks: 15 Memory: 30.1M CGroup: /system.slice/kubelet.service └─31759 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.063832 31759 controller.go:115] failed to ensure node lease exists, will retry in 3.2s, error: Get https://192.168.1.14:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/k8smaster1?timeout=10s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.152723 31759 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.252964 31759 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.353226 31759 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.453452 31759 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.553682 31759 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.653965 31759 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.754181 31759 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.854401 31759 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.954576 31759 kubelet.go:2244] node \u0026#34;k8smaster1\u0026#34; not found 4.检查API Server日志 查容器ID\n[root@k8smaster1 ~]$ docker ps -l CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 27a3b7787748 ecf910f40d6e \u0026#34;kube-apiserver --...\u0026#34; 18 seconds ago Up 18 seconds 查看日志\n[root@k8smaster1 ~]$ docker logs 27a Flag --insecure-port has been deprecated, This flag will be removed in a future version. I1109 10:55:42.281078 1 server.go:559] external host was not specified, using 192.168.1.15 I1109 10:55:42.281264 1 server.go:146] Version: v1.14.0 I1109 10:55:42.759000 1 plugins.go:158] Loaded 9 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I1109 10:55:42.759031 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. E1109 10:55:42.759943 1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted E1109 10:55:42.759976 1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted E1109 10:55:42.760007 1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted E1109 10:55:42.760038 1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted E1109 10:55:42.760065 1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted E1109 10:55:42.760086 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted I1109 10:55:42.760102 1 plugins.go:158] Loaded 9 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I1109 10:55:42.760110 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. I1109 10:55:42.763844 1 client.go:352] parsed scheme: \u0026#34;\u0026#34; I1109 10:55:42.763866 1 client.go:352] scheme \u0026#34;\u0026#34; not registered, fallback to default scheme I1109 10:55:42.764295 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}] I1109 10:55:42.764449 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 \u0026lt;nil\u0026gt;}] W1109 10:55:42.771517 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}. Err :connection error: desc = \u0026#34;transport: authentication handshake failed: x509: certificate has expired or is not yet valid\u0026#34;. Reconnecting... I1109 10:55:43.756910 1 client.go:352] parsed scheme: \u0026#34;\u0026#34; I1109 10:55:43.756939 1 client.go:352] scheme \u0026#34;\u0026#34; not registered, fallback to default scheme I1109 10:55:43.756999 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}] I1109 10:55:43.757077 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 \u0026lt;nil\u0026gt;}] W1109 10:55:43.763355 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}. Err :connection error: desc = \u0026#34;transport: authentication handshake failed: x509: certificate has expired or is not yet valid\u0026#34;. Reconnecting... W1109 10:55:43.770837 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}. Err :connection error: desc = \u0026#34;transport: authentication handshake failed: x509: certificate has expired or is not yet valid\u0026#34;. Reconnecting... W1109 10:55:44.764994 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}. Err :connection error: desc = \u0026#34;transport: authentication handshake failed: x509: certificate has expired or is not yet valid\u0026#34;. Reconnecting... W1109 10:55:45.541971 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}. Err :connection error: desc = \u0026#34;transport: authentication handshake failed: x509: certificate has expired or is not yet valid\u0026#34;. Reconnecting... W1109 10:55:46.657328 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}. Err :connection error: desc = \u0026#34;transport: authentication handshake failed: x509: certificate has expired or is not yet valid\u0026#34;. Reconnecting... W1109 10:55:47.938720 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}. Err :connection error: desc = \u0026#34;transport: authentication handshake failed: x509: certificate has expired or is not yet valid\u0026#34;. Reconnecting... W1109 10:55:48.730961 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}. Err :connection error: desc = \u0026#34;transport: authentication handshake failed: x509: certificate has expired or is not yet valid\u0026#34;. Reconnecting... W1109 10:55:51.899665 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}. Err :connection error: desc = \u0026#34;transport: authentication handshake failed: x509: certificate has expired or is not yet valid\u0026#34;. Reconnecting... W1109 10:55:52.961615 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}. Err :connection error: desc = \u0026#34;transport: authentication handshake failed: x509: certificate has expired or is not yet valid\u0026#34;. Reconnecting... W1109 10:55:59.193916 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}. Err :connection error: desc = \u0026#34;transport: authentication handshake failed: x509: certificate has expired or is not yet valid\u0026#34;. Reconnecting... W1109 10:56:00.025333 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 \u0026lt;nil\u0026gt;}. Err :connection error: desc = \u0026#34;transport: authentication handshake failed: x509: certificate has expired or is not yet valid\u0026#34;. Reconnecting... F1109 10:56:02.764150 1 storage_decorator.go:57] Unable to create storage backend: config (\u0026amp;{ /registry {[https://127.0.0.1:2379] /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/etcd/ca.crt} false true 0xc0007a94d0 apiextensions.k8s.io/v1beta1 \u0026lt;nil\u0026gt; 5m0s 1m0s}), err (context deadline exceeded) 日志最后提示：authentication handshake failed: x509: certificate has expired or is not yet valid\n以此信息查询解决方案\n解决方案 外部推荐方案 最终在该网站找到解决方案\nKubeadm证书过期时间调整 — Devops、小铁匠\n采用方案 由于生产环境集群版本为1.14，是支持自动轮换证书的，故首先解决生产环境问题，先手动重新签发证书，再开启证书自动续签\n重新签发证书 国内网络直接签发证书会因为无法连接Google而报错\n[root@k8smaster1 ~]$ kubeadm alpha certs renew all I1109 19:33:40.266778 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL \u0026#34;https://dl.k8s.io/release/stable-1.txt\u0026#34;: Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers) I1109 19:33:40.266978 22487 version.go:97] falling back to the local client version: v1.14.0 I1109 19:33:51.139051 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL \u0026#34;https://dl.k8s.io/release/stable-1.txt\u0026#34;: Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers) I1109 19:33:51.139111 22487 version.go:97] falling back to the local client version: v1.14.0 I1109 19:34:01.511909 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL \u0026#34;https://dl.k8s.io/release/stable-1.txt\u0026#34;: Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I1109 19:34:01.511967 22487 version.go:97] falling back to the local client version: v1.14.0 I1109 19:34:11.945411 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL \u0026#34;https://dl.k8s.io/release/stable-1.txt\u0026#34;: Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers) I1109 19:34:11.945504 22487 version.go:97] falling back to the local client version: v1.14.0 I1109 19:34:22.145905 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL \u0026#34;https://dl.k8s.io/release/stable-1.txt\u0026#34;: Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers) I1109 19:34:22.145975 22487 version.go:97] falling back to the local client version: v1.14.0 I1109 19:34:32.532514 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL \u0026#34;https://dl.k8s.io/release/stable-1.txt\u0026#34;: Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers) I1109 19:34:32.532561 22487 version.go:97] falling back to the local client version: v1.14.0 I1109 19:34:43.335574 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL \u0026#34;https://dl.k8s.io/release/stable-1.txt\u0026#34;: Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I1109 19:34:43.335628 22487 version.go:97] falling back to the local client version: v1.14.0 先准备一份连接国内镜像的kubeadm.conf文件\napiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration kubernetesVersion: v1.14.0 #--\u0026gt;这里改成你集群对应的版本 imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers #这里使用国内的镜像仓库，否则在重新签发的时候会报错：could not fetch a Kubernetes version from the internet: unable to get URL \u0026#34;https://dl.k8s.io/release/stable-1.txt\u0026#34; 重新签发，即可，并重启kubelet\n[root@k8smaster1 ~]$ kubeadm alpha certs renew all --config=/root/kubeadm.conf [root@k8smaster1 ~]$ systemctl restart kubelet 重新签发后若要使用kubectl命令，要重新生成~/.kube/config配置文件\n[root@k8smaster1 kubernetes]$ kubeadm init phase kubeconfig all --config=/root/kubeadm.conf [kubeconfig] Using kubeconfig folder \u0026#34;/etc/kubernetes\u0026#34; [kubeconfig] Writing \u0026#34;admin.conf\u0026#34; kubeconfig file [kubeconfig] Writing \u0026#34;kubelet.conf\u0026#34; kubeconfig file [kubeconfig] Writing \u0026#34;controller-manager.conf\u0026#34; kubeconfig file [kubeconfig] Writing \u0026#34;scheduler.conf\u0026#34; kubeconfig file [root@k8smaster1 kubernetes]$ cp -i /etc/kubernetes/admin.conf ~/.kube/config cp：是否覆盖\u0026#34;/root/.kube/config\u0026#34;？ y 命令正常使用\n[root@k8smaster1 kubernetes]$ kubectl get pods NAME READY STATUS RESTARTS AGE glusterfs-hfdmh 1/1 Running 0 351d glusterfs-k766z 1/1 Running 0 351d glusterfs-rrc7x 1/1 Running 0 351d heketi-68f9dfdfbf-2k58b 1/1 Running 0 351d nginx 1/1 Running 1 370d nginx-ds-29dbc 1/1 Running 5 366d nginx-ds-4w6cn 1/1 Running 1 370d nginx-ds-6lhsk 0/1 Evicted 0 216d nginx-ds-xq4h7 1/1 Running 3 366d tomcat-demo-6bc7d5b6f4-75rgc 0/1 Evicted 0 351d 重新启动集群上的所有服务，可见服务正常\n证书自动续签 kubelet证书分为server和client两种， k8s 1.9默认启用了client证书的自动轮换，但server证书自动轮换需要用户开启\n查询kubelet配置文件位置\n[root@k8smaster1 ~]$ find / -name 10-kubeadm.conf /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf [root@k8smaster1 ~]$ vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf 添加配置如下\n# 在/etc/systemd/system/kubelet.service.d/10-kubeadm.conf 增加如下参数 Environment=\u0026#34;KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true\u0026#34; 添加Controller-Manager参数\napiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: component: kube-controller-manager tier: control-plane name: kube-controller-manager namespace: kube-system spec: containers: - command: - kube-controller-manager - --allocate-node-cidrs=true # 证书有效期 - --experimental-cluster-signing-duration=87600h0m0s # 证书自动签发 - --feature-gates=RotateKubeletServerCertificate=true - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf - --bind-address=127.0.0.1 - --client-ca-file=/etc/kubernetes/pki/ca.crt - --cluster-cidr=172.22.0.0/16 - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key - --controllers=*,bootstrapsigner,tokencleaner - --kubeconfig=/etc/kubernetes/controller-manager.conf - --leader-elect=true - --node-cidr-mask-size=24 - --requestheader-client-ca-file=/etc/k 创建RBAC对象\ncat \u0026gt; ca-update.yaml \u0026lt;\u0026lt; EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: \u0026#34;true\u0026#34; labels: kubernetes.io/bootstrapping: rbac-defaults name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver rules: - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests/selfnodeserver verbs: - create --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubeadm:node-autoapprove-certificate-server roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:nodes EOF [root@k8smaster1 ~]$ kubectl create -f ca-update.yaml clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeserver created clusterrolebinding.rbac.authorization.k8s.io/kubeadm:node-autoapprove-certificate-server created ","date":"2020-12-27T12:00:58+08:00","permalink":"/archives/262/","title":"k8s证书失效问题排查"},{"content":"组建高性能MQ服务 双主双从模式 保障无单点故障问题的高可用集群 集群拓扑图 IP 资源 服务 10.139.12.52 1C2G20GB rocketmq-nameserver1、rocketmq-master1 10.139.12.53 1C2G20GB rocketmq-nameserver2、rocketmq-master2 10.139.12.54 1C2G20GB rocketmq-nameserver3、rocketmq-master1-slave 10.139.12.55 1C2G20GB rocketmq-nameserver4、rocketmqmaster2-slave 配置文件最优设置解析 修改host文件映射 10.139.12.52 rocketmq-nameserver1 rocketmq-master1 10.139.12.53 rocketmq-nameserver2 rocketmq-master2 10.139.12.54 rocketmq-nameserver3 rocketmq-master1-slave 10.139.12.55 rocketmq-nameserver4 rocketmq-master2-slave 将4个节点的hosts均改为如上内容，并将4.4.0版本包放入环境中解压\n创建存储目录 $ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store $ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog $ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue $ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index 建立软连接 $ cd ~/rocketmq-test $ ln -s rocketmq-all-4.4.0-bin-release rocketmq 修改配置文件 解压后默认的配置文件如下\n[user@301 ~/rocketmq-test/rocketmq/conf]$ll total 36 drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave -rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf -rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml -rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml -rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml -rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml -rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml 若想保障生产环境高可用，推荐使用双主双从同步双写模式，可以让一个主节点对应多个从节点，但从节点越多，同步效率越慢\n本次采用双主双从同步双写模式，一个主节点对应一个从节点\n[user@301 ~/rocketmq-test/rocketmq/conf/2m-2s-sync]$ll total 16 -rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-a.properties -rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties -rw-r--r-- 1 ocmuser gpocm 928 Jan 17 2019 broker-b.properties -rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties 用途如下\nIP 资源 服务 配置文件 10.139.12.52 1C2G20GB rocketmq-nameserver1、rocketmq-master1 broker-a.properties 10.139.12.53 1C2G20GB rocketmq-nameserver2、rocketmq-master2 broker-b.properties 10.139.12.54 1C2G20GB rocketmq-nameserver3、rocketmq-slave1 broker-a-s.properties 10.139.12.55 1C2G20GB rocketmq-nameserver4、rocketmq-slave2 broker-b-s.properties broker-a.properties 配置 #所属集群名字 brokerClusterName=rocketmq-cluster #broker 名字，注意此处不同的配置文件填写的不一样 brokerName=broker-a #0 表示 Master，\u0026gt;0 表示 Slave brokerId=0 #nameServer 地址，分号分割 namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876 #在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数 defaultTopicQueueNums=4 #是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭 autoCreateTopicEnable=false #是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭 autoCreateSubscriptionGroup=false #Broker 对外服务的监听端口 listenPort=10911 #删除文件时间点，默认凌晨 4 点 deleteWhen=04 #文件保留时间，默认 48 小时 fileReservedTime=120 #commitLog 每个文件的大小默认 1G mapedFileSizeCommitLog=1073741824 #ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整 mapedFileSizeConsumeQueue=300000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #检测物理文件磁盘空间，最大100% diskMaxUsedSpaceRatio=88 #存储路径 storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store #commitLog 存储路径 storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog #消费队列存储路径存储路径 storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue #消息索引存储路径 storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index #checkpoint 文件存储路径 storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint #abort 文件存储路径 abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort #限制的消息大小 maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Broker 的角色 #- ASYNC_MASTER 异步复制 Master #- SYNC_MASTER 同步双写 Master #- SLAVE brokerRole=SYNC_MASTER #刷盘方式 #- ASYNC_FLUSH 异步刷盘 #- SYNC_FLUSH 同步刷盘 flushDiskType=ASYNC_FLUSH #checkTransactionMessageEnable=false #发消息线程池数量 #sendMessageThreadPoolNums=128 #拉消息线程池数量 #pullMessageThreadPoolNums=128 broker-b.properties 配置 #所属集群名字 brokerClusterName=rocketmq-cluster #broker 名字，注意此处不同的配置文件填写的不一样 brokerName=broker-b #0 表示 Master，\u0026gt;0 表示 Slave brokerId=0 #nameServer 地址，分号分割 namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876 #在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数 defaultTopicQueueNums=4 #是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭 autoCreateTopicEnable=false #是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭 autoCreateSubscriptionGroup=false #Broker 对外服务的监听端口 listenPort=10911 #删除文件时间点，默认凌晨 4 点 deleteWhen=04 #文件保留时间，默认 48 小时 fileReservedTime=120 #commitLog 每个文件的大小默认 1G mapedFileSizeCommitLog=1073741824 #ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整 mapedFileSizeConsumeQueue=300000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #检测物理文件磁盘空间，最大100% diskMaxUsedSpaceRatio=88 #存储路径 storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store #commitLog 存储路径 storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog #消费队列存储路径存储路径 storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue #消息索引存储路径 storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index #checkpoint 文件存储路径 storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint #abort 文件存储路径 abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort #限制的消息大小 maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Broker 的角色 #- ASYNC_MASTER 异步复制 Master #- SYNC_MASTER 同步双写 Master #- SLAVE brokerRole=ASYNC_MASTER #刷盘方式 #- ASYNC_FLUSH 异步刷盘 #- SYNC_FLUSH 同步刷盘 flushDiskType=ASYNC_FLUSH #checkTransactionMessageEnable=false #发消息线程池数量 #sendMessageThreadPoolNums=128 #拉消息线程池数量 #pullMessageThreadPoolNums=128 broker-a-s.properties 配置 #所属集群名字 brokerClusterName=rocketmq-cluster #broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对 brokerName=broker-a #0 表示 Master，\u0026gt;0 表示 Slave brokerId=1 #nameServer 地址，分号分割 namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876 #在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数 defaultTopicQueueNums=4 #是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭 autoCreateTopicEnable=true #是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭 autoCreateSubscriptionGroup=true #Broker 对外服务的监听端口 listenPort=10911 #删除文件时间点，默认凌晨 4 点 deleteWhen=04 #文件保留时间，默认 48 小时 fileReservedTime=120 #commitLog 每个文件的大小默认 1G mapedFileSizeCommitLog=1073741824 #ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整 mapedFileSizeConsumeQueue=300000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #检测物理文件磁盘空间 diskMaxUsedSpaceRatio=88 #存储路径 storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store #commitLog 存储路径 storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog #消费队列存储路径存储路径 storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue #消息索引存储路径 storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index #checkpoint 文件存储路径 storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint #abort 文件存储路径 abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort #限制的消息大小 maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Broker 的角色 #- ASYNC_MASTER 异步复制 Master #- SYNC_MASTER 同步双写 Master #- SLAVE brokerRole=SLAVE #刷盘方式 #- ASYNC_FLUSH 异步刷盘 #- SYNC_FLUSH 同步刷盘 flushDiskType=ASYNC_FLUSH #checkTransactionMessageEnable=false #发消息线程池数量 #sendMessageThreadPoolNums=128 #拉消息线程池数量 #pullMessageThreadPoolNums=128 broker-b-s.properties 配置 #所属集群名字 brokerClusterName=rocketmq-cluster #broker 名字，注意此处不同的配置文件填写的不一样，与 Master 通过 brokerName 来配对 brokerName=broker-b #0 表示 Master，\u0026gt;0 表示 Slave brokerId=1 #nameServer 地址，分号分割 namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876;rocketmq-nameserver4:9876 #在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数 defaultTopicQueueNums=4 #是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭 autoCreateTopicEnable=true #是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭 autoCreateSubscriptionGroup=true #Broker 对外服务的监听端口 listenPort=10911 #删除文件时间点，默认凌晨 4 点 deleteWhen=04 #文件保留时间，默认 48 小时 fileReservedTime=120 #commitLog 每个文件的大小默认 1G mapedFileSizeCommitLog=1073741824 #ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整 mapedFileSizeConsumeQueue=300000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #检测物理文件磁盘空间 diskMaxUsedSpaceRatio=88 #存储路径 storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq/store #commitLog 存储路径 storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq/store/commitlog #消费队列存储路径存储路径 storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq/store/consumequeue #消息索引存储路径 storePathIndex=/home/ocmuser/rocketmq-test/rocketmq/store/index #checkpoint 文件存储路径 storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq/store/checkpoint #abort 文件存储路径 abortFile=/home/ocmuser/rocketmq-test/rocketmq/store/abort #限制的消息大小 maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Broker 的角色 #- ASYNC_MASTER 异步复制 Master #- SYNC_MASTER 同步双写 Master #- SLAVE brokerRole=SLAVE #刷盘方式 #- ASYNC_FLUSH 异步刷盘 #- SYNC_FLUSH 同步刷盘 flushDiskType=ASYNC_FLUSH #checkTransactionMessageEnable=false #发消息线程池数量 #sendMessageThreadPoolNums=128 #拉消息线程池数量 #pullMessageThreadPoolNums=128 注意：RocketMQ对配置文件行尾空格敏感，注意不要有行尾空格\n分发配置文件\n$ scp broker-* ocmuser@rocketmq-master2-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync $ scp broker-* ocmuser@rocketmq-master2:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync $ scp broker-* ocmuser@rocketmq-master1-slave:/home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync 修改日志配置文件 $ mkdir -p /home/ocmuser/rocketmq-test/rocketmq/logs $ cd /home/ocmuser/rocketmq-test/rocketmq/conf \u0026amp;\u0026amp; sed -i \u0026#39;s#${user.home}#/home/ocmuser/rocketmq-test/#g\u0026#39; *.xml 修改脚本启动参数 /home/ocmuser/rocketmq-test/rocketmq/bin/runbroker.sh内容如下，至少保障1g内存\n#====================================================================================== # 开发环境JVM Configuration #====================================================================================== JAVA_OPT=\u0026#34;${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m\u0026#34; /home/ocmuser/rocketmq-test/rocketmq/bin/runserver.sh内容如下\n#====================================================================================== # JVM Configuration #====================================================================================== JAVA_OPT=\u0026#34;${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m\u0026#34; 启动 NameServer $ cd /home/ocmuser/rocketmq-test/rocketmq/bin $ nohup sh mqnamesrv \u0026amp; 启动 Master1:BrokerServerA $ cd /home/ocmuser/rocketmq-test/rocketmq/bin $ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a.properties \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 \u0026amp; $ netstat -ntlp $ jps $ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log $ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log 启动 Master2:BrokerServerB $ cd /home/ocmuser/rocketmq-test/rocketmq/bin $ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b.properties \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 \u0026amp; $ netstat -ntlp $ jps $ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log $ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log 启动 Master1-Slave:BrokerServerC $ cd cd /home/ocmuser/rocketmq-test/rocketmq/bin $ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-a-s.properties \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 \u0026amp; $ netstat -ntlp $ jps $ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log $ tail -f -n 500 cd /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log 启动 Master2-Slave:BrokerServerD $ cd /home/ocmuser/rocketmq-test/rocketmq/bin $ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq/conf/2m-2s-sync/broker-b-s.properties \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 \u0026amp; $ netstat -ntlp $ jps $ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/broker.log $ tail -f -n 500 /home/ocmuser/rocketmq-test/logs/rocketmqlogs/namesrv.log 查看进程和日志，可见部署完成\n角色为同步双写，异步刷盘，4个节点通信正常\n控制台监控验证 修改连接配置\nrocketmq.config.namesrvAddr=10.139.12.51:9876;10.139.12.52:9876;10.139.12.53:9876;10.139.12.54:9876 连接正常\n查看各节点角色和状态\n服务停止 首先关闭 4 个 BrokerServer，再关闭 4 个 NameServer\n$ cd /home/ocmuser/rocketmq-test/rocketmq/bin $ sh mqshutdown broker $ sh mqshutdown namesrv 双主双从环境构建与调优 为取得可靠性与效率的平衡，建议采用同步双写，异步刷盘的模式 ","date":"2020-12-23T19:15:00+08:00","permalink":"/archives/256/","title":"RocketMQ学习笔记6——集群实战"},{"content":"消息存储结构 CommitLog：消息主体以及元数据的存储主体，存储Producer端写入的消息主体内容,消息内容不是定长的。消息主要是顺序写入日志文件，当文件满了，写入下一个文件；其日志信息科用于数据恢复，作为MQ数据保障兜底策略。 ConsumerQueue：消息消费队列，存储消息在commit log 中所在的这个位置，也可以说是一个偏移量.引入的目的主要是提高消息消费的性能。 IndexFile：IndexFile（索引文件）提供了一种可以通过key或时间区间来查询消息的方法。 每一个topic下的每一个message queue都对应一个ConsumerQueue文件，具体存储位置通过配置文件配置。 RocketMQ采用的是混合型的存储结构，即为Broker单个实例下所有的队列（ConsumerQueue）共用一个日志数据文件（即为CommitLog）来存储。 刷盘采用顺序写随机读的模式。 消费端采用“0拷贝”模式提升性能。 同步\\异步刷盘 消息存储：内存+磁盘存储，两种刷盘方式 异步刷盘 能够充分利用OS的PageCache的优势，只要消息写入PageCache（内存）即可将成功的ACK（成功信息）返回给Producer端。 消息堆积到一定量才触发刷盘。 消息刷盘采用后台异步线程提交的方式进行，降低了读写延迟，提高了MQ的性能和吞吐量。 同步刷盘 只有在消息真正持久化至磁盘后RocketMQ的Broker端才会真正返回给Producer端一个成功的ACK响应。 同步刷盘对MQ消息可靠性来说是一种不错的保障，但是性能上会有较大影响，一般适用于金融业务应用该模式较多。 同步\\异步复制 针对集群模式设计，同一组Broker有Master-Slave角色 推荐采用同步双写，异步复制的模式保持性能与可靠性的平衡 同步复制 Master与Slave都成功才返回ACK（成功信息）给Producer 异步复制 Master节点刷盘成功后立即返回ACK（成功信息）给Producer 再通过异步线程将信息同步给其他Slave节点 高性能，但存在数据丢失风险 同步双写 每个Master配置一个Slave，有多对Master-Slave，HA采用同步双写方式，即只有主备都写成功，才向应用返回成功，这种模式的优缺点如下：\n优点：数据与服务都无单点故障，Master宕机情况下，消息无延迟，服务可用性与数据可用性都非常高； 缺点：性能比异步复制模式略低（大约低10%左右），发送单个消息的RT会略高，且目前版本在主节点宕机后，备机不能自动切换为主机。 高可用机制 Master-Slave搭配实现高可用机制 通过BrokerId区分节点 Master支持读写，Slave只读 当Master繁忙或者不可用时，可以自动切换到Slave读取消息 最佳实践 创建Topic时，将Topic的Message Queue创建到多个Broker组中，使得逻辑上高可用 由于RocketMQ不支持主从切换，故可以考虑使用监控程序监听主节点的有效性，若连接超时，则将Slave节点的配置改为Master并重启，升级为主节点，实现高可用。 NameServer协调者 NameServer是一个非常简单的Topic路由注册中心，其角色类似Dubbo中的zookeeper，支持Broker的动态注册与发现，是整个集群的状态服务器。 主要包括两个功能： Broker管理，NameServer接受Broker集群的注册信息并且保存下来作为路由信息的基本数据。然后提供心跳检测机制，检查Broker是否还存活； 路由信息管理，每个NameServer将保存关于Broker集群的整个路由信息和用于客户端查询的队列信息。然后Producer和Conumser通过NameServer就可以知道整个Broker集群的路由信息，从而进行消息的投递和消费。 NameServer通常也是集群的方式部署，各实例间相互不进行信息通讯。Broker是向每一台NameServer注册自己的路由信息，所以每一个NameServer实例上面都保存一份完整的路由信息。 当某个NameServer因某种原因下线了，Broker仍然可以向其它NameServer同步其路由信息，Producer,Consumer仍然可以动态感知Broker的路由的信息。 为什么不用zookeeper RocketMQ 3.x以前使用zookeeper实现，后来改为nameserver实现 zookeeper提供主从选举，广播原子性等特性在MQ中用不上，功能冗余 zk使用主从自动切换提供高可用保障，nameserver使用多副本节点保障高可用 nameserver只提供路由等基本状态信息维护，代码量小，更加轻量，便于维护。 NameServer信息维护源码\npackage org.apache.rocketmq.namesrv.routeinfo; public class RouteInfoManager { private final ReadWriteLock lock = new ReentrantReadWriteLock(); private final HashMap\u0026lt;String/* topic */, List\u0026lt;QueueData\u0026gt;\u0026gt; topicQueueTable; private final HashMap\u0026lt;String/* brokerName */, BrokerData\u0026gt; brokerAddrTable; private final HashMap\u0026lt;String/* clusterName */, Set\u0026lt;String/* brokerName */\u0026gt;\u0026gt; clusterAddrTable; private final HashMap\u0026lt;String/* brokerAddr */, BrokerLiveInfo\u0026gt; brokerLiveTable; private final HashMap\u0026lt;String/* brokerAddr */, List\u0026lt;String\u0026gt;/* Filter Server */\u0026gt; filterServerTable; } ","date":"2020-12-23T19:13:00+08:00","permalink":"/archives/255/","title":"RocketMQ学习笔记5——原理解析"},{"content":"配置参数详解 PushConsumer核心参数 ConsumeFromWhere源码，支持三种参数配置，默认是CONSUME_FROM_LAST_OFFSET\npackage org.apache.rocketmq.common.consumer; public enum ConsumeFromWhere { /** * 第一次启动时从队列最后开始消费 * 下次启动从上次读取的位置开始消费 */ CONSUME_FROM_LAST_OFFSET, @Deprecated CONSUME_FROM_LAST_OFFSET_AND_FROM_MIN_WHEN_BOOT_FIRST, @Deprecated CONSUME_FROM_MIN_OFFSET, @Deprecated CONSUME_FROM_MAX_OFFSET, /** * 从队列最开始消费 * 下次启动从上次读取的位置开始消费 */ CONSUME_FROM_FIRST_OFFSET, /** * 指定具体时间点开始消费 * 下次启动从上次读取的位置开始消费 */ CONSUME_FROM_TIMESTAMP, } AllocateMessageQueueStrategy接口用于集群模式下消息策略，默认配置为平均消息分配器AllocateMessageQueueAveragely\n默认配置源码\npackage org.apache.rocketmq.client.consumer; public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer { /** * Constructor specifying consumer group and enabled msg trace flag. * * @param consumerGroup Consumer group. * @param enableMsgTrace Switch flag instance for message trace. */ public DefaultMQPushConsumer(final String consumerGroup, boolean enableMsgTrace) { this(null, consumerGroup, null, new AllocateMessageQueueAveragely(), enableMsgTrace, null); } } subscription订阅参数，传递两个值，一个是主题，一个是表达式tag过滤\npackage org.apache.rocketmq.client.consumer; public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer { /** * Subscription relationship */ private Map\u0026lt;String /* topic */, String /* sub expression */\u0026gt; subscription = new HashMap\u0026lt;String, String\u0026gt;(); } 样例：\nconsumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;TagA||TagB\u0026#34;); consumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;TagA\u0026#34;); consumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;*\u0026#34;); offsetStore配置核心偏移量，有两个实现，用于支持远程存储和本地存储，即支持集群和单机模式\nconsumeThreadMin与consumeThreadMax线程池自动调整参数，默认为20\npackage org.apache.rocketmq.client.consumer; public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer { /** * Minimum consumer thread number */ private int consumeThreadMin = 20; /** * Max consumer thread number */ private int consumeThreadMax = 20; } 流控参数consumeConcurrentlyMaxSpan与pullThresholdForQueue\npackage org.apache.rocketmq.client.consumer; public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer { /** * Concurrently max span offset.it has no effect on sequential consumption * 单个队列并行消费的最大跨度 */ private int consumeConcurrentlyMaxSpan = 2000; /** * Flow control threshold on queue level, each message queue will cache at most 1000 messages by default, * Consider the {@code pullBatchSize}, the instantaneous value may exceed the limit * 一个队列最大的消息消费个数 */ private int pullThresholdForQueue = 1000; } 消息拉取参数pullInterval与pullBatchSize\npackage org.apache.rocketmq.client.consumer; public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer { /** * Message pull Interval * 消息拉取的时间间隔 */ private long pullInterval = 0; /** * Batch pull size * 一次拉取的数据量 */ private int pullBatchSize = 32; } consumeMessageBatchMaxSize一次消息最大拉取的数据数量配置\npackage org.apache.rocketmq.client.consumer; public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer { /** * Batch consumption size * 一次消息最大拉取的数据数量 */ private int consumeMessageBatchMaxSize = 1; } MessageListener用于消息监听配置，包括并发、顺序读取等\nMessageModel配置消息模式，广播或者集群，默认为集群模式\npackage org.apache.rocketmq.client.consumer; public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer { /** * Message model defines the way how messages are delivered to each consumer clients. * \u0026lt;/p\u0026gt; * * RocketMQ supports two message models: clustering and broadcasting. If clustering is set, consumer clients with * the same {@link #consumerGroup} would only consume shards of the messages subscribed, which achieves load * balances; Conversely, if the broadcasting is set, each consumer client will consume all subscribed messages * separately. * \u0026lt;/p\u0026gt; * * This field defaults to clustering. */ private MessageModel messageModel = MessageModel.CLUSTERING; } 集群与广播模式 集群模式 MessageModel源码支持两种模式，集群和广播\npackage org.apache.rocketmq.common.protocol.heartbeat; /** * Message model */ public enum MessageModel { /** * broadcast */ BROADCASTING(\u0026#34;BROADCASTING\u0026#34;), /** * clustering（默认） */ CLUSTERING(\u0026#34;CLUSTERING\u0026#34;); private String modeCN; MessageModel(String modeCN) { this.modeCN = modeCN; } public String getModeCN() { return modeCN; } } 集群模式下GroupName用于把多个Consumer组织到一起，所以同一个集群，GroupName必须相同\n相同的GroupName的Consumer只消费所订阅消息的一部分，即支持负载均衡\n优势：集群模式下增加节点能动态增加负载均衡\n一条消息只会被一个Consumer消费\n样例 消息消费与奇偶性无关，随机负载均衡，并且负载均衡存在一定的差值，并非100%平分 为提高附在均衡度，可以用2个consume监听4个队列，或者3个consume监听6个队列 Producer只能保证平均分配到各个队列，不能保证被平均的消费 生产者\npublic class Producer { public static void main(String[] args) throws MQClientException, InterruptedException { String group_name = \u0026#34;test_model_producer_name\u0026#34;; DefaultMQProducer producer = new DefaultMQProducer(group_name); producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE); producer.start(); for (int i = 0; i \u0026lt; 10; i++) { try { String tag = (i % 2 ==0) ? \u0026#34;TagA\u0026#34;:\u0026#34;TagB\u0026#34;; Message msg = new Message(\u0026#34;test_model_topic2\u0026#34;,// topic tag,// tag (\u0026#34;信息内容\u0026#34; + i).getBytes()// body ); SendResult sendResult = producer.send(msg); System.out.println(sendResult); } catch (Exception e) { e.printStackTrace(); Thread.sleep(1000); } } producer.shutdown(); } } 消费者1\npublic class Consumer1 { public Consumer1() { try { String group_name = \u0026#34;test_model_consumer_name\u0026#34;; DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name); consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE); consumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;*\u0026#34;); consumer.setMessageModel(MessageModel.CLUSTERING); //\tconsumer.setMessageModel(MessageModel.BROADCASTING); consumer.registerMessageListener(new Listener()); consumer.start(); } catch (Exception e) { e.printStackTrace(); } } class Listener implements MessageListenerConcurrently { public ConsumeConcurrentlyStatus consumeMessage(List\u0026lt;MessageExt\u0026gt; msgs, ConsumeConcurrentlyContext context) { try { for(MessageExt msg : msgs){ String topic = msg.getTopic(); String msgBody = new String(msg.getBody(),\u0026#34;utf-8\u0026#34;); String tags = msg.getTags(); //if(tags.equals(\u0026#34;TagA\u0026#34;)) { System.out.println(\u0026#34;收到消息：\u0026#34; + \u0026#34; topic :\u0026#34; + topic + \u0026#34; ,tags : \u0026#34; + tags + \u0026#34; ,msg : \u0026#34; + msgBody); //} } } catch (Exception e) {\te.printStackTrace(); return ConsumeConcurrentlyStatus.RECONSUME_LATER; }\treturn ConsumeConcurrentlyStatus.CONSUME_SUCCESS; } } public static void main(String[] args) { Consumer1 c1 = new Consumer1(); System.out.println(\u0026#34;c1 start..\u0026#34;); } } 消费者2\npublic class Consumer2 { public Consumer2() { try { String group_name = \u0026#34;test_model_consumer_name\u0026#34;; DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name); consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE); consumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;*\u0026#34;); consumer.setMessageModel(MessageModel.CLUSTERING); //\tconsumer.setMessageModel(MessageModel.BROADCASTING); consumer.registerMessageListener(new Listener()); consumer.start(); } catch (Exception e) { e.printStackTrace(); } } class Listener implements MessageListenerConcurrently { public ConsumeConcurrentlyStatus consumeMessage(List\u0026lt;MessageExt\u0026gt; msgs, ConsumeConcurrentlyContext context) { try { for(MessageExt msg : msgs){ String topic = msg.getTopic(); String msgBody = new String(msg.getBody(),\u0026#34;utf-8\u0026#34;); String tags = msg.getTags(); //if(tags.equals(\u0026#34;TagB\u0026#34;)) { System.out.println(\u0026#34;收到消息：\u0026#34; + \u0026#34; topic :\u0026#34; + topic + \u0026#34; ,tags : \u0026#34; + tags + \u0026#34; ,msg : \u0026#34; + msgBody); //} } } catch (Exception e) {\te.printStackTrace(); return ConsumeConcurrentlyStatus.RECONSUME_LATER; }\treturn ConsumeConcurrentlyStatus.CONSUME_SUCCESS; } } public static void main(String[] args) { Consumer2 c2 = new Consumer2(); System.out.println(\u0026#34;c2 start..\u0026#34;); } } 发送10条消息结果\n消费者1结果\n消费者2结果\n注意，若使用集群模式，则tag表达式应该为*号，才能支持多consumer消费，若集群模式下指定tag表达式，则会导致消费结果不确定，原因如下\nconsumerA监听4条队列中的2条，consumerB监听4条队列中的另外2条。 Producer将包含TagA和TagB的消息均分到4条队列中。 consumerA消费TagA的消息，那么此时consumerA监听的2条队列中的TagB消息则永远不会被消费 样例\nconsumer1改成监听TagA，其他内容不变\nconsumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;TagA\u0026#34;); 此时发送10条消息，consumer1监听的结果\n此时将consumer2改成监听TagB并启动\nconsumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;TagB\u0026#34;); 此时发送10条消息，consumer1监听的结果\nconsumer2监听结果为空，大量消息丢失\n经过实测发现，只要集群模式下指定了tag，那么只有一个consumer能正常进行消费，其他consumer都为空\n广播模式 同一个ConsumeGroup里的Consumer都消费订阅Topic全部信息 也就是一条消息会被每个Consume消费 样例\n将consumer1和2改成广播模式，并监听所有tag\nconsumer.setMessageModel(MessageModel.BROADCASTING); consumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;*\u0026#34;); 发送10条消息consumer1消费结果\nconsumer2消费结果\n注意广播模式下使用服务端tag过滤仍然有bug\n将consumer1设置为只监听tagA消息\nconsumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;TagA\u0026#34;); consumer2设置为只监听tag2消息\nconsumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;TagB\u0026#34;); 发送10条消息consumer1消费结果\nconsumer2消费结果\n结果只有其中一个consumer能正常消费，与集群模式一样的问题\n目前（4.4.0）若想要实现同一个topic的tag过滤，则只能让consumer监听所有消息并主动判断\n样例：\npublic class Consumer1 { public Consumer1() { try { String group_name = \u0026#34;test_model_consumer_name\u0026#34;; DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name); consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE); consumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;*\u0026#34;); consumer.setMessageModel(MessageModel.BROADCASTING); consumer.registerMessageListener(new Listener()); consumer.start(); } catch (Exception e) { e.printStackTrace(); } } class Listener implements MessageListenerConcurrently { public ConsumeConcurrentlyStatus consumeMessage(List\u0026lt;MessageExt\u0026gt; msgs, ConsumeConcurrentlyContext context) { try { for(MessageExt msg : msgs){ String topic = msg.getTopic(); String msgBody = new String(msg.getBody(),\u0026#34;utf-8\u0026#34;); String tags = msg.getTags(); if(tags.equals(\u0026#34;TagA\u0026#34;)) { System.out.println(\u0026#34;收到消息：\u0026#34; + \u0026#34; topic :\u0026#34; + topic + \u0026#34; ,tags : \u0026#34; + tags + \u0026#34; ,msg : \u0026#34; + msgBody); } } } catch (Exception e) {\te.printStackTrace(); return ConsumeConcurrentlyStatus.RECONSUME_LATER; }\treturn ConsumeConcurrentlyStatus.CONSUME_SUCCESS; } } public static void main(String[] args) { Consumer1 c1 = new Consumer1(); System.out.println(\u0026#34;c1 start..\u0026#34;); } } 发送10条消息结果\n消费者1结果\n消费者2结果\n对于consumer消费指定队列消息，推荐做法是将消息投递到专属的队列中，由同一个consumer监听，即相当于用groupname+topic唯一定位一个consumer，该方法集群与广播模式均适用\n样例\n此时consumer1变为test_model_consumer_name1\npublic class Consumer1 { public Consumer1() { try { String group_name = \u0026#34;test_model_consumer_name1\u0026#34;; DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name); consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE); consumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;TagA\u0026#34;); //\tconsumer.setMessageModel(MessageModel.CLUSTERING); consumer.setMessageModel(MessageModel.BROADCASTING); consumer.registerMessageListener(new Listener()); consumer.start(); } catch (Exception e) { e.printStackTrace(); } } class Listener implements MessageListenerConcurrently { public ConsumeConcurrentlyStatus consumeMessage(List\u0026lt;MessageExt\u0026gt; msgs, ConsumeConcurrentlyContext context) { try { for(MessageExt msg : msgs){ String topic = msg.getTopic(); String msgBody = new String(msg.getBody(),\u0026#34;utf-8\u0026#34;); String tags = msg.getTags(); if(tags.equals(\u0026#34;TagA\u0026#34;)) { System.out.println(\u0026#34;收到消息：\u0026#34; + \u0026#34; topic :\u0026#34; + topic + \u0026#34; ,tags : \u0026#34; + tags + \u0026#34; ,msg : \u0026#34; + msgBody); } } } catch (Exception e) {\te.printStackTrace(); return ConsumeConcurrentlyStatus.RECONSUME_LATER; }\treturn ConsumeConcurrentlyStatus.CONSUME_SUCCESS; } } public static void main(String[] args) { Consumer1 c1 = new Consumer1(); System.out.println(\u0026#34;c1 start..\u0026#34;); } } consumer2变为test_model_consumer_name2\npublic class Consumer2 { public Consumer2() { try { String group_name = \u0026#34;test_model_consumer_name2\u0026#34;; DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(group_name); consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE); consumer.subscribe(\u0026#34;test_model_topic2\u0026#34;, \u0026#34;TagB\u0026#34;); //\tconsumer.setMessageModel(MessageModel.CLUSTERING); consumer.setMessageModel(MessageModel.BROADCASTING); consumer.registerMessageListener(new Listener()); consumer.start(); } catch (Exception e) { e.printStackTrace(); } } class Listener implements MessageListenerConcurrently { public ConsumeConcurrentlyStatus consumeMessage(List\u0026lt;MessageExt\u0026gt; msgs, ConsumeConcurrentlyContext context) { try { for(MessageExt msg : msgs){ String topic = msg.getTopic(); String msgBody = new String(msg.getBody(),\u0026#34;utf-8\u0026#34;); String tags = msg.getTags(); if(tags.equals(\u0026#34;TagB\u0026#34;)) { System.out.println(\u0026#34;收到消息：\u0026#34; + \u0026#34; topic :\u0026#34; + topic + \u0026#34; ,tags : \u0026#34; + tags + \u0026#34; ,msg : \u0026#34; + msgBody); } } } catch (Exception e) {\te.printStackTrace(); return ConsumeConcurrentlyStatus.RECONSUME_LATER; }\treturn ConsumeConcurrentlyStatus.CONSUME_SUCCESS; } } public static void main(String[] args) { Consumer2 c2 = new Consumer2(); System.out.println(\u0026#34;c2 start..\u0026#34;); } } 发送10条消息结果\n消费者1结果\n消费者2结果\n消费存储核心-OffSet Offset是消费者消费进度的核心 Offset指某个topic下的一条消息在某个MessageQueue里的位置 通过Offset可以进行定位到这条消息 Offset的存储实现分为远程文件类型和本地文件类型两种 集群模式-RemoteBrokerOffsetStore 默认集群模式clustering，采用远程文件存储Offset 本质上因为多消费模式，每个Consumer消费所订阅主题的一部分 广播模式-LocalFileOffsetStore 广播模式下，由于每个Consumer都会收到消息且消费 各个Consumer之间没有任何干扰，独立线程消费 所以使用LocalFileOffsetStore，也就是把Offset存储到本地 PushConsumer长轮询模式分析 DefaultMQPushConsumer是使用长轮询模式进行实现的 通常主流的消息获取模式：Push消息推送模式与Pull消息拉取模式 Push模式消耗broker端性能，若消费端消费速度不一致，可能导致消息堆积 pull模式消耗消费端性能，消费端不知道消息达到时间 RocketMQ采用长轮询机制（消费端主动获取） 长轮询过程 客户端发起获取消息请求，假设一次请求超时时间为15s 服务端判断是否有消息，有则直接返回 服务端无消息，则等待5s（阈值），5s后再次检查是否有消息，有则返回，无则继续等待 15s后循环检查结束，若仍然无消息，则返回给客户端 优势：\n主动权在客户端，当服务端消息堆积，不会全推送到客户端 缺点：\n连接返回时间较长，占用客户端资源（请求阻塞，等待服务端返回） 源码分析 源码实现位置\nPullRequestHoldService继承自Runnable接口\n核心类PullRequestHoldService源码全貌\npackage org.apache.rocketmq.broker.longpolling; public class PullRequestHoldService extends ServiceThread { private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.BROKER_LOGGER_NAME); private static final String TOPIC_QUEUEID_SEPARATOR = \u0026#34;@\u0026#34;; private final BrokerController brokerController; private final SystemClock systemClock = new SystemClock(); private ConcurrentMap\u0026lt;String/* topic@queueId */, ManyPullRequest\u0026gt; pullRequestTable = new ConcurrentHashMap\u0026lt;String, ManyPullRequest\u0026gt;(1024); public PullRequestHoldService(final BrokerController brokerController) { this.brokerController = brokerController; } public void suspendPullRequest(final String topic, final int queueId, final PullRequest pullRequest) { String key = this.buildKey(topic, queueId); ManyPullRequest mpr = this.pullRequestTable.get(key); if (null == mpr) { mpr = new ManyPullRequest(); ManyPullRequest prev = this.pullRequestTable.putIfAbsent(key, mpr); if (prev != null) { mpr = prev; } } mpr.addPullRequest(pullRequest); } private String buildKey(final String topic, final int queueId) { StringBuilder sb = new StringBuilder(); sb.append(topic); sb.append(TOPIC_QUEUEID_SEPARATOR); sb.append(queueId); return sb.toString(); } @Override public void run() { log.info(\u0026#34;{} service started\u0026#34;, this.getServiceName()); while (!this.isStopped()) { try { // 判断是否启用长轮询 if (this.brokerController.getBrokerConfig().isLongPollingEnable()) { // 等待5s this.waitForRunning(5 * 1000); } else { this.waitForRunning(this.brokerController.getBrokerConfig().getShortPollingTimeMills()); } long beginLockTimestamp = this.systemClock.now(); // 检查请求 this.checkHoldRequest(); long costTime = this.systemClock.now() - beginLockTimestamp; if (costTime \u0026gt; 5 * 1000) { log.info(\u0026#34;[NOTIFYME] check hold request cost {} ms.\u0026#34;, costTime); } } catch (Throwable e) { log.warn(this.getServiceName() + \u0026#34; service has exception. \u0026#34;, e); } } log.info(\u0026#34;{} service end\u0026#34;, this.getServiceName()); } @Override public String getServiceName() { return PullRequestHoldService.class.getSimpleName(); } private void checkHoldRequest() { // 遍历所有请求的channel for (String key : this.pullRequestTable.keySet()) { String[] kArray = key.split(TOPIC_QUEUEID_SEPARATOR); if (2 == kArray.length) { String topic = kArray[0]; int queueId = Integer.parseInt(kArray[1]); final long offset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId); try { // 唤醒 this.notifyMessageArriving(topic, queueId, offset); } catch (Throwable e) { log.error(\u0026#34;check hold request failed. topic={}, queueId={}\u0026#34;, topic, queueId, e); } } } } public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset) { notifyMessageArriving(topic, queueId, maxOffset, null, 0, null, null); } public void notifyMessageArriving(final String topic, final int queueId, final long maxOffset, final Long tagsCode, long msgStoreTime, byte[] filterBitMap, Map\u0026lt;String, String\u0026gt; properties) { String key = this.buildKey(topic, queueId); // 取出等待的channel（请求） ManyPullRequest mpr = this.pullRequestTable.get(key); if (mpr != null) { List\u0026lt;PullRequest\u0026gt; requestList = mpr.cloneListAndClear(); if (requestList != null) { List\u0026lt;PullRequest\u0026gt; replayList = new ArrayList\u0026lt;PullRequest\u0026gt;(); // 循环所有的请求 for (PullRequest request : requestList) { long newestOffset = maxOffset; if (newestOffset \u0026lt;= request.getPullFromThisOffset()) { newestOffset = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId); } if (newestOffset \u0026gt; request.getPullFromThisOffset()) { boolean match = request.getMessageFilter().isMatchedByConsumeQueue(tagsCode, new ConsumeQueueExt.CqExtUnit(tagsCode, msgStoreTime, filterBitMap)); // match by bit map, need eval again when properties is not null. if (match \u0026amp;\u0026amp; properties != null) { match = request.getMessageFilter().isMatchedByCommitLog(null, properties); } // 如果匹配上 if (match) { try { // 唤醒request this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(), request.getRequestCommand()); } catch (Throwable e) { log.error(\u0026#34;execute request when wakeup failed.\u0026#34;, e); } continue; } } if (System.currentTimeMillis() \u0026gt;= (request.getSuspendTimestamp() + request.getTimeoutMillis())) { try { this.brokerController.getPullMessageProcessor().executeRequestWhenWakeup(request.getClientChannel(), request.getRequestCommand()); } catch (Throwable e) { log.error(\u0026#34;execute request when wakeup failed.\u0026#34;, e); } continue; } replayList.add(request); } if (!replayList.isEmpty()) { mpr.addPullRequest(replayList); } } } } } ManyPullRequest里存储的实际为PullRequest数组\npublic class ManyPullRequest { private final ArrayList\u0026lt;PullRequest\u0026gt; pullRequestList = new ArrayList\u0026lt;\u0026gt;(); ... } PullRequest实际存储NIO请求\npublic class PullRequest { private final RemotingCommand requestCommand; /** NIO channel */ private final Channel clientChannel; private final long timeoutMillis; private final long suspendTimestamp; private final long pullFromThisOffset; private final SubscriptionData subscriptionData; private final MessageFilter messageFilter; ... } DefaultMQPullConsumer消息拉取模式 实现类DefaultMQPullConsumer 主要逻辑 获取Message Queue并遍历 维护OffsetStore 根据不同消息状态做不同处理 生产者样例 public class Producer { public static void main(String[] args) throws MQClientException, InterruptedException { String group_name = \u0026#34;test_pull_producer_name\u0026#34;; DefaultMQProducer producer = new DefaultMQProducer(group_name); producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE); producer.start(); for (int i = 0; i \u0026lt; 10; i++) { try { Message msg = new Message(\u0026#34;test_pull_topic\u0026#34;,// topic \u0026#34;TagA\u0026#34;,// tag (\u0026#34;Hello RocketMQ \u0026#34; + i).getBytes()// body ); // 配置延迟 msg.setDelayTimeLevel(2); SendResult sendResult = producer.send(msg, new MessageQueueSelector() { // 外部方法的arg会传递到Object中 @Override public MessageQueue select(List\u0026lt;MessageQueue\u0026gt; list, Message message, Object o) { Integer queueNumber = (Integer) o; return list.get(queueNumber); } },1); SendStatus status = sendResult.getSendStatus(); System.out.println(status); Thread.sleep(1000); } catch (Exception e) { e.printStackTrace(); Thread.sleep(3000); } } producer.shutdown(); } } 发送消息后的结果，在队列1有45条消息\n使用普通Pull轮询模式 public class PullConsumer { //Map\u0026lt;key, value\u0026gt; key为指定的队列，value为这个队列拉取数据的最后位置 private static final Map\u0026lt;MessageQueue, Long\u0026gt; offseTable = new HashMap\u0026lt;MessageQueue, Long\u0026gt;(); public static void main(String[] args) throws MQClientException { String group_name = \u0026#34;test_pull_consumer_name\u0026#34;; DefaultMQPullConsumer consumer = new DefaultMQPullConsumer(group_name); consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE); consumer.start(); System.err.println(\u0026#34;consumer start\u0026#34;); //\t从TopicTest这个主题去获取所有的队列（默认会有4个队列） Set\u0026lt;MessageQueue\u0026gt; mqs = consumer.fetchSubscribeMessageQueues(\u0026#34;test_pull_topic\u0026#34;); //\t遍历每一个队列，进行拉取数据 for (MessageQueue mq : mqs) { System.out.println(\u0026#34;Consume from the queue: \u0026#34; + mq); SINGLE_MQ: while (true) { try { //\t从queue中获取数据，从什么位置开始拉取数据 单次对多拉取32条记录 PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32); System.out.println(pullResult); System.out.println(pullResult.getPullStatus()); System.out.println(); // 客户端自行维护Offset putMessageQueueOffset(mq, pullResult.getNextBeginOffset()); switch (pullResult.getPullStatus()) { case FOUND: List\u0026lt;MessageExt\u0026gt; list = pullResult.getMsgFoundList(); for(MessageExt msg : list){ System.out.println(new String(msg.getBody())); } break; case NO_MATCHED_MSG: break; case NO_NEW_MSG: System.out.println(\u0026#34;没有新的数据啦...\u0026#34;); break SINGLE_MQ; case OFFSET_ILLEGAL: break; default: break; } } catch (Exception e) { e.printStackTrace(); } } } consumer.shutdown(); } private static void putMessageQueueOffset(MessageQueue mq, long offset) { offseTable.put(mq, offset); } private static long getMessageQueueOffset(MessageQueue mq) { Long offset = offseTable.get(mq); if (offset != null) return offset; return 0; } } 消费结果，其会随机选择队列读取，导致服务端积压的消息得不到有效的消费\n而且该模式需要客户端编码实现Offset存储\nPull定时任务实现样例 public class PullScheduleService { public static void main(String[] args) throws MQClientException { String group_name = \u0026#34;test_pull_consumer_name\u0026#34;; // 使用定时任务 final MQPullConsumerScheduleService scheduleService = new MQPullConsumerScheduleService(group_name); scheduleService.getDefaultMQPullConsumer().setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE); scheduleService.setMessageModel(MessageModel.CLUSTERING); scheduleService.registerPullTaskCallback(\u0026#34;test_pull_topic\u0026#34;, new PullTaskCallback() { @Override public void doPullTask(MessageQueue mq, PullTaskContext context) { MQPullConsumer consumer = context.getPullConsumer(); System.err.println(\u0026#34;-------------- queueId: \u0026#34; + mq.getQueueId() + \u0026#34;-------------\u0026#34;); try { // 获取从哪里拉取 long offset = consumer.fetchConsumeOffset(mq, false); if (offset \u0026lt; 0) offset = 0; PullResult pullResult = consumer.pull(mq, \u0026#34;*\u0026#34;, offset, 32); switch (pullResult.getPullStatus()) { case FOUND: List\u0026lt;MessageExt\u0026gt; list = pullResult.getMsgFoundList(); for(MessageExt msg : list){ //消费数据... System.out.println(new String(msg.getBody())); } break; case NO_MATCHED_MSG: break; case NO_NEW_MSG: case OFFSET_ILLEGAL: break; default: break; } // 客户端自动更新Offset，无需本地持久化维护 consumer.updateConsumeOffset(mq, pullResult.getNextBeginOffset()); // 设置再过3000ms后重新拉取 context.setPullNextDelayTimeMillis(3000); } catch (Exception e) { e.printStackTrace(); } } }); scheduleService.start(); } } 运行结果，由于采用多线程读取所有队列，所以当服务端有消息时可以及时获取\n实现原理 MQPullConsumerScheduleService中将PullTaskCallback方法作为Map存储 并使用ScheduledThreadPoolExecutor作为线程池，配置线程数为10 线程通过调度taskTable中的PullTaskImpl任务，实现方法回调 package org.apache.rocketmq.client.consumer; public class MQPullConsumerScheduleService { private final InternalLogger log = ClientLogger.getLog(); private final MessageQueueListener messageQueueListener = new MessageQueueListenerImpl(); private final ConcurrentMap\u0026lt;MessageQueue, PullTaskImpl\u0026gt; taskTable = new ConcurrentHashMap\u0026lt;MessageQueue, PullTaskImpl\u0026gt;(); private DefaultMQPullConsumer defaultMQPullConsumer; private int pullThreadNums = 20; private ConcurrentMap\u0026lt;String /* topic */, PullTaskCallback\u0026gt; callbackTable = new ConcurrentHashMap\u0026lt;String, PullTaskCallback\u0026gt;(); private ScheduledThreadPoolExecutor scheduledThreadPoolExecutor; public MQPullConsumerScheduleService(final String consumerGroup) { this.defaultMQPullConsumer = new DefaultMQPullConsumer(consumerGroup); this.defaultMQPullConsumer.setMessageModel(MessageModel.CLUSTERING); } public void putTask(String topic, Set\u0026lt;MessageQueue\u0026gt; mqNewSet) { Iterator\u0026lt;Entry\u0026lt;MessageQueue, PullTaskImpl\u0026gt;\u0026gt; it = this.taskTable.entrySet().iterator(); while (it.hasNext()) { Entry\u0026lt;MessageQueue, PullTaskImpl\u0026gt; next = it.next(); if (next.getKey().getTopic().equals(topic)) { if (!mqNewSet.contains(next.getKey())) { next.getValue().setCancelled(true); it.remove(); } } } for (MessageQueue mq : mqNewSet) { if (!this.taskTable.containsKey(mq)) { // 需要调度的任务 PullTaskImpl command = new PullTaskImpl(mq); this.taskTable.put(mq, command); // 使用多线程调度 this.scheduledThreadPoolExecutor.schedule(command, 0, TimeUnit.MILLISECONDS); } } } public void start() throws MQClientException { final String group = this.defaultMQPullConsumer.getConsumerGroup(); this.scheduledThreadPoolExecutor = new ScheduledThreadPoolExecutor( this.pullThreadNums, new ThreadFactoryImpl(\u0026#34;PullMsgThread-\u0026#34; + group) ); this.defaultMQPullConsumer.setMessageQueueListener(this.messageQueueListener); this.defaultMQPullConsumer.start(); log.info(\u0026#34;MQPullConsumerScheduleService start OK, {} {}\u0026#34;, this.defaultMQPullConsumer.getConsumerGroup(), this.callbackTable); } public void registerPullTaskCallback(final String topic, final PullTaskCallback callback) { // 放入回调任务集合 this.callbackTable.put(topic, callback); this.defaultMQPullConsumer.registerMessageQueueListener(topic, null); } class MessageQueueListenerImpl implements MessageQueueListener { @Override public void messageQueueChanged(String topic, Set\u0026lt;MessageQueue\u0026gt; mqAll, Set\u0026lt;MessageQueue\u0026gt; mqDivided) { MessageModel messageModel = MQPullConsumerScheduleService.this.defaultMQPullConsumer.getMessageModel(); switch (messageModel) { case BROADCASTING: MQPullConsumerScheduleService.this.putTask(topic, mqAll); break; case CLUSTERING: MQPullConsumerScheduleService.this.putTask(topic, mqDivided); break; default: break; } } } class PullTaskImpl implements Runnable { private final MessageQueue messageQueue; private volatile boolean cancelled = false; public PullTaskImpl(final MessageQueue messageQueue) { this.messageQueue = messageQueue; } @Override public void run() { String topic = this.messageQueue.getTopic(); if (!this.isCancelled()) { // 获取回调任务 PullTaskCallback pullTaskCallback = MQPullConsumerScheduleService.this.callbackTable.get(topic); if (pullTaskCallback != null) { final PullTaskContext context = new PullTaskContext(); context.setPullConsumer(MQPullConsumerScheduleService.this.defaultMQPullConsumer); try { // 执行客户端实现的回调任务 pullTaskCallback.doPullTask(this.messageQueue, context); } catch (Throwable e) { context.setPullNextDelayTimeMillis(1000); log.error(\u0026#34;doPullTask Exception\u0026#34;, e); } if (!this.isCancelled()) { MQPullConsumerScheduleService.this.scheduledThreadPoolExecutor.schedule(this, context.getPullNextDelayTimeMillis(), TimeUnit.MILLISECONDS); } else { log.warn(\u0026#34;The Pull Task is cancelled after doPullTask, {}\u0026#34;, messageQueue); } } else { log.warn(\u0026#34;Pull Task Callback not exist , {}\u0026#34;, topic); } } else { log.warn(\u0026#34;The Pull Task is cancelled, {}\u0026#34;, messageQueue); } } } } ","date":"2020-12-23T19:11:00+08:00","permalink":"/archives/254/","title":"RocketMQ学习笔记4——消费者核心"},{"content":"配置参数 producerGroup：组名 createTopicKey：创建Topic（不允许使用，应当由架构设计定义） defaultTopicQueueNums： 创建topic时默认的队列数量 （默认为4） sendMsgTimeout： 发送消息时的超时时间 （ 默认值：3000，单位：毫秒 ） compressMsgBodyOverHowmuch ：消息压缩字节（默认4096） retryTimesWhenSendFailed ： 同步模式下，在返回发送失败之前，内部尝试重新发送消息的最大次数。（ 默认值：2） retryTimesWhenSendAsyncFailed： 异步模式下，在发送失败之前，内部尝试重新发送消息的最大次数。 （ 默认值：2） retryAnotherBrokerWhenNotStoreOK： 同步模式下，消息保存失败时是否重试其他broker 。（ 默认值：false ） maxMessageSize：最大消息限制。（ 默认值：1024 * 1024 * 4，单位：字节 ） 主从同步机制 主从同步 同步信息：数据内容+元数据信息 元数据（配置信息）同步：Broker角色识别，为Slave则启动同步任务 通过Netty传输 实质为定时任务 消息同步（通过Socket）：HAService、HAconnection、WaitNotifyObject 通过原生Socket传输，性能高，数据包支持容量大 元数据同步源码分析 元信息同步工程源码位于broker中\n发起定时任务源码BrokerController\npackage org.apache.rocketmq.broker; public class BrokerController { public boolean initialize() throws CloneNotSupportedException { ... if (result) { ... // 进行角色判断 if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) { if (this.messageStoreConfig.getHaMasterAddress() != null \u0026amp;\u0026amp; this.messageStoreConfig.getHaMasterAddress().length() \u0026gt;= 6) { this.messageStore.updateHaMasterAddress(this.messageStoreConfig.getHaMasterAddress()); this.updateMasterHAServerAddrPeriodically = false; } else { this.updateMasterHAServerAddrPeriodically = true; } // 启动定时任务进行元信息同步 this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { BrokerController.this.slaveSynchronize.syncAll(); } catch (Throwable e) { log.error(\u0026#34;ScheduledTask syncAll slave exception\u0026#34;, e); } } }, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS); } else { this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { BrokerController.this.printMasterAndSlaveDiff(); } catch (Throwable e) { log.error(\u0026#34;schedule printMasterAndSlaveDiff error.\u0026#34;, e); } } }, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS); } } ... } } syncAll方法实现\npackage org.apache.rocketmq.broker.slave; public class SlaveSynchronize { ... public void syncAll() { // 同步Topic配置信息 this.syncTopicConfig(); // 同步消费者偏移量 this.syncConsumerOffset(); // 同步延迟偏移量 this.syncDelayOffset(); // 同步订阅组配置信息 this.syncSubscriptionGroupConfig(); } ... } 同步Topic配置信息源码 syncTopicConfig方法实现如下\npackage org.apache.rocketmq.broker.slave; public class SlaveSynchronize { ... private void syncTopicConfig() { String masterAddrBak = this.masterAddr; if (masterAddrBak != null) { try { TopicConfigSerializeWrapper topicWrapper = this.brokerController.getBrokerOuterAPI().getAllTopicConfig(masterAddrBak); if (!this.brokerController.getTopicConfigManager().getDataVersion() .equals(topicWrapper.getDataVersion())) { this.brokerController.getTopicConfigManager().getDataVersion() .assignNewOne(topicWrapper.getDataVersion()); this.brokerController.getTopicConfigManager().getTopicConfigTable().clear(); this.brokerController.getTopicConfigManager().getTopicConfigTable() .putAll(topicWrapper.getTopicConfigTable()); this.brokerController.getTopicConfigManager().persist(); log.info(\u0026#34;Update slave topic config from master, {}\u0026#34;, masterAddrBak); } } catch (Exception e) { log.error(\u0026#34;SyncTopicConfig Exception, {}\u0026#34;, masterAddrBak, e); } } } ... } 其中getAllTopicConfig方法实现如下\npackage org.apache.rocketmq.broker.out; public class BrokerOuterAPI { ... public TopicConfigSerializeWrapper getAllTopicConfig( final String addr) throws RemotingConnectException, RemotingSendRequestException, RemotingTimeoutException, InterruptedException, MQBrokerException { RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_TOPIC_CONFIG, null); // 发起远程同步调用 RemotingCommand response = this.remotingClient.invokeSync(MixAll.brokerVIPChannel(true, addr), request, 3000); assert response != null; switch (response.getCode()) { case ResponseCode.SUCCESS: { return TopicConfigSerializeWrapper.decode(response.getBody(), TopicConfigSerializeWrapper.class); } default: break; } throw new MQBrokerException(response.getCode(), response.getRemark()); } ... } invokeSync通过调用封装过的Netty客户端实现\npackage org.apache.rocketmq.remoting.netty; public class NettyRemotingClient extends NettyRemotingAbstract implements RemotingClient { ... /** * 同步请求 * @param addr 请求地址 * @param request 请求命令 * @param timeoutMillis 超时时间 */ @Override public RemotingCommand invokeSync(String addr, final RemotingCommand request, long timeoutMillis) throws InterruptedException, RemotingConnectException, RemotingSendRequestException, RemotingTimeoutException { long beginStartTime = System.currentTimeMillis(); final Channel channel = this.getAndCreateChannel(addr); if (channel != null \u0026amp;\u0026amp; channel.isActive()) { try { doBeforeRpcHooks(addr, request); // 算出极限耗时代价 long costTime = System.currentTimeMillis() - beginStartTime; if (timeoutMillis \u0026lt; costTime) { // 如果同步等待时长 \u0026lt; 极限耗时 说明请求不成立，抛出异常 throw new RemotingTimeoutException(\u0026#34;invokeSync call timeout\u0026#34;); } // 同步请求实现 RemotingCommand response = this.invokeSyncImpl(channel, request, timeoutMillis - costTime); doAfterRpcHooks(RemotingHelper.parseChannelRemoteAddr(channel), request, response); return response; } catch (RemotingSendRequestException e) { log.warn(\u0026#34;invokeSync: send request exception, so close the channel[{}]\u0026#34;, addr); this.closeChannel(addr, channel); throw e; } catch (RemotingTimeoutException e) { if (nettyClientConfig.isClientCloseSocketIfTimeout()) { this.closeChannel(addr, channel); log.warn(\u0026#34;invokeSync: close socket because of timeout, {}ms, {}\u0026#34;, timeoutMillis, addr); } log.warn(\u0026#34;invokeSync: wait response timeout exception, the channel[{}]\u0026#34;, addr); throw e; } } else { this.closeChannel(addr, channel); throw new RemotingConnectException(addr); } } ... } invokeSyncImpl实现逻辑\npackage org.apache.rocketmq.remoting.netty; public abstract class NettyRemotingAbstract { ... public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis) throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException { final int opaque = request.getOpaque(); try { final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null); this.responseTable.put(opaque, responseFuture); final SocketAddress addr = channel.remoteAddress(); // 使用netty channel channel.writeAndFlush(request).addListener(new ChannelFutureListener() { // 轮训监听 @Override public void operationComplete(ChannelFuture f) throws Exception { if (f.isSuccess()) { responseFuture.setSendRequestOK(true); return; } else { responseFuture.setSendRequestOK(false); } // 异常情况 responseTable.remove(opaque); responseFuture.setCause(f.cause()); responseFuture.putResponse(null); log.warn(\u0026#34;send a request command to channel \u0026lt;\u0026#34; + addr + \u0026#34;\u0026gt; failed.\u0026#34;); } }); RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis); if (null == responseCommand) { if (responseFuture.isSendRequestOK()) { throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis, responseFuture.getCause()); } else { throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause()); } } return responseCommand; } finally { this.responseTable.remove(opaque); } } ... } 同步消费者偏移量源码 其中syncConsumerOffset方法实现如下\npackage org.apache.rocketmq.broker.slave; public class SlaveSynchronize { ... private void syncConsumerOffset() { String masterAddrBak = this.masterAddr; if (masterAddrBak != null) { try { ConsumerOffsetSerializeWrapper offsetWrapper = this.brokerController.getBrokerOuterAPI().getAllConsumerOffset(masterAddrBak); this.brokerController.getConsumerOffsetManager().getOffsetTable() .putAll(offsetWrapper.getOffsetTable()); this.brokerController.getConsumerOffsetManager().persist(); log.info(\u0026#34;Update slave consumer offset from master, {}\u0026#34;, masterAddrBak); } catch (Exception e) { log.error(\u0026#34;SyncConsumerOffset Exception, {}\u0026#34;, masterAddrBak, e); } } } ... } getAllConsumerOffset实现逻辑\npackage org.apache.rocketmq.broker.out; public class BrokerOuterAPI { ... public ConsumerOffsetSerializeWrapper getAllConsumerOffset( final String addr) throws InterruptedException, RemotingTimeoutException, RemotingSendRequestException, RemotingConnectException, MQBrokerException { // 封装CMD指令向主节点发起同步offset信息请求，同步时间上限为3000ms RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_CONSUMER_OFFSET, null); RemotingCommand response = this.remotingClient.invokeSync(addr, request, 3000); assert response != null; switch (response.getCode()) { case ResponseCode.SUCCESS: { return ConsumerOffsetSerializeWrapper.decode(response.getBody(), ConsumerOffsetSerializeWrapper.class); } default: break; } throw new MQBrokerException(response.getCode(), response.getRemark()); } ... } 封装的CMD指令\npackage org.apache.rocketmq.common.protocol; public class RequestCode { ... public static final int NOTIFY_CONSUMER_IDS_CHANGED = 40; public static final int LOCK_BATCH_MQ = 41; public static final int UNLOCK_BATCH_MQ = 42; public static final int GET_ALL_CONSUMER_OFFSET = 43; public static final int GET_ALL_DELAY_OFFSET = 45; ... } 远程命令对象构建，构建后通过remotingClient发送即可\npackage org.apache.rocketmq.remoting.protocol; public class RemotingCommand { ... public static RemotingCommand createRequestCommand(int code, CommandCustomHeader customHeader) { RemotingCommand cmd = new RemotingCommand(); cmd.setCode(code); cmd.customHeader = customHeader; setCmdVersion(cmd); return cmd; } ... } 消息同步源码 消息同步工程源码位于store中\nHAConnection实现消息读写\nHAservice包含节点连接实现HAClient\n逻辑 Master节点：\nAcceptSocketService：接收Slave节点连接 HAConnection： ReadSocketService：读取来自Slave节点的数据 WriteSocketService：写入Slave节点的数据 Slave节点：\nHAService： HAClient：对Master节点连接，读写数据 通信协议：Master节点与Slave节点通信协议较简单，主要如下\n对象 用途 第几位 字段 数据类型 字节数 说明 Slave=\u0026gt;Master 上报CommitLog已经同步的物理位置 0 maxPhyOffset Long 8 CommitLog最大物理位置 Master=\u0026gt;Slave 传输新的CommitLog数据 0 fromPhyOffset Long 8 传输CommitLog开始的物理位置 1 size Int 4 传输CommitLog数据长度 2 body Bytes 传输CommitLog数据 连接Master源码 package org.apache.rocketmq.store.ha; public class HAService { ... class HAClient extends ServiceThread { /** * 客户端连接Master，使用NIO函数，提高效率 * @return * @throws ClosedChannelException */ private boolean connectMaster() throws ClosedChannelException { if (null == socketChannel) { String addr = this.masterAddress.get(); if (addr != null) { SocketAddress socketAddress = RemotingUtil.string2SocketAddress(addr); if (socketAddress != null) { this.socketChannel = RemotingUtil.connect(socketAddress); if (this.socketChannel != null) { this.socketChannel.register(this.selector, SelectionKey.OP_READ); } } } this.currentReportedOffset = HAService.this.defaultMessageStore.getMaxPhyOffset(); this.lastWriteTimestamp = System.currentTimeMillis(); } return this.socketChannel != null; } /** * Client(Slave)主循环，实现不断从Master传输CommitLog数据 * 上传Master自己本地的CommitLog已经同步物理位置 */ @Override public void run() { log.info(this.getServiceName() + \u0026#34; service started\u0026#34;); while (!this.isStopped()) { try { if (this.connectMaster()) { if (this.isTimeToReportOffset()) { // 满足上报间隔，上报到Master进度 // 固定间隔（默认为5s）向Master上报Slave本地CommitLog已经同步到的物理位置 boolean result = this.reportSlaveMaxOffset(this.currentReportedOffset); if (!result) { this.closeMaster(); } } this.selector.select(1000); // 处理Master传输给Slave的CommitLog数据 boolean ok = this.processReadEvent(); if (!ok) { this.closeMaster(); } // 若进度有变化，上报到Master进度 if (!reportSlaveMaxOffsetPlus()) { continue; } // Master超时未返回，关闭连接 long interval = HAService.this.getDefaultMessageStore().getSystemClock().now() - this.lastWriteTimestamp; if (interval \u0026gt; HAService.this.getDefaultMessageStore().getMessageStoreConfig() .getHaHousekeepingInterval()) { log.warn(\u0026#34;HAClient, housekeeping, found this connection[\u0026#34; + this.masterAddress + \u0026#34;] expired, \u0026#34; + interval); this.closeMaster(); log.warn(\u0026#34;HAClient, master not response some time, so close connection\u0026#34;); } } else { this.waitForRunning(1000 * 5); } } catch (Exception e) { log.warn(this.getServiceName() + \u0026#34; service has exception. \u0026#34;, e); this.waitForRunning(1000 * 5); } } log.info(this.getServiceName() + \u0026#34; service end\u0026#34;); } ... } ... } RemotingUtil使用原生Socket实现\npackage org.apache.rocketmq.remoting.common; public class RemotingUtil { public static SocketAddress string2SocketAddress(final String addr) { String[] s = addr.split(\u0026#34;:\u0026#34;); InetSocketAddress isa = new InetSocketAddress(s[0], Integer.parseInt(s[1])); return isa; } public static SocketChannel connect(SocketAddress remote, final int timeoutMillis) { SocketChannel sc = null; try { // 打开channel sc = SocketChannel.open(); // 设置同步阻塞，可能存在bug sc.configureBlocking(true); // 设置关闭Socket的延迟事件，那么当线程执行Socket的close方法时，会进入阻塞状态 // 直到Socket发送完所有剩余数据，或者超过了方法设置的延迟时间，才从close方法返回 sc.socket().setSoLinger(false, -1); // 禁止使用Nagle算法，使用小数据传输 // Nagle算法通过将缓冲区的小封包自动相连，组成较大的包 sc.socket().setTcpNoDelay(true); // 设置接收缓冲区 sc.socket().setReceiveBufferSize(1024 * 64); // 设置发送缓冲区 sc.socket().setSendBufferSize(1024 * 64); // 连接 sc.socket().connect(remote, timeoutMillis); // bug？又设置非同步阻塞了 sc.configureBlocking(false); return sc; } catch (Exception e) { if (sc != null) { try { sc.close(); } catch (IOException e1) { e1.printStackTrace(); } } } return null; } ... } 核心读请求实现\npackage org.apache.rocketmq.store.ha; public class HAService { ... class HAClient extends ServiceThread { private boolean processReadEvent() { int readSizeZeroTimes = 0; while (this.byteBufferRead.hasRemaining()) { try { int readSize = this.socketChannel.read(this.byteBufferRead); if (readSize \u0026gt; 0) { lastWriteTimestamp = HAService.this.defaultMessageStore.getSystemClock().now(); readSizeZeroTimes = 0; boolean result = this.dispatchReadRequest(); if (!result) { log.error(\u0026#34;HAClient, dispatchReadRequest error\u0026#34;); return false; } } else if (readSize == 0) { if (++readSizeZeroTimes \u0026gt;= 3) { break; } } else { log.info(\u0026#34;HAClient, processReadEvent read socket \u0026lt; 0\u0026#34;); return false; } } catch (IOException e) { log.info(\u0026#34;HAClient, processReadEvent read socket exception\u0026#34;, e); return false; } } return true; } /** * 核心读请求实现 * * 读取Master传输的CommitLog数据，并返回是否正确 * 如果读取到数据，写入CommitLog * 异常原因： * 1. Master传输来的数据开始位置offset(masterPhyOffset)不等于Slave的CommitLog数最大offset(slavePhyOffset) * 2. 上报到Master进度失败 * @return 是否正确 */ private boolean dispatchReadRequest() { final int msgHeaderSize = 8 + 4; // phyoffset + size int readSocketPos = this.byteBufferRead.position(); while (true) { // 读取到请求数据 int diff = this.byteBufferRead.position() - this.dispatchPostion; if (diff \u0026gt;= msgHeaderSize) { // 读取masterPhyOffset、bodySize long masterPhyOffset = this.byteBufferRead.getLong(this.dispatchPostion); int bodySize = this.byteBufferRead.getInt(this.dispatchPostion + 8); // 设置Slave节点CommitLog文件最大的offset位置 long slavePhyOffset = HAService.this.defaultMessageStore.getMaxPhyOffset(); // 校验Master传输来的数据开始位置offset和slave的commitLog数最大offset是否相同 if (slavePhyOffset != 0) { if (slavePhyOffset != masterPhyOffset) { log.error(\u0026#34;master pushed offset not equal the max phy offset in slave, SLAVE: \u0026#34; + slavePhyOffset + \u0026#34; MASTER: \u0026#34; + masterPhyOffset); return false; } } // 读取消息 if (diff \u0026gt;= (msgHeaderSize + bodySize)) { // 写入CommitLog byte[] bodyData = new byte[bodySize]; this.byteBufferRead.position(this.dispatchPostion + msgHeaderSize); this.byteBufferRead.get(bodyData); HAService.this.defaultMessageStore.appendToCommitLog(masterPhyOffset, bodyData); // 设置处理到的位置 this.byteBufferRead.position(readSocketPos); this.dispatchPostion += msgHeaderSize + bodySize; // 上报到Master进度 if (!reportSlaveMaxOffsetPlus()) { return false; } // 继续读数据 continue; } } // 空间写满，重新分配空间 if (!this.byteBufferRead.hasRemaining()) { this.reallocateByteBuffer(); } break; } return true; } ... } ... } 同步\\异步消息发送 实现分类 同步消息发送 发送：producer.send(Message msg) 核心实现：DefaultMQProducerImpl 异步消息发送 发送：producer.send(Message msg,SendCallback sendCallback) 核心实现：DefaultMQProducerImpl 实例 public class Producer { public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException { DefaultMQProducer producer = new DefaultMQProducer(\u0026#34;test_quick_producer_name\u0026#34;); producer.setNamesrvAddr(Const.NAMESRV_ADDR_MASTER_SLAVE); producer.start(); for(int i = 0 ; i \u0026lt;5; i ++) { //\t1.\t创建消息 Message message = new Message(\u0026#34;test_quick_topic\u0026#34;,\t//\t主题 \u0026#34;TagA\u0026#34;, //\t标签 \u0026#34;key\u0026#34; + i,\t// 用户自定义的key ,唯一的标识 (\u0026#34;Hello RocketMQ\u0026#34; + i).getBytes());\t//\t消息内容实体（byte[]） // 异步发送消息 producer.send(message, new SendCallback() { @Override public void onSuccess(SendResult sendResult) { System.err.println(\u0026#34;msgId: \u0026#34; + sendResult.getMsgId() + \u0026#34;, status: \u0026#34; + sendResult.getSendStatus()); producer.shutdown(); } @Override public void onException(Throwable e) { e.printStackTrace(); System.err.println(\u0026#34;------发送失败\u0026#34;); } }); }\t} } 同步消息发送源码分析 发送消息方法\npackage org.apache.rocketmq.client.producer; public class DefaultMQProducer extends ClientConfig implements MQProducer { ... @Override public SendResult send(Message msg, long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException { return this.defaultMQProducerImpl.send(msg, timeout); } } 模式枚举\npackage org.apache.rocketmq.client.impl; public enum CommunicationMode { SYNC, ASYNC, // 单向，不需要响应 ONEWAY, } 消息发送实现类\npackage org.apache.rocketmq.client.impl.producer; public class DefaultMQProducerImpl implements MQProducerInner { ... public SendResult send(Message msg, long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException { return this.sendDefaultImpl(msg, CommunicationMode.SYNC, null, timeout); } private SendResult sendDefaultImpl( Message msg, final CommunicationMode communicationMode, final SendCallback sendCallback, final long timeout ) throws MQClientException, RemotingException, MQBrokerException, InterruptedException { // 校验运行状态 this.makeSureStateOK(); // 检查消息格式 Validators.checkMessage(msg, this.defaultMQProducer); // 回调ID，用户标记是否同一次消息 final long invokeID = random.nextLong(); // 开始发送消息的时间 long beginTimestampFirst = System.currentTimeMillis(); long beginTimestampPrev = beginTimestampFirst; long endTimestamp = beginTimestampFirst; // 查找topic TopicPublishInfo topicPublishInfo = this.tryToFindTopicPublishInfo(msg.getTopic()); if (topicPublishInfo != null \u0026amp;\u0026amp; topicPublishInfo.ok()) { boolean callTimeout = false; // 要发送的消息队列 MessageQueue mq = null; Exception exception = null; SendResult sendResult = null; // 重试次数，从配置参数中读取 int timesTotal = communicationMode == CommunicationMode.SYNC ? 1 + this.defaultMQProducer.getRetryTimesWhenSendFailed() : 1; int times = 0; // 存储每次发送消息选择的broker名称 String[] brokersSent = new String[timesTotal]; for (; times \u0026lt; timesTotal; times++) { // 最后一次消息发送的brokerName进行存储 String lastBrokerName = null == mq ? null : mq.getBrokerName(); // 消息队列选择策略 MessageQueue mqSelected = this.selectOneMessageQueue(topicPublishInfo, lastBrokerName); if (mqSelected != null) { mq = mqSelected; brokersSent[times] = mq.getBrokerName(); try { beginTimestampPrev = System.currentTimeMillis(); // 计算代价 long costTime = beginTimestampPrev - beginTimestampFirst; if (timeout \u0026lt; costTime) { callTimeout = true; break; } // 消息发送实现 sendResult = this.sendKernelImpl(msg, mq, communicationMode, sendCallback, topicPublishInfo, timeout - costTime); endTimestamp = System.currentTimeMillis(); // 更新broker可用信息（容错） this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false); switch (communicationMode) { case ASYNC: return null; case ONEWAY: return null; case SYNC: // 同步发送等待返回结果 if (sendResult.getSendStatus() != SendStatus.SEND_OK) { if (this.defaultMQProducer.isRetryAnotherBrokerWhenNotStoreOK()) { continue; } } return sendResult; default: break; } } catch (RemotingException e) { endTimestamp = System.currentTimeMillis(); // 抛异常则更新broker容错信息 this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true); log.warn(String.format(\u0026#34;sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s\u0026#34;, invokeID, endTimestamp - beginTimestampPrev, mq), e); log.warn(msg.toString()); exception = e; continue; } catch (MQClientException e) { endTimestamp = System.currentTimeMillis(); this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true); log.warn(String.format(\u0026#34;sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s\u0026#34;, invokeID, endTimestamp - beginTimestampPrev, mq), e); log.warn(msg.toString()); exception = e; continue; } catch (MQBrokerException e) { endTimestamp = System.currentTimeMillis(); this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true); log.warn(String.format(\u0026#34;sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s\u0026#34;, invokeID, endTimestamp - beginTimestampPrev, mq), e); log.warn(msg.toString()); exception = e; // 抛出如下异常则进行重试 switch (e.getResponseCode()) { case ResponseCode.TOPIC_NOT_EXIST: case ResponseCode.SERVICE_NOT_AVAILABLE: case ResponseCode.SYSTEM_ERROR: case ResponseCode.NO_PERMISSION: case ResponseCode.NO_BUYER_ID: case ResponseCode.NOT_IN_CURRENT_UNIT: continue; default: if (sendResult != null) { return sendResult; } throw e; } } catch (InterruptedException e) { endTimestamp = System.currentTimeMillis(); this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false); log.warn(String.format(\u0026#34;sendKernelImpl exception, throw exception, InvokeID: %s, RT: %sms, Broker: %s\u0026#34;, invokeID, endTimestamp - beginTimestampPrev, mq), e); log.warn(msg.toString()); log.warn(\u0026#34;sendKernelImpl exception\u0026#34;, e); log.warn(msg.toString()); throw e; } } else { break; } } if (sendResult != null) { return sendResult; } String info = String.format(\u0026#34;Send [%d] times, still failed, cost [%d]ms, Topic: %s, BrokersSent: %s\u0026#34;, times, System.currentTimeMillis() - beginTimestampFirst, msg.getTopic(), Arrays.toString(brokersSent)); info += FAQUrl.suggestTodo(FAQUrl.SEND_MSG_FAILED); MQClientException mqClientException = new MQClientException(info, exception); if (callTimeout) { throw new RemotingTooMuchRequestException(\u0026#34;sendDefaultImpl call timeout\u0026#34;); } if (exception instanceof MQBrokerException) { mqClientException.setResponseCode(((MQBrokerException) exception).getResponseCode()); } else if (exception instanceof RemotingConnectException) { mqClientException.setResponseCode(ClientErrorCode.CONNECT_BROKER_EXCEPTION); } else if (exception instanceof RemotingTimeoutException) { mqClientException.setResponseCode(ClientErrorCode.ACCESS_BROKER_TIMEOUT); } else if (exception instanceof MQClientException) { mqClientException.setResponseCode(ClientErrorCode.BROKER_NOT_EXIST_EXCEPTION); } throw mqClientException; } List\u0026lt;String\u0026gt; nsList = this.getmQClientFactory().getMQClientAPIImpl().getNameServerAddressList(); if (null == nsList || nsList.isEmpty()) { throw new MQClientException( \u0026#34;No name server address, please set it.\u0026#34; + FAQUrl.suggestTodo(FAQUrl.NAME_SERVER_ADDR_NOT_EXIST_URL), null).setResponseCode(ClientErrorCode.NO_NAME_SERVER_EXCEPTION); } throw new MQClientException(\u0026#34;No route info of this topic, \u0026#34; + msg.getTopic() + FAQUrl.suggestTodo(FAQUrl.NO_TOPIC_ROUTE_INFO), null).setResponseCode(ClientErrorCode.NOT_FOUND_TOPIC_EXCEPTION); } } 查找topic方法实现\npackage org.apache.rocketmq.client.impl.producer; public class DefaultMQProducerImpl implements MQProducerInner { ... private TopicPublishInfo tryToFindTopicPublishInfo(final String topic) { // 使用Map缓存 TopicPublishInfo topicPublishInfo = this.topicPublishInfoTable.get(topic); if (null == topicPublishInfo || !topicPublishInfo.ok()) { // 当无可用topic发布信息时，主动向NameServer拉取一次 this.topicPublishInfoTable.putIfAbsent(topic, new TopicPublishInfo()); this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic); topicPublishInfo = this.topicPublishInfoTable.get(topic); } // 若可用 if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) { return topicPublishInfo; } else { // 否则根据设置判断是否新建Topic this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, true, this.defaultMQProducer); topicPublishInfo = this.topicPublishInfoTable.get(topic); return topicPublishInfo; } } } 客户端获取路由信息实现\npackage org.apache.rocketmq.client.impl.factory; public class MQClientInstance { public boolean updateTopicRouteInfoFromNameServer(final String topic) { return updateTopicRouteInfoFromNameServer(topic, false, null); } public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault, DefaultMQProducer defaultMQProducer) { try { if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) { try { TopicRouteData topicRouteData; if (isDefault \u0026amp;\u0026amp; defaultMQProducer != null) { topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(), 1000 * 3); if (topicRouteData != null) { for (QueueData data : topicRouteData.getQueueDatas()) { int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums()); data.setReadQueueNums(queueNums); data.setWriteQueueNums(queueNums); } } } else { topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, 1000 * 3); } if (topicRouteData != null) { TopicRouteData old = this.topicRouteTable.get(topic); boolean changed = topicRouteDataIsChange(old, topicRouteData); if (!changed) { changed = this.isNeedUpdateTopicRouteInfo(topic); } else { log.info(\u0026#34;the topic[{}] route info changed, old[{}] ,new[{}]\u0026#34;, topic, old, topicRouteData); } if (changed) { TopicRouteData cloneTopicRouteData = topicRouteData.cloneTopicRouteData(); for (BrokerData bd : topicRouteData.getBrokerDatas()) { this.brokerAddrTable.put(bd.getBrokerName(), bd.getBrokerAddrs()); } // Update Pub info { TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData); publishInfo.setHaveTopicRouterInfo(true); Iterator\u0026lt;Entry\u0026lt;String, MQProducerInner\u0026gt;\u0026gt; it = this.producerTable.entrySet().iterator(); while (it.hasNext()) { Entry\u0026lt;String, MQProducerInner\u0026gt; entry = it.next(); MQProducerInner impl = entry.getValue(); if (impl != null) { impl.updateTopicPublishInfo(topic, publishInfo); } } } // Update sub info { Set\u0026lt;MessageQueue\u0026gt; subscribeInfo = topicRouteData2TopicSubscribeInfo(topic, topicRouteData); Iterator\u0026lt;Entry\u0026lt;String, MQConsumerInner\u0026gt;\u0026gt; it = this.consumerTable.entrySet().iterator(); while (it.hasNext()) { Entry\u0026lt;String, MQConsumerInner\u0026gt; entry = it.next(); MQConsumerInner impl = entry.getValue(); if (impl != null) { impl.updateTopicSubscribeInfo(topic, subscribeInfo); } } } log.info(\u0026#34;topicRouteTable.put. Topic = {}, TopicRouteData[{}]\u0026#34;, topic, cloneTopicRouteData); this.topicRouteTable.put(topic, cloneTopicRouteData); return true; } } else { log.warn(\u0026#34;updateTopicRouteInfoFromNameServer, getTopicRouteInfoFromNameServer return null, Topic: {}\u0026#34;, topic); } } catch (Exception e) { if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX) \u0026amp;\u0026amp; !topic.equals(MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC)) { log.warn(\u0026#34;updateTopicRouteInfoFromNameServer Exception\u0026#34;, e); } } finally { this.lockNamesrv.unlock(); } } else { log.warn(\u0026#34;updateTopicRouteInfoFromNameServer tryLock timeout {}ms\u0026#34;, LOCK_TIMEOUT_MILLIS); } } catch (InterruptedException e) { log.warn(\u0026#34;updateTopicRouteInfoFromNameServer Exception\u0026#34;, e); } return false; } } 消息队列选择策略实现\npackage org.apache.rocketmq.client.impl.producer; public class DefaultMQProducerImpl implements MQProducerInner { ... public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) { return this.mqFaultStrategy.selectOneMessageQueue(tpInfo, lastBrokerName); } } 负载均衡消息选择策略实现\npackage org.apache.rocketmq.client.latency; public class MQFaultStrategy { private final static InternalLogger log = ClientLogger.getLog(); private final LatencyFaultTolerance\u0026lt;String\u0026gt; latencyFaultTolerance = new LatencyFaultToleranceImpl(); // 发送消息延迟容错开关 private boolean sendLatencyFaultEnable = false; // 消息发送时长的延迟级别数组 private long[] latencyMax = {50L, 100L, 550L, 1000L, 2000L, 3000L, 15000L}; // broker预估不可用时长级别数组 private long[] notAvailableDuration = {0L, 0L, 30000L, 60000L, 120000L, 180000L, 600000L}; /** * 负载均衡消息选择策略 */ public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) { if (this.sendLatencyFaultEnable) { try { // 获取可用队列 int index = tpInfo.getSendWhichQueue().getAndIncrement(); for (int i = 0; i \u0026lt; tpInfo.getMessageQueueList().size(); i++) { int pos = Math.abs(index++) % tpInfo.getMessageQueueList().size(); if (pos \u0026lt; 0) pos = 0; MessageQueue mq = tpInfo.getMessageQueueList().get(pos); if (latencyFaultTolerance.isAvailable(mq.getBrokerName())) { if (null == lastBrokerName || mq.getBrokerName().equals(lastBrokerName)) return mq; } } // 选择最优队列 final String notBestBroker = latencyFaultTolerance.pickOneAtLeast(); int writeQueueNums = tpInfo.getQueueIdByBroker(notBestBroker); if (writeQueueNums \u0026gt; 0) { final MessageQueue mq = tpInfo.selectOneMessageQueue(); if (notBestBroker != null) { mq.setBrokerName(notBestBroker); mq.setQueueId(tpInfo.getSendWhichQueue().getAndIncrement() % writeQueueNums); } return mq; } else { latencyFaultTolerance.remove(notBestBroker); } } catch (Exception e) { log.error(\u0026#34;Error occurred when selecting message queue\u0026#34;, e); } return tpInfo.selectOneMessageQueue(); } return tpInfo.selectOneMessageQueue(lastBrokerName); } } 更新broker可用信息（容错）实现\npackage org.apache.rocketmq.client.impl.producer; public class DefaultMQProducerImpl implements MQProducerInner { ... public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) { this.mqFaultStrategy.updateFaultItem(brokerName, currentLatency, isolation); } } 与负载均衡消息选择策略位于相同类中\npackage org.apache.rocketmq.client.latency; public class MQFaultStrategy { public void updateFaultItem(final String brokerName, final long currentLatency, boolean isolation) { if (this.sendLatencyFaultEnable) { long duration = computeNotAvailableDuration(isolation ? 30000 : currentLatency); this.latencyFaultTolerance.updateFaultItem(brokerName, currentLatency, duration); } } } 消息发送实现\npackage org.apache.rocketmq.client.impl.producer; public class DefaultMQProducerImpl implements MQProducerInner { ... private SendResult sendKernelImpl(final Message msg, final MessageQueue mq, final CommunicationMode communicationMode, final SendCallback sendCallback, final TopicPublishInfo topicPublishInfo, final long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException { long beginStartTime = System.currentTimeMillis(); // 获得地址 String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName()); if (null == brokerAddr) { tryToFindTopicPublishInfo(mq.getTopic()); brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName()); } SendMessageContext context = null; if (brokerAddr != null) { brokerAddr = MixAll.brokerVIPChannel(this.defaultMQProducer.isSendMessageWithVIPChannel(), brokerAddr); // 记录消息内容，因为下面逻辑可能改变消息内容，例如消息压缩 byte[] prevBody = msg.getBody(); try { //for MessageBatch,ID has been set in the generating process if (!(msg instanceof MessageBatch)) { MessageClientIDSetter.setUniqID(msg); } int sysFlag = 0; boolean msgBodyCompressed = false; if (this.tryToCompressMessage(msg)) { sysFlag |= MessageSysFlag.COMPRESSED_FLAG; msgBodyCompressed = true; } final String tranMsg = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED); if (tranMsg != null \u0026amp;\u0026amp; Boolean.parseBoolean(tranMsg)) { sysFlag |= MessageSysFlag.TRANSACTION_PREPARED_TYPE; } // 消息发送前校验 if (hasCheckForbiddenHook()) { CheckForbiddenContext checkForbiddenContext = new CheckForbiddenContext(); checkForbiddenContext.setNameSrvAddr(this.defaultMQProducer.getNamesrvAddr()); checkForbiddenContext.setGroup(this.defaultMQProducer.getProducerGroup()); checkForbiddenContext.setCommunicationMode(communicationMode); checkForbiddenContext.setBrokerAddr(brokerAddr); checkForbiddenContext.setMessage(msg); checkForbiddenContext.setMq(mq); checkForbiddenContext.setUnitMode(this.isUnitMode()); this.executeCheckForbiddenHook(checkForbiddenContext); } // 消息发送前回调处理 if (this.hasSendMessageHook()) { context = new SendMessageContext(); context.setProducer(this); context.setProducerGroup(this.defaultMQProducer.getProducerGroup()); context.setCommunicationMode(communicationMode); context.setBornHost(this.defaultMQProducer.getClientIP()); context.setBrokerAddr(brokerAddr); context.setMessage(msg); context.setMq(mq); String isTrans = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED); if (isTrans != null \u0026amp;\u0026amp; isTrans.equals(\u0026#34;true\u0026#34;)) { context.setMsgType(MessageType.Trans_Msg_Half); } if (msg.getProperty(\u0026#34;__STARTDELIVERTIME\u0026#34;) != null || msg.getProperty(MessageConst.PROPERTY_DELAY_TIME_LEVEL) != null) { context.setMsgType(MessageType.Delay_Msg); } this.executeSendMessageHookBefore(context); } // 构建消息发送请求头 SendMessageRequestHeader requestHeader = new SendMessageRequestHeader(); requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup()); requestHeader.setTopic(msg.getTopic()); requestHeader.setDefaultTopic(this.defaultMQProducer.getCreateTopicKey()); requestHeader.setDefaultTopicQueueNums(this.defaultMQProducer.getDefaultTopicQueueNums()); requestHeader.setQueueId(mq.getQueueId()); requestHeader.setSysFlag(sysFlag); requestHeader.setBornTimestamp(System.currentTimeMillis()); requestHeader.setFlag(msg.getFlag()); requestHeader.setProperties(MessageDecoder.messageProperties2String(msg.getProperties())); requestHeader.setReconsumeTimes(0); requestHeader.setUnitMode(this.isUnitMode()); requestHeader.setBatch(msg instanceof MessageBatch); if (requestHeader.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) { String reconsumeTimes = MessageAccessor.getReconsumeTime(msg); if (reconsumeTimes != null) { requestHeader.setReconsumeTimes(Integer.valueOf(reconsumeTimes)); MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_RECONSUME_TIME); } String maxReconsumeTimes = MessageAccessor.getMaxReconsumeTimes(msg); if (maxReconsumeTimes != null) { requestHeader.setMaxReconsumeTimes(Integer.valueOf(maxReconsumeTimes)); MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_MAX_RECONSUME_TIMES); } } SendResult sendResult = null; // 消息发送逻辑 switch (communicationMode) { // 异步发送 case ASYNC: Message tmpMessage = msg; if (msgBodyCompressed) { //If msg body was compressed, msgbody should be reset using prevBody. //Clone new message using commpressed message body and recover origin massage. //Fix bug:https://github.com/apache/rocketmq-externals/issues/66 tmpMessage = MessageAccessor.cloneMessage(msg); msg.setBody(prevBody); } long costTimeAsync = System.currentTimeMillis() - beginStartTime; if (timeout \u0026lt; costTimeAsync) { throw new RemotingTooMuchRequestException(\u0026#34;sendKernelImpl call timeout\u0026#34;); } sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage( brokerAddr, mq.getBrokerName(), tmpMessage, requestHeader, timeout - costTimeAsync, communicationMode, sendCallback, topicPublishInfo, this.mQClientFactory, this.defaultMQProducer.getRetryTimesWhenSendAsyncFailed(), context, this); break; case ONEWAY: // 同步发送 case SYNC: long costTimeSync = System.currentTimeMillis() - beginStartTime; if (timeout \u0026lt; costTimeSync) { throw new RemotingTooMuchRequestException(\u0026#34;sendKernelImpl call timeout\u0026#34;); } sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage( brokerAddr, mq.getBrokerName(), msg, requestHeader, timeout - costTimeSync, communicationMode, context, this); break; default: assert false; break; } if (this.hasSendMessageHook()) { // 发送结果放入上下文后返回 context.setSendResult(sendResult); this.executeSendMessageHookAfter(context); } return sendResult; } catch (RemotingException e) { if (this.hasSendMessageHook()) { context.setException(e); this.executeSendMessageHookAfter(context); } throw e; } catch (MQBrokerException e) { if (this.hasSendMessageHook()) { context.setException(e); this.executeSendMessageHookAfter(context); } throw e; } catch (InterruptedException e) { if (this.hasSendMessageHook()) { context.setException(e); this.executeSendMessageHookAfter(context); } throw e; } finally { msg.setBody(prevBody); } } throw new MQClientException(\u0026#34;The broker[\u0026#34; + mq.getBrokerName() + \u0026#34;] not exist\u0026#34;, null); } } 使用MQClientAPIImpl真正发起Netty网络请求\npackage org.apache.rocketmq.client.impl; public class MQClientAPIImpl { ... public SendResult sendMessage( final String addr, final String brokerName, final Message msg, final SendMessageRequestHeader requestHeader, final long timeoutMillis, final CommunicationMode communicationMode, final SendMessageContext context, final DefaultMQProducerImpl producer ) throws RemotingException, MQBrokerException, InterruptedException { return sendMessage(addr, brokerName, msg, requestHeader, timeoutMillis, communicationMode, null, null, null, 0, context, producer); } /** * 封装 RemotingCommand 对象后真正发起网络请求 */ public SendResult sendMessage( final String addr, final String brokerName, final Message msg, final SendMessageRequestHeader requestHeader, final long timeoutMillis, final CommunicationMode communicationMode, final SendCallback sendCallback, final TopicPublishInfo topicPublishInfo, final MQClientInstance instance, final int retryTimesWhenSendFailed, final SendMessageContext context, final DefaultMQProducerImpl producer ) throws RemotingException, MQBrokerException, InterruptedException { long beginStartTime = System.currentTimeMillis(); RemotingCommand request = null; if (sendSmartMsg || msg instanceof MessageBatch) { SendMessageRequestHeaderV2 requestHeaderV2 = SendMessageRequestHeaderV2.createSendMessageRequestHeaderV2(requestHeader); request = RemotingCommand.createRequestCommand(msg instanceof MessageBatch ? RequestCode.SEND_BATCH_MESSAGE : RequestCode.SEND_MESSAGE_V2, requestHeaderV2); } else { request = RemotingCommand.createRequestCommand(RequestCode.SEND_MESSAGE, requestHeader); } // 设置请求体 request.setBody(msg.getBody()); switch (communicationMode) { case ONEWAY: this.remotingClient.invokeOneway(addr, request, timeoutMillis); return null; case ASYNC: final AtomicInteger times = new AtomicInteger(); long costTimeAsync = System.currentTimeMillis() - beginStartTime; if (timeoutMillis \u0026lt; costTimeAsync) { throw new RemotingTooMuchRequestException(\u0026#34;sendMessage call timeout\u0026#34;); } this.sendMessageAsync(addr, brokerName, msg, timeoutMillis - costTimeAsync, request, sendCallback, topicPublishInfo, instance, retryTimesWhenSendFailed, times, context, producer); return null; case SYNC: long costTimeSync = System.currentTimeMillis() - beginStartTime; if (timeoutMillis \u0026lt; costTimeSync) { throw new RemotingTooMuchRequestException(\u0026#34;sendMessage call timeout\u0026#34;); } return this.sendMessageSync(addr, brokerName, msg, timeoutMillis - costTimeSync, request); default: assert false; break; } return null; } private SendResult sendMessageSync( final String addr, final String brokerName, final Message msg, final long timeoutMillis, final RemotingCommand request ) throws RemotingException, MQBrokerException, InterruptedException { // 同步发网络请求，返回响应结果 RemotingCommand response = this.remotingClient.invokeSync(addr, request, timeoutMillis); assert response != null; return this.processSendResponse(brokerName, msg, response); } } 底层Netty通信 类图\nRemotingClient源码 package org.apache.rocketmq.remoting; public interface RemotingClient extends RemotingService { /** * 更新协调服务地址列表 * @param addrs */ void updateNameServerAddressList(final List\u0026lt;String\u0026gt; addrs); /** * 更新协调服务地址列表 * @return */ List\u0026lt;String\u0026gt; getNameServerAddressList(); /** * 同步消息发送 * @param addr * @param request * @param timeoutMillis * @return * @throws InterruptedException * @throws RemotingConnectException * @throws RemotingSendRequestException * @throws RemotingTimeoutException */ RemotingCommand invokeSync(final String addr, final RemotingCommand request, final long timeoutMillis) throws InterruptedException, RemotingConnectException, RemotingSendRequestException, RemotingTimeoutException; /** * 异步消息发送 * @param addr * @param request * @param timeoutMillis * @param invokeCallback * @throws InterruptedException * @throws RemotingConnectException * @throws RemotingTooMuchRequestException * @throws RemotingTimeoutException * @throws RemotingSendRequestException */ void invokeAsync(final String addr, final RemotingCommand request, final long timeoutMillis, final InvokeCallback invokeCallback) throws InterruptedException, RemotingConnectException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException; /** * 单次消息发送 * @param addr * @param request * @param timeoutMillis * @throws InterruptedException * @throws RemotingConnectException * @throws RemotingTooMuchRequestException * @throws RemotingTimeoutException * @throws RemotingSendRequestException */ void invokeOneway(final String addr, final RemotingCommand request, final long timeoutMillis) throws InterruptedException, RemotingConnectException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException; /** *用于注册一些底层通信服务，比如进行元数据信息的同步工作，commitlog的同步等 * @param requestCode 底层业务通信规则码 * @param processor 注册器（单线程） * @param executor 线程池，用于执行注册器的业务逻辑 */ void registerProcessor(final int requestCode, final NettyRequestProcessor processor, final ExecutorService executor); /** * 回调函数执行线程设置，用于生产者发送消息后的回调函数 * @param callbackExecutor */ void setCallbackExecutor(final ExecutorService callbackExecutor); /** * 获取回调池：NameServer地址 * @return */ ExecutorService getCallbackExecutor(); /** * 用于远程服务地址判断是否可以进行通信 * @param addr * @return */ boolean isChannelWritable(final String addr); } registerProcessor方法在实现类中用于初始化时进行注册逻辑\npackage org.apache.rocketmq.client.impl; public class MQClientAPIImpl { /** * Netty 服务启动时初始化工作 * @param nettyClientConfig 配置信息 * @param clientRemotingProcessor * @param rpcHook * @param clientConfig */ public MQClientAPIImpl(final NettyClientConfig nettyClientConfig, final ClientRemotingProcessor clientRemotingProcessor, RPCHook rpcHook, final ClientConfig clientConfig) { this.clientConfig = clientConfig; topAddressing = new TopAddressing(MixAll.getWSAddr(), clientConfig.getUnitName()); this.remotingClient = new NettyRemotingClient(nettyClientConfig, null); this.clientRemotingProcessor = clientRemotingProcessor; this.remotingClient.registerRPCHook(rpcHook); this.remotingClient.registerProcessor(RequestCode.CHECK_TRANSACTION_STATE, this.clientRemotingProcessor, null); this.remotingClient.registerProcessor(RequestCode.NOTIFY_CONSUMER_IDS_CHANGED, this.clientRemotingProcessor, null); this.remotingClient.registerProcessor(RequestCode.RESET_CONSUMER_CLIENT_OFFSET, this.clientRemotingProcessor, null); this.remotingClient.registerProcessor(RequestCode.GET_CONSUMER_STATUS_FROM_CLIENT, this.clientRemotingProcessor, null); this.remotingClient.registerProcessor(RequestCode.GET_CONSUMER_RUNNING_INFO, this.clientRemotingProcessor, null); this.remotingClient.registerProcessor(RequestCode.CONSUME_MESSAGE_DIRECTLY, this.clientRemotingProcessor, null); } ... } setCallbackExecutor方法用于回调函数执行线程设置，用于生产者发送消息后的回调线程池，例如DefaultMQProducerImpl中的使用方式\npackage org.apache.rocketmq.client.impl.producer; public class DefaultMQProducerImpl implements MQProducerInner { ... public void setCallbackExecutor(final ExecutorService callbackExecutor) { this.mQClientFactory.getMQClientAPIImpl().getRemotingClient().setCallbackExecutor(callbackExecutor); } /** * It will be removed at 4.4.0 cause for exception handling and the wrong Semantics of timeout. * A new one will be provided in next version * @param msg * @param sendCallback * @param timeout the \u0026lt;code\u0026gt;sendCallback\u0026lt;/code\u0026gt; will be invoked at most time * @throws RejectedExecutionException */ @Deprecated public void send(final Message msg, final SendCallback sendCallback, final long timeout) throws MQClientException, RemotingException, InterruptedException { final long beginStartTime = System.currentTimeMillis(); // 获取回调线程池 ExecutorService executor = this.getCallbackExecutor(); try { executor.submit(new Runnable() { @Override public void run() { long costTime = System.currentTimeMillis() - beginStartTime; if (timeout \u0026gt; costTime) { try { sendDefaultImpl(msg, CommunicationMode.ASYNC, sendCallback, timeout - costTime); } catch (Exception e) { sendCallback.onException(e); } } else { sendCallback.onException( new RemotingTooMuchRequestException(\u0026#34;DEFAULT ASYNC send call timeout\u0026#34;)); } } }); } catch (RejectedExecutionException e) { throw new MQClientException(\u0026#34;executor rejected \u0026#34;, e); } } ... } RemotingServer源码 package org.apache.rocketmq.remoting; public interface RemotingServer extends RemotingService { /** * 用于注册一些底层通信服务，比如进行元数据信息的同步工作，commitlog的同步等 * @param requestCode 底层业务通信规则码 * @param processor 注册器（单线程） * @param executor 线程池，用于执行注册器的业务逻辑 */ void registerProcessor(final int requestCode, final NettyRequestProcessor processor, final ExecutorService executor); /** * 默认注册方法：比如注册：{@link org.apache.rocketmq.broker.processor.AdminBrokerProcessor}管理Broker工作线程或者namesrv有关的流程器， * 比如同步地址、路由topic等信息 * @param processor * @param executor */ void registerDefaultProcessor(final NettyRequestProcessor processor, final ExecutorService executor); /** * 获取端口 * @return */ int localListenPort(); /** * 根据业务规则获取Pair\u0026lt;执行器，线程池\u0026gt; * @param requestCode 业务规则码 * @return */ Pair\u0026lt;NettyRequestProcessor, ExecutorService\u0026gt; getProcessorPair(final int requestCode); /** * 同步消息发送 * @param channel * @param request * @param timeoutMillis * @return * @throws InterruptedException * @throws RemotingSendRequestException * @throws RemotingTimeoutException */ RemotingCommand invokeSync(final Channel channel, final RemotingCommand request, final long timeoutMillis) throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException; /** * 异步消息发送 * @param channel * @param request * @param timeoutMillis * @param invokeCallback * @throws InterruptedException * @throws RemotingTooMuchRequestException * @throws RemotingTimeoutException * @throws RemotingSendRequestException */ void invokeAsync(final Channel channel, final RemotingCommand request, final long timeoutMillis, final InvokeCallback invokeCallback) throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException; /** * 单向消息发送 * @param channel * @param request * @param timeoutMillis * @throws InterruptedException * @throws RemotingTooMuchRequestException * @throws RemotingTimeoutException * @throws RemotingSendRequestException */ void invokeOneway(final Channel channel, final RemotingCommand request, final long timeoutMillis) throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException; } NettyRemotingServer源码 其唯一实现类NettyRemotingServer，主要实现Netty编程模型\npackage org.apache.rocketmq.remoting.netty; public class NettyRemotingServer extends NettyRemotingAbstract implements RemotingServer { private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING); /** * NettyServer辅助类 */ private final ServerBootstrap serverBootstrap; /** * selector组（用于注册并监听event） */ private final EventLoopGroup eventLoopGroupSelector; /** * 主线程组（用于接收TCP连接） */ private final EventLoopGroup eventLoopGroupBoss; /** * NettyServer配置 */ private final NettyServerConfig nettyServerConfig; /** * 主要使用在registerProcessor方法中 */ private final ExecutorService publicExecutor; /** * channel事件监听 */ private final ChannelEventListener channelEventListener; public NettyRemotingServer(final NettyServerConfig nettyServerConfig, final ChannelEventListener channelEventListener) { super(nettyServerConfig.getServerOnewaySemaphoreValue(), nettyServerConfig.getServerAsyncSemaphoreValue()); this.serverBootstrap = new ServerBootstrap(); this.nettyServerConfig = nettyServerConfig; this.channelEventListener = channelEventListener; int publicThreadNums = nettyServerConfig.getServerCallbackExecutorThreads(); if (publicThreadNums \u0026lt;= 0) { publicThreadNums = 4; } this.publicExecutor = Executors.newFixedThreadPool(publicThreadNums, new ThreadFactory() { private AtomicInteger threadIndex = new AtomicInteger(0); @Override public Thread newThread(Runnable r) { return new Thread(r, \u0026#34;NettyServerPublicExecutor_\u0026#34; + this.threadIndex.incrementAndGet()); } }); this.eventLoopGroupBoss = new NioEventLoopGroup(1, new ThreadFactory() { private AtomicInteger threadIndex = new AtomicInteger(0); @Override public Thread newThread(Runnable r) { return new Thread(r, String.format(\u0026#34;NettyBoss_%d\u0026#34;, this.threadIndex.incrementAndGet())); } }); // 如果是Linux平台，并且开启了native epoll，就用EpollEventLoopGroup，这个也就是用JNI，调用c写的epoll；否则就用java NIO的NioEventLoopGroup if (useEpoll()) { this.eventLoopGroupSelector = new EpollEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() { private AtomicInteger threadIndex = new AtomicInteger(0); private int threadTotal = nettyServerConfig.getServerSelectorThreads(); @Override public Thread newThread(Runnable r) { return new Thread(r, String.format(\u0026#34;NettyServerEPOLLSelector_%d_%d\u0026#34;, threadTotal, this.threadIndex.incrementAndGet())); } }); } else { this.eventLoopGroupSelector = new NioEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() { private AtomicInteger threadIndex = new AtomicInteger(0); private int threadTotal = nettyServerConfig.getServerSelectorThreads(); @Override public Thread newThread(Runnable r) { return new Thread(r, String.format(\u0026#34;NettyServerNIOSelector_%d_%d\u0026#34;, threadTotal, this.threadIndex.incrementAndGet())); } }); } loadSslContext(); } @Override public void start() { // 根据配置创建工作线程组 this.defaultEventExecutorGroup = new DefaultEventExecutorGroup( nettyServerConfig.getServerWorkerThreads(), new ThreadFactory() { private AtomicInteger threadIndex = new AtomicInteger(0); @Override public Thread newThread(Runnable r) { return new Thread(r, \u0026#34;NettyServerCodecThread_\u0026#34; + this.threadIndex.incrementAndGet()); } }); // Netty编程模型 ServerBootstrap childHandler = this.serverBootstrap.group(this.eventLoopGroupBoss, this.eventLoopGroupSelector) .channel(useEpoll() ? EpollServerSocketChannel.class : NioServerSocketChannel.class) // sync + accept = backlog .option(ChannelOption.SO_BACKLOG, 1024) // 允许重复使用本地地址和端口 .option(ChannelOption.SO_REUSEADDR, true) .option(ChannelOption.SO_KEEPALIVE, false) .childOption(ChannelOption.TCP_NODELAY, true) .childOption(ChannelOption.SO_SNDBUF, nettyServerConfig.getServerSocketSndBufSize()) .childOption(ChannelOption.SO_RCVBUF, nettyServerConfig.getServerSocketRcvBufSize()) .localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort())) .childHandler(new ChannelInitializer\u0026lt;SocketChannel\u0026gt;() { @Override public void initChannel(SocketChannel ch) throws Exception { ch.pipeline() .addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME, new HandshakeHandler(TlsSystemConfig.tlsMode)) .addLast(defaultEventExecutorGroup, new NettyEncoder(), new NettyDecoder(), new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()), new NettyConnectManageHandler(), new NettyServerHandler() ); } }); if (nettyServerConfig.isServerPooledByteBufAllocatorEnable()) { childHandler.childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT); } try { ChannelFuture sync = this.serverBootstrap.bind().sync(); InetSocketAddress addr = (InetSocketAddress) sync.channel().localAddress(); this.port = addr.getPort(); } catch (InterruptedException e1) { throw new RuntimeException(\u0026#34;this.serverBootstrap.bind().sync() InterruptedException\u0026#34;, e1); } if (this.channelEventListener != null) { this.nettyEventExecutor.start(); } this.timer.scheduleAtFixedRate(new TimerTask() { @Override public void run() { try { NettyRemotingServer.this.scanResponseTable(); } catch (Throwable e) { log.error(\u0026#34;scanResponseTable exception\u0026#34;, e); } } }, 1000 * 3, 1000); } ... } NettyRemotingAbstract源码 NettyRemotingAbstract具体实现三种发送方法\npackage org.apache.rocketmq.remoting.netty; public abstract class NettyRemotingAbstract { /** * 异步调用具体实现 * @param channel * @param request * @param timeoutMillis * @param invokeCallback * @throws InterruptedException * @throws RemotingTooMuchRequestException * @throws RemotingTimeoutException * @throws RemotingSendRequestException */ public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis, final InvokeCallback invokeCallback) throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException { long beginStartTime = System.currentTimeMillis(); final int opaque = request.getOpaque(); boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS); if (acquired) { final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync); long costTime = System.currentTimeMillis() - beginStartTime; if (timeoutMillis \u0026lt; costTime) { throw new RemotingTooMuchRequestException(\u0026#34;invokeAsyncImpl call timeout\u0026#34;); } final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis - costTime, invokeCallback, once); this.responseTable.put(opaque, responseFuture); try { channel.writeAndFlush(request).addListener(new ChannelFutureListener() { @Override public void operationComplete(ChannelFuture f) throws Exception { if (f.isSuccess()) { responseFuture.setSendRequestOK(true); return; } requestFail(opaque); log.warn(\u0026#34;send a request command to channel \u0026lt;{}\u0026gt; failed.\u0026#34;, RemotingHelper.parseChannelRemoteAddr(channel)); } }); } catch (Exception e) { responseFuture.release(); log.warn(\u0026#34;send a request command to channel \u0026lt;\u0026#34; + RemotingHelper.parseChannelRemoteAddr(channel) + \u0026#34;\u0026gt; Exception\u0026#34;, e); throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e); } } else { if (timeoutMillis \u0026lt;= 0) { throw new RemotingTooMuchRequestException(\u0026#34;invokeAsyncImpl invoke too fast\u0026#34;); } else { String info = String.format(\u0026#34;invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d\u0026#34;, timeoutMillis, this.semaphoreAsync.getQueueLength(), this.semaphoreAsync.availablePermits() ); log.warn(info); throw new RemotingTimeoutException(info); } } } /** * 单向发送具体实现 * @param channel * @param request * @param timeoutMillis * @throws InterruptedException * @throws RemotingTooMuchRequestException * @throws RemotingTimeoutException * @throws RemotingSendRequestException */ public void invokeOnewayImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis) throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException { request.markOnewayRPC(); boolean acquired = this.semaphoreOneway.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS); if (acquired) { final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreOneway); try { channel.writeAndFlush(request).addListener(new ChannelFutureListener() { @Override public void operationComplete(ChannelFuture f) throws Exception { once.release(); if (!f.isSuccess()) { log.warn(\u0026#34;send a request command to channel \u0026lt;\u0026#34; + channel.remoteAddress() + \u0026#34;\u0026gt; failed.\u0026#34;); } } }); } catch (Exception e) { once.release(); log.warn(\u0026#34;write send a request command to channel \u0026lt;\u0026#34; + channel.remoteAddress() + \u0026#34;\u0026gt; failed.\u0026#34;); throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e); } } else { if (timeoutMillis \u0026lt;= 0) { throw new RemotingTooMuchRequestException(\u0026#34;invokeOnewayImpl invoke too fast\u0026#34;); } else { String info = String.format( \u0026#34;invokeOnewayImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d\u0026#34;, timeoutMillis, this.semaphoreOneway.getQueueLength(), this.semaphoreOneway.availablePermits() ); log.warn(info); throw new RemotingTimeoutException(info); } } } /** * 同步调用具体实现 * @param channel * @param request * @param timeoutMillis * @return * @throws InterruptedException * @throws RemotingSendRequestException * @throws RemotingTimeoutException */ public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis) throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException { final int opaque = request.getOpaque(); try { final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null); this.responseTable.put(opaque, responseFuture); final SocketAddress addr = channel.remoteAddress(); channel.writeAndFlush(request).addListener(new ChannelFutureListener() { @Override public void operationComplete(ChannelFuture f) throws Exception { if (f.isSuccess()) { responseFuture.setSendRequestOK(true); return; } else { responseFuture.setSendRequestOK(false); } responseTable.remove(opaque); responseFuture.setCause(f.cause()); responseFuture.putResponse(null); log.warn(\u0026#34;send a request command to channel \u0026lt;\u0026#34; + addr + \u0026#34;\u0026gt; failed.\u0026#34;); } }); RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis); if (null == responseCommand) { if (responseFuture.isSendRequestOK()) { throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis, responseFuture.getCause()); } else { throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause()); } } return responseCommand; } finally { this.responseTable.remove(opaque); } } ... } Netty编码解码源码解析——RemotingCommand源码 netty消息数据包结构\nRemotingCommand中编码、解码方法体现了其数据包格式和用法\npackage org.apache.rocketmq.remoting.protocol; public class RemotingCommand { public ByteBuffer encode() { // 1\u0026gt; header length size int length = 4; // 2\u0026gt; header data length byte[] headerData = this.headerEncode(); length += headerData.length; // 3\u0026gt; body data length if (this.body != null) { length += body.length; } ByteBuffer result = ByteBuffer.allocate(4 + length); // length result.putInt(length); // header length result.put(markProtocolType(headerData.length, serializeTypeCurrentRPC)); // header data result.put(headerData); // body data; if (this.body != null) { result.put(this.body); } result.flip(); return result; } public static RemotingCommand decode(final ByteBuffer byteBuffer) { int length = byteBuffer.limit(); int oriHeaderLen = byteBuffer.getInt(); int headerLength = getHeaderLength(oriHeaderLen); byte[] headerData = new byte[headerLength]; // 此处使用的是java nio buffer byteBuffer.get(headerData); // header封装，长度+类型 对应数据包结构第二位 RemotingCommand cmd = headerDecode(headerData, getProtocolType(oriHeaderLen)); int bodyLength = length - 4 - headerLength; byte[] bodyData = null; if (bodyLength \u0026gt; 0) { bodyData = new byte[bodyLength]; byteBuffer.get(bodyData); } cmd.body = bodyData; return cmd; } ... } 其支持的序列化类型有两个\nSerializeType序列化类型枚举\npackage org.apache.rocketmq.remoting.protocol; public enum SerializeType { JSON((byte) 0), ROCKETMQ((byte) 1); ... } 在NettyRemotingClient中的Netty编程模型也有编码、解码配置，其使用的是netty buffer\npackage org.apache.rocketmq.remoting.netty; public class NettyRemotingClient extends NettyRemotingAbstract implements RemotingClient { @Override public void start() { this.defaultEventExecutorGroup = new DefaultEventExecutorGroup( nettyClientConfig.getClientWorkerThreads(), new ThreadFactory() { private AtomicInteger threadIndex = new AtomicInteger(0); @Override public Thread newThread(Runnable r) { return new Thread(r, \u0026#34;NettyClientWorkerThread_\u0026#34; + this.threadIndex.incrementAndGet()); } }); Bootstrap handler = this.bootstrap.group(this.eventLoopGroupWorker).channel(NioSocketChannel.class) .option(ChannelOption.TCP_NODELAY, true) .option(ChannelOption.SO_KEEPALIVE, false) .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, nettyClientConfig.getConnectTimeoutMillis()) .option(ChannelOption.SO_SNDBUF, nettyClientConfig.getClientSocketSndBufSize()) .option(ChannelOption.SO_RCVBUF, nettyClientConfig.getClientSocketRcvBufSize()) .handler(new ChannelInitializer\u0026lt;SocketChannel\u0026gt;() { @Override public void initChannel(SocketChannel ch) throws Exception { ChannelPipeline pipeline = ch.pipeline(); if (nettyClientConfig.isUseTLS()) { if (null != sslContext) { pipeline.addFirst(defaultEventExecutorGroup, \u0026#34;sslHandler\u0026#34;, sslContext.newHandler(ch.alloc())); log.info(\u0026#34;Prepend SSL handler\u0026#34;); } else { log.warn(\u0026#34;Connections are insecure as SSLContext is null!\u0026#34;); } } pipeline.addLast( // 使用工作线程 defaultEventExecutorGroup, // 编码 new NettyEncoder(), // 解码 new NettyDecoder(), new IdleStateHandler(0, 0, nettyClientConfig.getClientChannelMaxIdleTimeSeconds()), // 连接管理器 new NettyConnectManageHandler(), new NettyClientHandler()); } }); this.timer.scheduleAtFixedRate(new TimerTask() { @Override public void run() { try { NettyRemotingClient.this.scanResponseTable(); } catch (Throwable e) { log.error(\u0026#34;scanResponseTable exception\u0026#34;, e); } } }, 1000 * 3, 1000); if (this.channelEventListener != null) { this.nettyEventExecutor.start(); } } } NettyEncoder源码\npackage org.apache.rocketmq.remoting.netty; import io.netty.buffer.ByteBuf; import java.nio.ByteBuffer; public class NettyEncoder extends MessageToByteEncoder\u0026lt;RemotingCommand\u0026gt; { private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING); @Override public void encode(ChannelHandlerContext ctx, RemotingCommand remotingCommand, ByteBuf out) throws Exception { try { ByteBuffer header = remotingCommand.encodeHeader(); out.writeBytes(header); byte[] body = remotingCommand.getBody(); if (body != null) { out.writeBytes(body); } } catch (Exception e) { log.error(\u0026#34;encode exception, \u0026#34; + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e); if (remotingCommand != null) { log.error(remotingCommand.toString()); } RemotingUtil.closeChannel(ctx.channel()); } } } NettyDecoder源码\n其继承LengthFieldBasedFrameDecoder，支持灵活配置编解码参数\npackage org.apache.rocketmq.remoting.netty; import io.netty.buffer.ByteBuf; import io.netty.handler.codec.LengthFieldBasedFrameDecoder; import java.nio.ByteBuffer; public class NettyDecoder extends LengthFieldBasedFrameDecoder { private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING); private static final int FRAME_MAX_LENGTH = Integer.parseInt(System.getProperty(\u0026#34;com.rocketmq.remoting.frameMaxLength\u0026#34;, \u0026#34;16777216\u0026#34;)); public NettyDecoder() { super(FRAME_MAX_LENGTH, 0, 4, 0, 4); } @Override public Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception { ByteBuf frame = null; try { frame = (ByteBuf) super.decode(ctx, in); if (null == frame) { return null; } ByteBuffer byteBuffer = frame.nioBuffer(); return RemotingCommand.decode(byteBuffer); } catch (Exception e) { log.error(\u0026#34;decode exception, \u0026#34; + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e); RemotingUtil.closeChannel(ctx.channel()); } finally { if (null != frame) { frame.release(); } } return null; } } 消息返回状态详解 SEND_OK：消息发送成功。 FLUSH_DISK_TIMEOUT：消息发送成功，但消息刷盘失败，此时服务器宕机则消息丢失 FLUSH_SLAVE_TIMEOUT：主从同步消息时超时，从节点已经收到消息，若此时从节点宕机，则从节点消息丢失 SLAVE_NOT_AVAILABLE：从节点不可用，此时主节点宕机，则消息丢失 对于可靠消息投递，后三种情况均应该考虑补偿。\n延迟消息投递 场景：消息发送到Broker后，要特定的时间才会被Consumer消费\n特点：目前只支持固定精度的定时消息\n固定精度消息原因：支持任意精度消息时，对消息持久化、排序产生较大性能开销\n使用方法：对message设置setDelayTimeLevel级别即可\n核心类：MessageStoreConfig和ScheduleMessageService\nMessageStoreConfig源码\npackage org.apache.rocketmq.store.config; public class MessageStoreConfig { // 延时投递级别，固定精度 private String messageDelayLevel = \u0026#34;1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h\u0026#34;; ... } 自定义消息发送规则 场景：把消息发送到指定的队列（Message Queue）\n处理类：MessageQueueSelector\n用法：调用send(Message msg, MessageQueueSelector selector, Object arg)方法即可\n实现代码片段，指定发送到id为1的队列中\nSendResult sendResult = producer.send(msg, new MessageQueueSelector() { // 外部方法的arg会传递到Object中 @Override public MessageQueue select(List\u0026lt;MessageQueue\u0026gt; list, Message message, Object o) { Integer queueNumber = (Integer) o; return list.get(queueNumber); } },1); ","date":"2020-12-23T19:09:00+08:00","permalink":"/archives/253/","title":"RocketMQ学习笔记3——生产者核心"},{"content":"生产者模型使用 步骤\n创建生产者对象DefaultMQProducer 设置NamesrvAddr 启动生产者服务 创建消息并发送 简单消息发送\n创建测试主题\n常量定义\npublic class Const { public static final String NAMESRV_ADDR_SINGLE = \u0026#34;10.139.12.52:9876\u0026#34;; } public class Producer { public static void main(String[] args) throws MQClientException, RemotingException, MQBrokerException, InterruptedException { DefaultMQProducer producer = new DefaultMQProducer(\u0026#34;test_quick_producer_name\u0026#34;); producer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE); producer.start(); for(int i = 0 ; i \u0026lt;5; i ++) { //\t1.\t创建消息 Message message = new Message(\u0026#34;test_quick_topic\u0026#34;,\t//\t主题 \u0026#34;TagA\u0026#34;, //\t标签 \u0026#34;key\u0026#34; + i,\t// 用户自定义的key ,唯一的标识 (\u0026#34;Hello RocketMQ\u0026#34; + i).getBytes());\t//\t消息内容实体（byte[]） SendResult sr = producer.send(message, new MessageQueueSelector() { @Override public MessageQueue select(List\u0026lt;MessageQueue\u0026gt; mqs, Message msg, Object arg) { Integer queueNumber = (Integer)arg; return mqs.get(queueNumber); } }, 2); System.err.println(sr); System.err.println(\u0026#34;消息发出: \u0026#34; + sr); } producer.shutdown(); } } 运行结果\nSendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0] 消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA830000, offsetMsgId=0A8B0C3400002A9F0000000000000B5E, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=0] SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1] 消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FA990001, offsetMsgId=0A8B0C3400002A9F0000000000000C20, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=1] SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2] 消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAA10002, offsetMsgId=0A8B0C3400002A9F0000000000000CE2, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=2] SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3] 消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FAAE0003, offsetMsgId=0A8B0C3400002A9F0000000000000DA4, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=3] SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4] 消息发出: SendResult [sendStatus=SEND_OK, msgId=BA01D58B19CC18B4AAC230A7FABA0004, offsetMsgId=0A8B0C3400002A9F0000000000000E66, messageQueue=MessageQueue [topic=test_quick_topic, brokerName=broker-a, queueId=2], queueOffset=4] 10:45:15.129 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10911] result: true 10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:10909] result: true 10:45:15.132 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.139.12.52:9876] result: true 消费者模型使用 步骤\n创建消费者对象DefaultMQPushConsumer 设置NamesrvAddr及消费位置ConsumeFromWhere 进行主题订阅Subscribe 注册监听并消费registerMessageListener 简单消息消费\npublic class Consumer { public static void main(String[] args) throws MQClientException { // 采用Push方式 DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(\u0026#34;test_quick_consumer_name\u0026#34;); consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE); consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET); // 匹配所有Tag consumer.subscribe(\u0026#34;test_quick_topic\u0026#34;, \u0026#34;*\u0026#34;); consumer.registerMessageListener(new MessageListenerConcurrently() { @Override public ConsumeConcurrentlyStatus consumeMessage(List\u0026lt;MessageExt\u0026gt; msgs, ConsumeConcurrentlyContext context) { MessageExt me = msgs.get(0); try { String topic = me.getTopic(); String tags = me.getTags(); String keys = me.getKeys(); String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET); System.err.println(\u0026#34;topic: \u0026#34; + topic + \u0026#34;,tags: \u0026#34; + tags + \u0026#34;, keys: \u0026#34; + keys + \u0026#34;,body: \u0026#34; + msgBody); } catch (Exception e) { e.printStackTrace(); // 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h return ConsumeConcurrentlyStatus.RECONSUME_LATER; } return ConsumeConcurrentlyStatus.CONSUME_SUCCESS; } }); consumer.start(); System.err.println(\u0026#34;consumer start...\u0026#34;); } } 模拟异常重试\npublic class Consumer { public static void main(String[] args) throws MQClientException { // 采用Push方式 DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(\u0026#34;test_quick_consumer_name\u0026#34;); consumer.setNamesrvAddr(Const.NAMESRV_ADDR_SINGLE); consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET); // 匹配所有Tag consumer.subscribe(\u0026#34;test_quick_topic\u0026#34;, \u0026#34;*\u0026#34;); consumer.registerMessageListener(new MessageListenerConcurrently() { @Override public ConsumeConcurrentlyStatus consumeMessage(List\u0026lt;MessageExt\u0026gt; msgs, ConsumeConcurrentlyContext context) { MessageExt me = msgs.get(0); try { String topic = me.getTopic(); String tags = me.getTags(); String keys = me.getKeys(); // 模拟异常 if(keys.equals(\u0026#34;key1\u0026#34;)) { System.err.println(\u0026#34;消息消费失败..\u0026#34;); int a = 1/0; } String msgBody = new String(me.getBody(), RemotingHelper.DEFAULT_CHARSET); System.err.println(\u0026#34;topic: \u0026#34; + topic + \u0026#34;,tags: \u0026#34; + tags + \u0026#34;, keys: \u0026#34; + keys + \u0026#34;,body: \u0026#34; + msgBody); } catch (Exception e) { e.printStackTrace(); // 重发次数 int recousumeTimes = me.getReconsumeTimes(); System.err.println(\u0026#34;recousumeTimes: \u0026#34; + recousumeTimes); // 超过3次不再重发 if(recousumeTimes == 3) { //\t记录日志.... // 做补偿处理 return ConsumeConcurrentlyStatus.CONSUME_SUCCESS; } // 重发时间：messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h return ConsumeConcurrentlyStatus.RECONSUME_LATER; } return ConsumeConcurrentlyStatus.CONSUME_SUCCESS; } }); consumer.start(); System.err.println(\u0026#34;consumer start...\u0026#34;); } } 日志输出\nconsumer start... topic: test_quick_topic,tags: TagA, keys: key0,body: Hello RocketMQ0 消息消费失败.. java.lang.ArithmeticException: / by zero at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40) at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) recousumeTimes: 0 topic: test_quick_topic,tags: TagA, keys: key2,body: Hello RocketMQ2 topic: test_quick_topic,tags: TagA, keys: key3,body: Hello RocketMQ3 topic: test_quick_topic,tags: TagA, keys: key4,body: Hello RocketMQ4 消息消费失败.. java.lang.ArithmeticException: / by zero at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40) at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) recousumeTimes: 1 消息消费失败.. java.lang.ArithmeticException: / by zero at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40) at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) recousumeTimes: 2 消息消费失败.. java.lang.ArithmeticException: / by zero at com.bfxy.rocketmq.quickstart.Consumer$1.consumeMessage(Consumer.java:40) at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) recousumeTimes: 3 四种集群环境 集群环境拓扑图\n分类\n单点模式（不推荐） 主从模式 双主模式（不推荐） 双主双从、多主多从模式 主从集群环境构建 优势：\n主从模式可以保障消息的即时性与可靠性 当投递一条消息后关闭主节点，从节点可以继续提供消费者进行消费，但不能接受消息 主节点上线后会进行消费进度offset同步 基础环境 选择2台服务器进行部署验证\nIP IP GY11-SyOCM33-101 10.139.12.52 主 GY11-SyOCM33-201 10.139.12.53 从 53节点按照单点部署时操作好对应目录和日志配置\n修改主节点配置 52节点修改主节点配置文件broker-a.properties如下\n修改了namesrvAddr\n#所属集群名字 brokerClusterName=rocketmq-cluster #broker 名字，注意此处不同的配置文件填写的不一样 brokerName=broker-a #0 表示 Master，\u0026gt;0 表示 Slave brokerId=0 #nameServer 地址，分号分割 namesrvAddr=10.139.12.52:9876;10.139.12.53:9876 #在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数 defaultTopicQueueNums=4 #是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭 autoCreateTopicEnable=true #是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭 autoCreateSubscriptionGroup=true #Broker 对外服务的监听端口 listenPort=10911 #删除文件时间点，默认凌晨 4 点 deleteWhen=04 #文件保留时间，默认 48 小时 fileReservedTime=120 #commitLog 每个文件的大小默认 1G mapedFileSizeCommitLog=1073741824 #ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整 mapedFileSizeConsumeQueue=300000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #检测物理文件磁盘空间 diskMaxUsedSpaceRatio=88 #存储路径 storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store #commitLog 存储路径 storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog #消费队列存储路径存储路径 storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue #消息索引存储路径 storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index #checkpoint 文件存储路径 storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint #abort 文件存储路径 abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort #限制的消息大小 maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Broker 的角色 #- ASYNC_MASTER 异步复制 Master #- SYNC_MASTER 同步双写 Master #- SLAVE brokerRole=ASYNC_MASTER #刷盘方式 #- ASYNC_FLUSH 异步刷盘 #- SYNC_FLUSH 同步刷盘 flushDiskType=ASYNC_FLUSH #checkTransactionMessageEnable=false #发消息线程池数量 #sendMessageThreadPoolNums=128 #拉消息线程池数量 #pullMessageThreadPoolNums=128 修改从节点配置 修改从节点配置文件broker-a-s.properties如下\n拷贝主节点配置后修改id和brokerRole\n#所属集群名字 brokerClusterName=rocketmq-cluster #broker 名字，注意此处不同的配置文件填写的不一样 brokerName=broker-a #0 表示 Master，\u0026gt;0 表示 Slave brokerId=1 #nameServer 地址，分号分割 namesrvAddr=10.139.12.52:9876;10.139.12.53:9876 #在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数 defaultTopicQueueNums=4 #是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭 autoCreateTopicEnable=true #是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭 autoCreateSubscriptionGroup=true #Broker 对外服务的监听端口 listenPort=10911 #删除文件时间点，默认凌晨 4 点 deleteWhen=04 #文件保留时间，默认 48 小时 fileReservedTime=120 #commitLog 每个文件的大小默认 1G mapedFileSizeCommitLog=1073741824 #ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整 mapedFileSizeConsumeQueue=300000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #检测物理文件磁盘空间 diskMaxUsedSpaceRatio=88 #存储路径 storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store #commitLog 存储路径 storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog #消费队列存储路径存储路径 storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue #消息索引存储路径 storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index #checkpoint 文件存储路径 storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint #abort 文件存储路径 abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort #限制的消息大小 maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Broker 的角色 #- ASYNC_MASTER 异步复制 Master #- SYNC_MASTER 同步双写 Master #- SLAVE brokerRole=SLAVE #刷盘方式 #- ASYNC_FLUSH 异步刷盘 #- SYNC_FLUSH 同步刷盘 flushDiskType=ASYNC_FLUSH #checkTransactionMessageEnable=false #发消息线程池数量 #sendMessageThreadPoolNums=128 #拉消息线程池数量 #pullMessageThreadPoolNums=128 主从配置同步到53节点即可\n启动NameSrv 启动Namesrv（主从节点均执行）\n$ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin $ nohup sh mqnamesrv \u0026amp; 启动Broker 主节点启动\n$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 \u0026amp; 从节点启动\n$ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a-s.properties \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 \u0026amp; 启动控制台 修改控制台配置地址参数为\nrocketmq.config.namesrvAddr=10.139.12.52:9876;10.139.12.53:9876 数据高可用机制故障演练 投递一条消息\n关闭主节点，剩余一个slave\n启动消费者，消费正常\n主节点上线，同步消息位点（offset）使消费端不会重复消费\n","date":"2020-12-23T19:06:00+08:00","permalink":"/archives/252/","title":"RocketMQ学习笔记2——入门"},{"content":"RocketMQ整体认知 优势\n支持集群模型、负载均衡、水平扩展能力 亿级别的消息堆积能力 采用零拷贝的原理、顺序写盘、随机读 丰富的API使用 代码优秀，底层通信框架采用Netty NIO框架 NameServer代替Zookeeper 强调集群无单点，可扩展，任意一点高可用，水平可扩展 消息失败重试机制，消息可查询 开源社区活跃，成熟度高（经过双十一考验） 核心概念模型 基本概念 消息生产者（Producer）： 负责生产消息，一般由业务系统负责生产消息。 消息消费者（Consumer）： 负责消费消息，一般是后台系统负责异步消费。 推动式消费（Push Consumer）： Consumer消费的一种类型，该模式下Broker收到数据后会主动推送给消费端。 拉取式消费（Pull Consumer）： Consumer消费的一种类型，应用通常主动调用Consumer的拉消息方法从Broker服务器拉消息。 生产者组（Producer Group）： 同一类Producer的集合，这类Producer发送同一类消息且发送逻辑一致。 消费者组（Consumer Group）： 同一类Consumer的集合，这类Consumer通常消费同一类消息且消费逻辑一致。 Broker：MQ消息服务（中转角色，用于消息存储于生产消息转发） 源码结构 源码分析以4.4.0为例\n源码地址： https://github.com/apache/rocketmq\n结构：\n结构解析：\nrocketmq-common：公用数据结构 rocketmq-distribution：编译模块，编译输出 rocketmq-filter：进行Broker过滤的不感兴趣的消息传输，减小带宽压力 rocketmq-logappender、rocketmq-logging：日志服务 rocketmq-namesrv：Namesrv服务，用于服务协调 rocketmq-openmessaging：对外提供服务 rocketmq-remoting：远程调用接口，封装Netty底层通信 rocketmq-srvutil：提供一些公用的工具方法，比如命令行解析参数 rocketmq-store：消息存储 rocketmq-test、rocketmq-example rocketmq-tools：管理工具，比如有名的mqadmin工具 编译部署\u0026amp;环境搭建 选择1台服务器进行部署验证\nIP IP GY11-SyOCM33-101 10.139.12.52 编译部署 在源码目录执行一下命令编译工程\n$ mvn -Prelease-all -DskipTests clean package 编译成功后在rocketmq\\distribution\\target目录下生成apache-rocketmq.tar.gz包，使用该包解压与从官网下载一致，其他步骤参考二进制部署。\n二进制包部署（单点） 此处从官网下载4.4.0版本并上传至服务器解压\n下载地址\n解压到~/rocketmq-test/rocketmq-all-4.4.0-bin-release/如下\n$ ll total 36 drwxr-xr-x 2 ocmuser gpocm 83 Jan 17 2019 benchmark drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 bin drwxr-xr-x 5 ocmuser gpocm 196 Jan 17 2019 conf drwxr-xr-x 2 ocmuser gpocm 4096 Jan 17 2019 lib -rw-r--r-- 1 ocmuser gpocm 17336 Jan 17 2019 LICENSE -rw-r--r-- 1 ocmuser gpocm 1337 Jan 17 2019 NOTICE -rw-r--r-- 1 ocmuser gpocm 2481 Jan 17 2019 README.md 创建存储目录 $ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store $ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog $ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue $ mkdir -p ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index RocketMQ 配置文件 进入提供好的样例配置文件目录，其中包含双主双从异步复制、双主双从同步复制和双主集群配置文件，此处选择双主双从异步复制作为配置模板\n[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf]$ll total 36 drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-async drwxr-xr-x 2 ocmuser gpocm 118 Jan 17 2019 2m-2s-sync drwxr-xr-x 2 ocmuser gpocm 91 Jan 17 2019 2m-noslave -rw-r--r-- 1 ocmuser gpocm 949 Jan 17 2019 broker.conf -rw-r--r-- 1 ocmuser gpocm 14978 Jan 17 2019 logback_broker.xml -rw-r--r-- 1 ocmuser gpocm 3836 Jan 17 2019 logback_namesrv.xml -rw-r--r-- 1 ocmuser gpocm 3761 Jan 17 2019 logback_tools.xml -rw-r--r-- 1 ocmuser gpocm 1277 Jan 17 2019 plain_acl.yml -rw-r--r-- 1 ocmuser gpocm 833 Jan 17 2019 tools.yml 选择broker-a配置文件\n[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async]$ll total 16 -rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-a.properties -rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-a-s.properties -rw-r--r-- 1 ocmuser gpocm 929 Jan 17 2019 broker-b.properties -rw-r--r-- 1 ocmuser gpocm 922 Jan 17 2019 broker-b-s.properties 删除原有配置，加入如下配置（单点）\n#所属集群名字 brokerClusterName=rocketmq-cluster #broker 名字，注意此处不同的配置文件填写的不一样 brokerName=broker-a #0 表示 Master，\u0026gt;0 表示 Slave brokerId=0 #nameServer 地址，分号分割 namesrvAddr=10.139.12.52:9876 #在发送消息时，自动创建服务器不存在的 topic，默认创建的队列数 defaultTopicQueueNums=4 #是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭 autoCreateTopicEnable=true #是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭 autoCreateSubscriptionGroup=true #Broker 对外服务的监听端口 listenPort=10911 #删除文件时间点，默认凌晨 4 点 deleteWhen=04 #文件保留时间，默认 48 小时 fileReservedTime=120 #commitLog 每个文件的大小默认 1G mapedFileSizeCommitLog=1073741824 #ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整 mapedFileSizeConsumeQueue=300000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #检测物理文件磁盘空间 diskMaxUsedSpaceRatio=88 #存储路径 storePathRootDir=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store #commitLog 存储路径 storePathCommitLog=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog #消费队列存储路径存储路径 storePathConsumeQueue=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue #消息索引存储路径 storePathIndex=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index #checkpoint 文件存储路径 storeCheckpoint=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/checkpoint #abort 文件存储路径 abortFile=/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/abort #限制的消息大小 maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Broker 的角色 #- ASYNC_MASTER 异步复制 Master #- SYNC_MASTER 同步双写 Master #- SLAVE brokerRole=ASYNC_MASTER #刷盘方式 #- ASYNC_FLUSH 异步刷盘 #- SYNC_FLUSH 同步刷盘 flushDiskType=ASYNC_FLUSH #checkTransactionMessageEnable=false #发消息线程池数量 #sendMessageThreadPoolNums=128 #拉消息线程池数量 #pullMessageThreadPoolNums=128 修改日志配置文件 执行命令替换默认日志目录为指定日志目录\n$ mkdir -p /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs $ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf \u0026amp;\u0026amp; sed -i \u0026#39;s#${user.home}#/home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release#g\u0026#39; *.xml 修改启动脚本参数 将默认8g的内存配置改成1g\n$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runbroker.sh runbroker.sh\n#=========================================================================================== # JVM Configuration #=========================================================================================== JAVA_OPT=\u0026#34;${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m\u0026#34; ... 同理\n$ vim /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin/runserver.sh runserver.sh #=========================================================================================== # JVM Configuration #=========================================================================================== JAVA_OPT=\u0026#34;${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:PermSize=128m -XX:MaxPermSize=320m\u0026#34; ... 启动 NameServer $ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin $ nohup sh mqnamesrv \u0026amp; 检查进程\n[ocmuser@GY11-SyOCM33-101 ~/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin]$jps 65322 NamesrvStartup 65534 Jps 启动 BrokerServer A $ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin $ nohup sh mqbroker -c /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/conf/2m-2s-async/broker-a.properties \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 \u0026amp; $ netstat -ntlp $ jps $ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/broker.log $ tail -f -n 500 /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/logs/rocketmqlogs/namesrv.log 停止与数据清理 $ cd /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/bin $ sh mqshutdown broker $ sh mqshutdown namesrv # --等待停止 $ rm -rf /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store $ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store $ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/commitlog $ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/consumequeue $ mkdir /home/ocmuser/rocketmq-test/rocketmq-all-4.4.0-bin-release/store/index # --按照上面步骤重启 NameServer 与 BrokerServer SpringBoot 控制台讲解 克隆工程：https://github.com/apache/rocketmq-externals.git\n进入控制台配置文件rocketmq-externals\\rocketmq-console\\src\\main\\resources\\application.properties修改配置\nserver.contextPath= server.port=9090 ## SSL setting #server.ssl.key-store=classpath:rmqcngkeystore.jks #server.ssl.key-store-password=rocketmq #server.ssl.keyStoreType=PKCS12 #server.ssl.keyAlias=rmqcngkey #spring.application.index=true spring.application.name=rocketmq-console spring.http.encoding.charset=UTF-8 spring.http.encoding.enabled=true spring.http.encoding.force=true logging.config=classpath:logback.xml #if this value is empty,use env value rocketmq.config.namesrvAddr NAMESRV_ADDR | now, you can set it in ops page.default localhost:9876 rocketmq.config.namesrvAddr=10.139.12.52:9876 #if you use rocketmq version \u0026lt; 3.5.8, rocketmq.config.isVIPChannel should be false.default true rocketmq.config.isVIPChannel= #rocketmq-console\u0026#39;s data path:dashboard/monitor rocketmq.config.dataPath=./data #set it false if you don\u0026#39;t want use dashboard.default true rocketmq.config.enableDashBoardCollect=true #set the message track trace topic if you don\u0026#39;t want use the default one ","date":"2020-12-23T19:04:00+08:00","permalink":"/archives/251/","title":"RocketMQ学习笔记1——初探"},{"content":"Spring Framework 手动装配 Spring模式注解装配 定义:一种用于声明在应用中扮演“组件”角色的注解 举例:@Component、@Service、@Configuration等 装配:\u0026lt; context: component-scan\u0026gt;或@Componentscan 官方仓库说明\nStereotype Annotations A *stereotype annotation* is an annotation that is used to declare the role that a component plays within the application. For example, the @Repository annotation in the Spring Framework is a marker for any class that fulfills the role or stereotype of a repository (also known as Data Access Object or DAO).\n@Component is a generic stereotype for any Spring-managed component. Any component annotated with @Component is a candidate for component scanning. Similarly, any component annotated with an annotation that is itself meta-annotated with @Component is also a candidate for component scanning. For example, @Service is meta-annotated with @Component.\nCore Spring provides several stereotype annotations out of the box, including but not limited to: @Component, @Service, @Repository, @Controller, @RestController, and @Configuration. @Repository, @Service, etc. are specializations of @Component.\n模式注解是一种用于声明在应用中扮演“组件”角色的注解。如 Spring Framework 中的 @Repository 标注在任何类上 ，用于扮演仓储角色的模式注解。\n@Component 作为一种由 Spring 容器托管的通用模式组件，任何被 @Component 标准的组件均为组件扫描的候选对象。类似地，凡是被 @Component 元标注（meta-annotated）的注解，如 @Service ，当任何组件标注它时，也被视作组件扫描的候选对象（派生性）\n模式注解举例 Spring Framework 注解 场景说明 起始版本 @Repository 数据仓储模式注解 2.0 @Component 通用组件模式注解 2.5 @Service 服务模式注解 2.5 @Controller Web控制器模式注解 2.5 @Configuration 配置类模式注解 3.0 装配方式 \u0026lt;context:component-scan\u0026gt; 方式\n\u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;UTF-8\u0026#34;?\u0026gt; \u0026lt;beans xmlns=\u0026#34;http://www.springframework.org/schema/beans\u0026#34; xmlns:xsi=\u0026#34;http://www.w3.org/2001/XMLSchema-instance\u0026#34; xmlns:context=\u0026#34;http://www.springframework.org/schema/context\u0026#34; xsi:schemaLocation=\u0026#34;http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring- context.xsd\u0026#34;\u0026gt; \u0026lt;!-- 激活注解驱动特性 --\u0026gt; \u0026lt;context:annotation-config /\u0026gt; \u0026lt;!-- 找寻被 @Component 或者其派生 Annotation 标记的类（Class），将它们注册为 Spring Bean --\u0026gt; \u0026lt;context:component-scan base-package=\u0026#34;top.fjy8018.spring.boot\u0026#34; /\u0026gt; \u0026lt;/beans\u0026gt; @ComponentScan 方式\n@ComponentScan(basePackages = \u0026#34;top.fjy8018.spring.boot\u0026#34;) public class SpringConfiguration { ... } Spring Framework手动装配 自定义模式注解 @Component “派生性” package top.fjy8018.springboot.annotation; import org.springframework.stereotype.Component; import org.springframework.stereotype.Repository; import java.lang.annotation.*; /** * @author F嘉阳 * @date 2018-09-09 10:36 */ @Target({ElementType.TYPE}) @Retention(RetentionPolicy.RUNTIME) @Documented @Repository public @interface FirstLevelRepository { /** * 保证签名一致性 */ String value() default \u0026#34;\u0026#34;; } 三个级别并无明显区别，体现注解派生性，即使用@FirstLevelRepository和使用@Component效果一致\n派生关系\n@Component @Repository @FirstLevelRepository package org.springframework.stereotype; /** * @since 2.5 */ @Target(ElementType.TYPE) @Retention(RetentionPolicy.RUNTIME) @Documented @Indexed public @interface Component { /** * Component的签名 */ String value() default \u0026#34;\u0026#34;; } Repository中元标注了@Component\npackage org.springframework.stereotype; /** * @since 2.0 * @see Component */ @Target({ElementType.TYPE}) @Retention(RetentionPolicy.RUNTIME) @Documented @Component public @interface Repository { /** * 签名与Component一致 */ @AliasFor(annotation = Component.class) String value() default \u0026#34;\u0026#34;; } @Component “层次性” package top.fjy8018.springboot.annotation; import org.springframework.stereotype.Component; import org.springframework.stereotype.Repository; import java.lang.annotation.*; /** * Component层次性 * 顶级 {@link Component} * 一级 {@link Repository @Repository} * 二级 {@link FirstLevelRepository} * 三级 {@link SecondLevelRepository} * * @author F嘉阳 * @date 2018-09-09 10:54 */ @Target({ElementType.TYPE}) @Retention(RetentionPolicy.RUNTIME) @Documented @FirstLevelRepository public @interface SecondLevelRepository { /** * 层次性、派生性都必须保证签名一致 */ String value() default \u0026#34;\u0026#34;; } @Component @Repository FirstLevelRepository SecondLevelRepository 注解目标类，value 为bean的名称，此处不管用@FirstLevelRepository还是用SecondLevelRepository，效果都和使用@Component一致\npackage top.fjy8018.springboot.repository; import top.fjy8018.springboot.annotation.FirstLevelRepository; /** * @author F嘉阳 * @date 2018-09-09 10:40 */ //@Component(\u0026#34;myRepository\u0026#34;) @FirstLevelRepository(value = \u0026#34;myRepository\u0026#34;) //@SecondLevelRepository(value = \u0026#34;myRepository\u0026#34;) public class RepositoryDemo { } 仓储引导类，使用@ComponentScan指定扫描Bean路径\npackage top.fjy8018.springboot.bootstrap; import org.springframework.boot.WebApplicationType; import org.springframework.boot.builder.SpringApplicationBuilder; import org.springframework.context.ConfigurableApplicationContext; import org.springframework.context.annotation.ComponentScan; import top.fjy8018.springboot.repository.RepositoryDemo; /** * @author F嘉阳 * @date 2018-09-09 10:43 */ @ComponentScan(basePackages = \u0026#34;top.fjy8018.springboot.repository\u0026#34;) public class RepositoryBootstrapDemo { public static void main(String[] args) { ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(RepositoryBootstrapDemo.class) .web(WebApplicationType.NONE) .run(args); // 通过bean名称获取bean RepositoryDemo myRepository = applicationContext.getBean(\u0026#34;myRepository\u0026#34;, RepositoryDemo.class); // myRepository是否存在，若不存在则报错 System.out.println(\u0026#34;myRepository是否存在：\u0026#34; + myRepository); // 关闭上下文 applicationContext.close(); } } 执行结果\nSpringBoot源码中也有体现该层次关系\n@SpringBootApplication @SpringBootConfiguration @Configuration @Component @EnableAutoConfiguration @AutoConfigurationPackage Spring @Enable 模块装配 Spring Framework 3.1 开始支持“@Enable 模块驱动”。所谓“模块”是指具备相同领域的功能组件集合， 组合所形成一个独立的单元。比如 Web MVC 模块、AspectJ代理模块、Caching（缓存）模块、JMX（Java 管理扩展）模块、Async（异步处理）模块等。\n举例:@EnableWebmvc、@EnableAutoConfiguration等 实现:注解方式、编程方式 @Enable 注解模块举例\n框架实现 @Enable 注解模块 激活模块 Spring Framework @EnableWebMvc Web @EnableTransactionManagement 事务管理模块 @EnableCaching Caching 模块 @EnableMBeanExport JMX 模块 @EnableAsync 异步处理模块 EnableWebFlux Web Flux 模块 @EnableAspectJAutoProxy AspectJ 代理模块 Spring Boot @EnableAutoConfiguration 自动装配模块 @EnableManagementContext Actuator 管理模块 @EnableConfigurationProperties 配置属性绑定模块 @EnableOAuth2Sso OAuth2 单点登录模块 Spring Cloud @EnableEurekaServer Eureka服务器模块 @EnableConfigServer 配置服务器模块 @EnableFeignClients Feign客户端模块 @EnableZuulProxy 服务网关 @EnableCircuitBreaker 服务熔断模块 实现方式 注解驱动方式 Spring源码，使用@Import注入配置Bean\npackage org.springframework.web.servlet.config.annotation; @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) @Documented @Import(DelegatingWebMvcConfiguration.class) public @interface EnableWebMvc {} 接口编程方式 Spring源码——激活缓存注解\npackage org.springframework.cache.annotation; @Target(ElementType.TYPE) @Retention(RetentionPolicy.RUNTIME) @Documented @Import(CachingConfigurationSelector.class) public @interface EnableCaching {...} CachingConfigurationSelector类图\nImportSelector接口源码，该接口在3.1引入，目的在于增加灵活度\nselectImports方法返回数组，即可以死是多个配置Bean\npackage org.springframework.context.annotation; /** * @since 3.1 */ public interface ImportSelector { String[] selectImports(AnnotationMetadata importingClassMetadata); } @Enable注解方式自定义实现 激活HelloWorld模块，使用@Import(HelloWorldConfiguration.class)直接注入HelloWorldConfiguration模块\npackage top.fjy8018.springboot.annotation; import org.springframework.context.annotation.Import; import top.fjy8018.springboot.configuration.HelloWorldConfiguration; import java.lang.annotation.*; /** * @author F嘉阳 * @date 2018-09-09 11:35 */ @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) @Documented @Import(HelloWorldConfiguration.class) public @interface EnableHelloWorld {} 目标类，其注解@Configuration在Spring3.0时引入\npackage top.fjy8018.springboot.configuration; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; /** * @author F嘉阳 * @date 2018-09-09 11:31 */ @Configuration public class HelloWorldConfiguration { /** * 方法名称即Bean名称 * @return Hello World! */ @Bean public String helloWorld() { return \u0026#34;Hello world!\u0026#34;; } } package org.springframework.context.annotation; /** * @since 3.0 */ @Target(ElementType.TYPE) @Retention(RetentionPolicy.RUNTIME) @Documented @Component public @interface Configuration { @AliasFor(annotation = Component.class) String value() default \u0026#34;\u0026#34;; } @Enable接口编程方式自定义实现 参考Spring包名路径创建自定义Selector，通过HelloWorldImportSelector实现HelloWorldConfiguration的配置Bean导入\npackage top.fjy8018.springboot.annotation; import org.springframework.context.annotation.ImportSelector; import org.springframework.core.type.AnnotationMetadata; import top.fjy8018.springboot.configuration.HelloWorldConfiguration; /** * @author F嘉阳 * @date 2018-09-09 11:28 */ public class HelloWorldImportSelector implements ImportSelector { /** * @param importingClassMetadata 元数据信息 * @return 一个或多个配置Bean */ @Override public String[] selectImports(AnnotationMetadata importingClassMetadata) { return new String[]{HelloWorldConfiguration.class.getName()}; } } 激活模块方式变化，使用@Import(HelloWorldImportSelector.class)注入HelloWorldImportSelector模块，在Selector里加载并返回 HelloWorldConfiguration 模块\npackage top.fjy8018.springboot.annotation; import org.springframework.context.annotation.Import; import top.fjy8018.springboot.configuration.HelloWorldConfiguration; import java.lang.annotation.*; /** * @author F嘉阳 * @date 2018-09-09 11:35 */ @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.TYPE) @Documented //@Import(HelloWorldConfiguration.class) @Import(HelloWorldImportSelector.class) public @interface EnableHelloWorld { } 实现逻辑 graph LR A(EnableHelloWorld) --\u003e B(HelloWorldImportSelector) B--\u003eC(HelloWorldConfiguration) C --\u003e D(helloWorld) 此方法比直接注解驱动更有弹性，可增加switch条件返回不同的模块，源码样例CachingConfigurationSelector\npackage org.springframework.cache.annotation; public class CachingConfigurationSelector extends AdviceModeImportSelector\u0026lt;EnableCaching\u0026gt; { @Override public String[] selectImports(AdviceMode adviceMode) { switch (adviceMode) { case PROXY: return getProxyImports(); case ASPECTJ: return getAspectJImports(); default: return null; } } ... } Spring条件装配 从 Spring Framework 3.1 开始，允许在 Bean 装配时增加前置条件判断\nSpring 注解 场景说明 起始版本 @Profile 配置化条件装配 3.1 @Conditional 编程条件装配 4.0 条件注解实现方式 配置方式 - @Profile 在spring4后实现方式发生变化，依赖@Conditional和ProfileCondition实现\npackage org.springframework.context.annotation; /** * @since 3.1 */ @Target({ElementType.TYPE, ElementType.METHOD}) @Retention(RetentionPolicy.RUNTIME) @Documented @Conditional(ProfileCondition.class) public @interface Profile {...} package org.springframework.context.annotation; /** * @since 4.0 */ class ProfileCondition implements Condition {...} 通过上下文对象和注解元信息判断\npackage org.springframework.context.annotation; /** * @since 4.0 */ @FunctionalInterface public interface Condition { boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata); } 编程方式 - @Conditional ConditionalOnClass用于判断标识类在当前classpath下是否存在，该注解使用@Conditional搭配OnClassCondition实现\npackage org.springframework.boot.autoconfigure.condition; /** * {@link Conditional} that only matches when the specified classes are on the classpath. */ @Target({ ElementType.TYPE, ElementType.METHOD }) @Retention(RetentionPolicy.RUNTIME) @Documented @Conditional(OnClassCondition.class) public @interface ConditionalOnClass {...} package org.springframework.boot.autoconfigure.condition; /** * {@link Condition} that checks if properties are defined in environment. * @since 1.1.0 * @see ConditionalOnProperty */ @Order(Ordered.HIGHEST_PRECEDENCE + 40) class OnPropertyCondition extends SpringBootCondition { @Override public ConditionOutcome getMatchOutcome(ConditionContext context, AnnotatedTypeMetadata metadata) { // 获取元信息 List\u0026lt;AnnotationAttributes\u0026gt; allAnnotationAttributes = annotationAttributesFromMultiValueMap( metadata.getAllAnnotationAttributes( ConditionalOnProperty.class.getName())); List\u0026lt;ConditionMessage\u0026gt; noMatch = new ArrayList\u0026lt;\u0026gt;(); List\u0026lt;ConditionMessage\u0026gt; match = new ArrayList\u0026lt;\u0026gt;(); for (AnnotationAttributes annotationAttributes : allAnnotationAttributes) { ConditionOutcome outcome = determineOutcome(annotationAttributes, context.getEnvironment()); (outcome.isMatch() ? match : noMatch).add(outcome.getConditionMessage()); } if (!noMatch.isEmpty()) { return ConditionOutcome.noMatch(ConditionMessage.of(noMatch)); } return ConditionOutcome.match(ConditionMessage.of(match)); } ... } OnPropertyCondition为抽象类，实际仍然是Condition接口的实现\npackage org.springframework.boot.autoconfigure.condition; public abstract class SpringBootCondition implements Condition {...} ConditionalOnProperty用于判断配置在上下文中是否存在\npackage org.springframework.boot.autoconfigure.condition; /** * @since 1.1.0 */ @Retention(RetentionPolicy.RUNTIME) @Target({ ElementType.TYPE, ElementType.METHOD }) @Documented @Conditional(OnPropertyCondition.class) public @interface ConditionalOnProperty {...} 自定义条件装配 目标服务\npackage top.fjy8018.springboot.service; /** * 计算模块 * @author F嘉阳 * @date 2018-09-09 11:56 */ public interface CalculateService { /** * 多整数求和 * @param values 多个整数 * @return 累加值 */ Integer sum(Integer... values); } 基于配置方式 服务实现类，分别使用 for循环实现和Lambda实现，均标注@Service作为Bean自动注入\n指定@Profile(\u0026quot;Java7\u0026quot;)，当条件为Java7时才装配\npackage top.fjy8018.springboot.service.impl; import org.springframework.context.annotation.Profile; import org.springframework.stereotype.Service; import top.fjy8018.springboot.service.CalculateService; /** * @author F嘉阳 * @date 2018-09-09 11:57 */ @Profile(\u0026#34;Java7\u0026#34;) @Service public class Java7CalculateServiceImpl implements CalculateService { /** * 多整数求和 * * @param values 多个整数 * @return 累加值 */ @Override public Integer sum(Integer... values) { System.out.println(\u0026#34;计算模块java7 for循环实现\u0026#34;); int sum = 0; for (int i = 0; i \u0026lt; values.length; i++) { sum += values[i]; } return sum; } } 指定@Profile(\u0026quot;Java8\u0026quot;)，当条件为Java7时才装配\npackage top.fjy8018.springboot.service.impl; import org.springframework.context.annotation.Profile; import org.springframework.stereotype.Service; import top.fjy8018.springboot.service.CalculateService; import java.util.stream.Stream; /** * @author F嘉阳 * @date 2018-09-09 11:57 */ @Profile(\u0026#34;Java8\u0026#34;) @Service public class Java8CalculateServiceImpl implements CalculateService { /** * 多整数求和 * @param values 多个整数 * @return 累加值 */ @Override public Integer sum(Integer... values) { System.out.println(\u0026#34;计算模块java8 Lambda实现\u0026#34;); return Stream.of(values).reduce(0, Integer::sum); } } 启动类，通过scanBasePackages指定报名扫描，该方法与@ComponentScan(basePackages = \u0026quot;xxx\u0026quot;)效果一致\npackage top.fjy8018.springboot.bootstrap; import org.springframework.boot.WebApplicationType; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.boot.builder.SpringApplicationBuilder; import org.springframework.context.ConfigurableApplicationContext; import top.fjy8018.springboot.service.CalculateService; /** * @author F嘉阳 * @date 2018-09-09 12:06 */ @SpringBootApplication(scanBasePackages = \u0026#34;top.fjy8018.springboot.service\u0026#34;) public class CalculateServiceDemo { public static void main(String[] args) { ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(CalculateServiceDemo.class) .profiles(\u0026#34;Java8\u0026#34;) .web(WebApplicationType.NONE) .run(args); // 直接用类获取bean CalculateService calculateService = applicationContext.getBean(CalculateService.class); // calculateService Bean 是否存在，若不添加Profile则必有报错 Integer[] values = new Integer[10]; for (int i = 1; i \u0026lt;= values.length; i++) { values[i - 1] = i; } System.out.println(\u0026#34;calculateService Bean 求和: \u0026#34; + calculateService.sum(values)); // 关闭上下文 applicationContext.close(); } } 执行结果——java8\n执行结果——java7\n若不指定配置则报错\n基于编程方式 系统属性条件判断实现类\npackage top.fjy8018.springboot.condition; import org.springframework.context.annotation.Condition; import org.springframework.context.annotation.ConditionContext; import org.springframework.core.type.AnnotatedTypeMetadata; import java.util.Map; /** * @author F嘉阳 * @date 2018-09-09 17:15 */ public class OnSystemPropertyCondition implements Condition { /** * 条件匹配规则，此处可实现复杂逻辑判断 * @param context 上下文 * @param metadata 元信息 * @return 是否匹配 */ @Override public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) { // 通过类名称取元信息 Map\u0026lt;String, Object\u0026gt; annotationAttributes = metadata.getAnnotationAttributes(ConditionalOnSystemProperty.class.getName()); String propertyName = String.valueOf(annotationAttributes.get(\u0026#34;name\u0026#34;)); String propertyValue = String.valueOf(annotationAttributes.get(\u0026#34;value\u0026#34;)); // 取系统属性 String systemPropertyValue = System.getProperty(propertyName); return propertyValue.equals(systemPropertyValue); } Java 系统属性条件判断类装配，系统属性通过Java API获取java.lang.System#getProperty(java.lang.String)方法获取的属性值\npackage top.fjy8018.springboot.condition; import org.springframework.context.annotation.Conditional; import java.lang.annotation.*; /** * @author F嘉阳 * @date 2018-09-09 17:14 */ @Target({ElementType.TYPE, ElementType.METHOD}) @Retention(RetentionPolicy.RUNTIME) @Documented @Conditional(OnSystemPropertyCondition.class) public @interface ConditionalOnSystemProperty { /** * Java 系统属性名称 * @return 系统属性名称 */ String name(); /** * 系统属性值 * @return 属性值 */ String value(); } 启动类，helloWorld仅在系统条件满足时自动装配，注解若作用于类上语义相同\npackage top.fjy8018.springboot.bootstrap; import org.springframework.boot.WebApplicationType; import org.springframework.boot.builder.SpringApplicationBuilder; import org.springframework.context.ConfigurableApplicationContext; import org.springframework.context.annotation.Bean; import top.fjy8018.springboot.condition.ConditionalOnSystemProperty; /** * 系统属性条件引导类 * * @author F嘉阳 * @date 2018-09-09 17:26 */ public class SystemPropertyConditionBootstrap { @Bean @ConditionalOnSystemProperty(name = \u0026#34;user.name\u0026#34;, value = \u0026#34;11860\u0026#34;) public String helloWorld() { return \u0026#34;Hello SpringBoot2.0\u0026#34;; } public static void main(String[] args) { ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(SystemPropertyConditionBootstrap.class) .web(WebApplicationType.NONE) .run(args); // 通过类型和方法名获取Bean String helloWorldBean = applicationContext.getBean(\u0026#34;helloWorld\u0026#34;, String.class); // 若系统参数条件不匹配则报找不到Bean错误，以此实现编程方式条件加载 System.out.println(\u0026#34;ConditionalOnSystemProperty 加载内容：\u0026#34; + helloWorldBean); applicationContext.close(); } } 执行结果\n条件不满足则报错\nSpring Boot 自动装配 在 Spring Boot 场景下，基于规约大于配置的原则，实现 Spring 组件自动装配的目的（0配置）。\n装配: 模式注解、@Eηable模块、条件装配、工厂加载机制 实现: 激活自动装配、实现自动装配、配置自动装配实现 底层装配技术 Spring 模式注解装配 Spring @Enable 模块装配 Spring 条件装配装配 Spring 工厂加载机制 实现类： SpringFactoriesLoader 配置资源： META-INF/spring.factories package org.springframework.core.io.support; /** * @since 3.2 */ public abstract class SpringFactoriesLoader { public static final String FACTORIES_RESOURCE_LOCATION = \u0026#34;META-INF/spring.factories\u0026#34;; public static \u0026lt;T\u0026gt; List\u0026lt;T\u0026gt; loadFactories(Class\u0026lt;T\u0026gt; factoryClass, @Nullable ClassLoader classLoader) { Assert.notNull(factoryClass, \u0026#34;\u0026#39;factoryClass\u0026#39; must not be null\u0026#34;); // 获取类加载器 ClassLoader classLoaderToUse = classLoader; if (classLoaderToUse == null) { classLoaderToUse = SpringFactoriesLoader.class.getClassLoader(); } List\u0026lt;String\u0026gt; factoryNames = loadFactoryNames(factoryClass, classLoaderToUse); if (logger.isTraceEnabled()) { logger.trace(\u0026#34;Loaded [\u0026#34; + factoryClass.getName() + \u0026#34;] names: \u0026#34; + factoryNames); } List\u0026lt;T\u0026gt; result = new ArrayList\u0026lt;\u0026gt;(factoryNames.size()); // 迭代装配 for (String factoryName : factoryNames) { result.add(instantiateFactory(factoryName, factoryClass, classLoaderToUse)); } AnnotationAwareOrderComparator.sort(result); return result; } ... } 自动装配举例 参考 META-INF/spring.factories\n实现方法 激活自动装配 - @EnableAutoConfiguration 实现自动装配 - XXXAutoConfiguration 配置自动装配实现 - META-INF/spring.factories 目的在于使用META-INF/spring.factories文件配置替代@ComponentScan手动装配注解，从而实现自动装配\n而Spring大多数自动装配都由starter模块负责\n自定义自动装配 设计逻辑\nHelloWorldAutoConfiguration 条件判断： user.name == \u0026quot;Mercy\u0026quot; 模式注解： @Configuration @Enable 模块： 流程图 引导类\npackage top.fjy8018.springboot.bootstrap; import org.springframework.boot.WebApplicationType; import org.springframework.boot.autoconfigure.EnableAutoConfiguration; import org.springframework.boot.builder.SpringApplicationBuilder; import org.springframework.context.ConfigurableApplicationContext; /** * @author F嘉阳 * @date 2018-09-10 17:13 */ @EnableAutoConfiguration public class HelloWorldAutoConfigurationBootstrap { public static void main(String[] args) { ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(HelloWorldAutoConfigurationBootstrap.class) .web(WebApplicationType.NONE) .run(args); // 通过bean名称获取helloWorld Bean String helloWorldBean = applicationContext.getBean(\u0026#34;helloWorld\u0026#34;, String.class); // helloWorld Bean 是否存在 System.out.println(\u0026#34;helloWorld Bean 内容: \u0026#34; + helloWorldBean); // 关闭上下文 applicationContext.close(); } } 配置Bean，使用模式注解标注 @Configuration，该类实际上实现了条件装配、模式注解、自动装配\npackage top.fjy8018.springboot.configuration; import org.springframework.context.annotation.Configuration; import top.fjy8018.springboot.annotation.EnableHelloWorld; import top.fjy8018.springboot.condition.ConditionalOnSystemProperty; /** * @author F嘉阳 * @date 2018-09-10 17:23 */ @Configuration @EnableHelloWorld @ConditionalOnSystemProperty(name = \u0026#34;user.name\u0026#34;, value = \u0026#34;11860\u0026#34;) public class HelloWorldAutoConfiguration {} 创建spring.factories，每行全类名顶格开始，防止在某些版本下产生不稳定的BUG\n# 自动装配 org.springframework.boot.autoconfigure.EnableAutoConfiguration=\\ top.fjy8018.springboot.configuration.HelloWorldAutoConfiguration 执行结果与之前一致\n","date":"2020-12-10T17:28:00+08:00","permalink":"/archives/249/","title":"SpringBoot源码分析2——走向自动装配"},{"content":"SpringBoot 2.0 特点 组件自动装配 规约大于配置,专注核心业务\n模式注解、@ Enable模块、条件装配、加载机制\n外部化配置 一次构建、按需调配,到处运行\nEnvironment抽象、生命周期、破坏性变更\n嵌入式容器 内置容器、无需部署、独立运行\nServlet Web容器、 Reactive Web容器\nSpring Boot Starter 简化依赖、按需装配、自我包含\n依赖管理、装配条件、装配顺序\nProduction-Ready 一站式运维、生态无缝整合\n健康检査、数据指标、@ Endpoint管控\nSpringBoot 与Java EE规范 Web: Servlet( JSR-315, JSR-340） SQL: JDBC(SR-221) 数据校验: Bean validation(JSR303、JSR-349) 缓存: Java Caching API(JSR-107) Web Sockets: Java api for Web Socket(JSR-356 Web services: JAX-WS( JSR-224） Java管理:JMX(JSR3) 消息:JMS(JSR-914) 核心特性介绍 三大特性 组件自动装配: Web mvc、 Web flux、JDBC等 嵌入式Web容器: Tomcat、 Jetty以及 Undertow 生产准备特性:指标、健康检査、外部化配置等 组件自动装配 激活： @EnableAutoConfiguration 配置： /META-INF/spring.factories 实质为key-value形式 实现： XXXAutoConfiguration\n官方文档样例\nimport org.springframework.boot.*; import org.springframework.boot.autoconfigure.*; import org.springframework.web.bind.annotation.*; @RestController @EnableAutoConfiguration public class Example { @RequestMapping(\u0026#34;/\u0026#34;) String home() { return \u0026#34;Hello World!\u0026#34;; } public static void main(String[] args) throws Exception { SpringApplication.run(Example.class, args); } } 实际上@SpringBootApplication已经包含该注解\n@Target(ElementType.TYPE) @Retention(RetentionPolicy.RUNTIME) @Documented @Inherited @SpringBootConfiguration @EnableAutoConfiguration @ComponentScan(excludeFilters = { @Filter(type = FilterType.CUSTOM, classes = TypeExcludeFilter.class), @Filter(type = FilterType.CUSTOM, classes = AutoConfigurationExcludeFilter.class) }) public @interface SpringBootApplication { } 在IDEA中使用ctrl+shift+n可查看Spring源码中包含大量spring.factories文件\n样例文件：\norg/springframework/boot/spring-boot-autoconfigure/2.0.4.RELEASE/spring-boot-autoconfigure-2.0.4.RELEASE.jar!/META-INF/spring.factories\n# Initializers org.springframework.context.ApplicationContextInitializer=\\ org.springframework.boot.autoconfigure.SharedMetadataReaderFactoryContextInitializer,\\ org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener # Application Listeners org.springframework.context.ApplicationListener=\\ org.springframework.boot.autoconfigure.BackgroundPreinitializer # Auto Configuration Import Listeners org.springframework.boot.autoconfigure.AutoConfigurationImportListener=\\ org.springframework.boot.autoconfigure.condition.ConditionEvaluationReportAutoConfigurationImportListener # Auto Configuration Import Filters org.springframework.boot.autoconfigure.AutoConfigurationImportFilter=\\ org.springframework.boot.autoconfigure.condition.OnClassCondition # 自动装配配置，自动装载Bean列表 org.springframework.boot.autoconfigure.EnableAutoConfiguration=\\ org.springframework.boot.autoconfigure.admin.SpringApplicationAdminJmxAutoConfiguration,\\ org.springframework.boot.autoconfigure.web.servlet.WebMvcAutoConfiguration,\\ ... WebMvcAutoConfiguration源码，其包含大量条件装配，若未检测到该类的依赖，则不会被装配\n@Configuration @ConditionalOnWebApplication(type = Type.SERVLET) @ConditionalOnClass({ Servlet.class, DispatcherServlet.class, WebMvcConfigurer.class }) @ConditionalOnMissingBean(WebMvcConfigurationSupport.class) @AutoConfigureOrder(Ordered.HIGHEST_PRECEDENCE + 10) @AutoConfigureAfter({ DispatcherServletAutoConfiguration.class, ValidationAutoConfiguration.class }) public class WebMvcAutoConfiguration { ... 嵌入式Web容器 Web servlet: Tomcat、 Jetty和 Undertow Web reactive Netty Web Server 生产准备特性 指标:/ actuator/ metrics 健康检查:/ actuator/ health 外部化配置:/ actuator/ configprops Web应用介绍 传统 Servlet应用概览 Servlet组件: Servlet、 Filter、 Listener Servlet注册: Servlet注解、 Spring Bean、 RegistrationBean 异步非阻塞:异步 Servlet、非阻塞 Servlet 传统 Servelt 应用 加入springboot应用通用依赖前缀多为spring-boot-starter-xxx\n加入Web依赖\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-web\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; Maven依赖变化\n由此可看出默认的Web容器为Tomcat（内嵌Servlet容器）\nServlet 组件 Servlet\n实现 @WebServlet HttpServlet 注册 URL 映射 @WebServlet(urlPatterns = \u0026quot;/my/servlet\u0026quot;) 注册 @ServletComponentScan(basePackages = \u0026quot;top.fjy8018.springboot.web.servlet\u0026quot;) Filter\nListener\nServlet 注册 Servlet 注解 @ServletComponentScan + @WebServlet @WebFilter @WebListener Spring Bean @Bean +\nServlet Filter Listener RegistrationBean\nServletRegistrationBean FilterRegistrationBean ServletListenerRegistrationBean Servlet 实现 参考源码包位置，由于出现了Reactive Web容器，SpringBoot 1.x的包路径与2.x有所不同，2.x包路径变更为org.springframework.boot.autoconfigure.web.servlet\nHttpServlet package top.fjy8018.springboot.web.servlet; import javax.servlet.ServletException; import javax.servlet.annotation.WebServlet; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import java.io.IOException; /** * 结合servlet规范注册 * * @author F嘉阳 * @date 2018-09-08 19:17 */ @WebServlet(urlPatterns = \u0026#34;/myservlet\u0026#34;) public class MyServlet extends HttpServlet { @Override protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException { resp.getWriter().print(\u0026#34;hello world\u0026#34;); } } Servlet 3.0规范要求必须实现HttpServlet\n其内部提供多种方法实现（ctrl+F12）\nurlPatterns = \u0026quot;/myservlet\u0026quot;指明浏览器映射\n注册 在主类中注册（包扫描）\npackage top.fjy8018.springboot; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.boot.web.servlet.ServletComponentScan; @SpringBootApplication @ServletComponentScan(basePackages = \u0026#34;top.fjy8018.springboot.web.servlet\u0026#34;) public class OverviewApplication { public static void main(String[] args) { SpringApplication.run(OverviewApplication.class, args); } } 启动后从控制台可看出映射已经成功\n异步非阻塞 Servlet 异步 Servlet javax.servlet.ServletRequest#startAsync() javax.servlet.AsyncContext 非阻塞 Servlet javax.servlet.ServletInputStream#setReadListener javax.servlet.ReadListener javax.servlet.ServletOutputStream#setWriteListener javax.servlet.WriteListener 异步非阻塞Servlet实现 package top.fjy8018.springboot.web.servlet; import javax.servlet.AsyncContext; import javax.servlet.ServletException; import javax.servlet.annotation.WebServlet; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import java.io.IOException; /** * 结合servlet规范注册 * asyncSupported指定异步servlet，默认false * * @author F嘉阳 * @date 2018-09-08 19:17 */ @WebServlet(urlPatterns = \u0026#34;/myservlet\u0026#34;, asyncSupported = true) public class MyServlet extends HttpServlet { @Override protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException { AsyncContext asyncContext = req.startAsync(); // 执行任务 asyncContext.start(() -\u0026gt; { try { resp.getWriter().print(\u0026#34;hello world\u0026#34;); // 触发完成，必须步骤，否则客户端发生超时 asyncContext.complete(); } catch (IOException e) { e.printStackTrace(); } }); } } asyncSupported = true 启用异步支持，否则报500错误 Spring Web MVC 应用介绍 Web mvo视图:模板引擎、内容协商、异常处理等 Web mVc rest:资源服务、资源跨域、服务发现等 Web MVC 视图 ViewResolver View 源码\npackage org.springframework.web.servlet; import java.util.Locale; import org.springframework.lang.Nullable; public interface ViewResolver { @Nullable View resolveViewName(String viewName, Locale locale) throws Exception; } 通过指定名称和路径加载视图\n视图渲染View\npackage org.springframework.web.servlet; public interface View { ... void render(@Nullable Map\u0026lt;String, ?\u0026gt; model, HttpServletRequest request, HttpServletResponse response) throws Exception; } 提供接口供不同的模板引擎实现ViewResolver\n模板引擎 Thymeleaf Freemarker JSP 若同时有多种模板引擎，则需要内容协商\n内容协商 ContentNegotiationConfigurer ContentNegotiationStrategy ContentNegotiatingViewResolver 异常处理 @ExceptionHandler HandlerExceptionResolver ExceptionHandlerExceptionResolver BasicErrorController (Spring Boot默认白页404) 源码\n@Controller @RequestMapping(\u0026#34;${server.error.path:${error.path:/error}}\u0026#34;) public class BasicErrorController extends AbstractErrorController { @RequestMapping(produces = \u0026#34;text/html\u0026#34;) public ModelAndView errorHtml(HttpServletRequest request, HttpServletResponse response) { HttpStatus status = getStatus(request); Map\u0026lt;String, Object\u0026gt; model = Collections.unmodifiableMap(getErrorAttributes( request, isIncludeStackTrace(request, MediaType.TEXT_HTML))); response.setStatus(status.value()); ModelAndView modelAndView = resolveErrorView(request, response, status, model); return (modelAndView != null) ? modelAndView : new ModelAndView(\u0026#34;error\u0026#34;, model); } @RequestMapping @ResponseBody public ResponseEntity\u0026lt;Map\u0026lt;String, Object\u0026gt;\u0026gt; error(HttpServletRequest request) { Map\u0026lt;String, Object\u0026gt; body = getErrorAttributes(request, isIncludeStackTrace(request, MediaType.ALL)); HttpStatus status = getStatus(request); return new ResponseEntity\u0026lt;\u0026gt;(body, status); } } 其根据浏览器类型提供两种方式渲染，分别为html和json\n{ \u0026#34;timestamp\u0026#34;: \u0026#34;2019-03-13T04:47:18.445+0000\u0026#34;, \u0026#34;status\u0026#34;: 404, \u0026#34;error\u0026#34;: \u0026#34;Not Found\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;Not Found\u0026#34;, \u0026#34;path\u0026#34;: \u0026#34;/hello\u0026#34; } Web MVC REST 资源服务 @RequestMapping @GetMapping @ResponseBody @RequestBody 从源码分析可知@GetMapping是从@RequestMapping变化而来，通过委派（别名）方式实现，并且在SpringMVC 4.3后引入\npackage org.springframework.web.bind.annotation; import java.lang.annotation.Documented; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; import org.springframework.core.annotation.AliasFor; /** * Annotation for mapping HTTP {@code GET} requests onto specific handler * methods. * * \u0026lt;p\u0026gt;Specifically, {@code @GetMapping} is a \u0026lt;em\u0026gt;composed annotation\u0026lt;/em\u0026gt; that * acts as a shortcut for {@code @RequestMapping(method = RequestMethod.GET)}. * * * @author Sam Brannen * @since 4.3 * @see PostMapping * @see PutMapping * @see DeleteMapping * @see PatchMapping * @see RequestMapping */ @Target(ElementType.METHOD) @Retention(RetentionPolicy.RUNTIME) @Documented @RequestMapping(method = RequestMethod.GET) public @interface GetMapping { /** * 委派目标 {@link RequestMapping#name}. */ @AliasFor(annotation = RequestMapping.class) String name() default \u0026#34;\u0026#34;; ... } 资源跨域 CrossOrigin WebMvcConfigurer#addCorsMappings 传统解决方案 IFrame JSONP 源码\npackage org.springframework.web.servlet.config.annotation; /** * @since 3.1 */ public interface WebMvcConfigurer { /** * 跨域配置 * @since 4.2 */ default void addCorsMappings(CorsRegistry registry) { } } 注解实现方式\npackage org.springframework.web.bind.annotation; /** * @since 4.2 */ @Target({ ElementType.METHOD, ElementType.TYPE }) @Retention(RetentionPolicy.RUNTIME) @Documented public @interface CrossOrigin {...} 两者均来自springframework，属于不同的实现维度（接口编程 / 注解驱动）\n服务发现 HATEOS 核心组件 DispatcherServlet——Servlet实现请求转发到映射URL HandlerMapping——处理映射URL方法 HandlerAdapter——方法处理 ViewResolver——交由视图处理 \u0026hellip; Spring WebFlux （Spring 5+） Reactor 基础 Java Lambda —— Reactive Stream实现（JDK 9 flow api、RxJava） Mono Flux Web Flux 核心 Web MVC 注解兼容 @Controller @RequestMapping @ResponseBody @RequestBody \u0026hellip; 函数式声明 RouterFunction 异步非阻塞 Servlet 3.1 +\nNetty Reactor\n使用场景 页面渲染 REST 应用 性能测试 国外博客\n结论：Webflux 性能不会显著提升，但Webflux提升吞吐量\nWeb Server 应用 切换 Web server 自定义 Servlet Web server 自定义 Reactive Web server 切换 Web Server 切换其他 Servlet 容器 Tomcat -\u0026gt; Jetty\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-web\u0026lt;/artifactId\u0026gt; \u0026lt;exclusions\u0026gt; \u0026lt;!-- tomcat加载优先级高于jetty，若不加排除，则仍然加载tomcat --\u0026gt; \u0026lt;exclusion\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-tomcat\u0026lt;/artifactId\u0026gt; \u0026lt;/exclusion\u0026gt; \u0026lt;/exclusions\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- 使用 Jetty 容器 --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-jetty\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; 替换 Servlet 容器 WebFlux \u0026lt;!--使用webflux必须注释web依赖，当web和flux共存时，web优先级更高--\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-webflux\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; 注释后相关类和的注解也已经失效，需要删除\n@SpringBootApplication //@ServletComponentScan(basePackages = \u0026#34;top.fjy8018.springboot.web.servlet\u0026#34;) public class SpringbootApplication { public static void main(String[] args) { SpringApplication.run(SpringbootApplication.class, args); } } 依赖变化\n启动后变为Netty容器\n自定义 Servlet Web Server WebServerFactoryCustomizer SpringBoot 2.0 新增支持接口\npackage org.springframework.boot.web.server; /** * @since 2.0.0 * @see WebServerFactoryCustomizerBeanPostProcessor */ @FunctionalInterface public interface WebServerFactoryCustomizer\u0026lt;T extends WebServerFactory\u0026gt; { void customize(T factory); } 该接口为主接口，其下有大量实现，IDEA通过ctrl+alt+B或者ctrl+alt+左键点击查看\n其中TomcatServletWebServerFactoryCustomizer就是实现类之一，但在启用Webflux后该类无效\npackage org.springframework.boot.autoconfigure.web.servlet; /** * {@link WebServerFactoryCustomizer} to apply {@link ServerProperties} to Tomcat web * servers. * * @author Brian Clozel * @author Phillip Webb * @since 2.0.0 */ public class TomcatServletWebServerFactoryCustomizer implements WebServerFactoryCustomizer\u0026lt;TomcatServletWebServerFactory\u0026gt;, Ordered { @Override public void customize(TomcatServletWebServerFactory factory) { ServerProperties.Tomcat tomcatProperties = this.serverProperties.getTomcat(); if (!ObjectUtils.isEmpty(tomcatProperties.getAdditionalTldSkipPatterns())) { factory.getTldSkipPatterns() .addAll(tomcatProperties.getAdditionalTldSkipPatterns()); } if (tomcatProperties.getRedirectContextRoot() != null) { customizeRedirectContextRoot(factory, tomcatProperties.getRedirectContextRoot()); } if (tomcatProperties.getUseRelativeRedirects() != null) { customizeUseRelativeRedirects(factory, tomcatProperties.getUseRelativeRedirects()); } } ... } 自定义 Reactive Web Server ReactiveWebServerFactoryCustomizer 其源码显示提供了最基础的实现，包括端口、SSL等，允许自行扩展实现\npackage org.springframework.boot.autoconfigure.web.reactive; /** * {@link WebServerFactoryCustomizer} to apply {@link ServerProperties} to reactive * servers. * @since 2.0.0 */ public class ReactiveWebServerFactoryCustomizer implements WebServerFactoryCustomizer\u0026lt;ConfigurableReactiveWebServerFactory\u0026gt;, Ordered { private final ServerProperties serverProperties; public ReactiveWebServerFactoryCustomizer(ServerProperties serverProperties) { this.serverProperties = serverProperties; } @Override public int getOrder() { return 0; } @Override public void customize(ConfigurableReactiveWebServerFactory factory) { PropertyMapper map = PropertyMapper.get().alwaysApplyingWhenNonNull(); map.from(this.serverProperties::getPort).to(factory::setPort); map.from(this.serverProperties::getAddress).to(factory::setAddress); map.from(this.serverProperties::getSsl).to(factory::setSsl); map.from(this.serverProperties::getCompression).to(factory::setCompression); map.from(this.serverProperties::getHttp2).to(factory::setHttp2); } } 数据相关介绍 关系型数据 JDBC:数据源、 JdbcTemplate、自动装配 JPA:实体映射关系、实体操作、自动装配 事务∶ Spring事务抽象、JDBC事务处理、自动装配 JDBC 依赖 \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-jdbc\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; 依赖变化，在SpringBoot2.0 后依赖新的连接池 HikariCP\n数据源 javax.sql.DataSource JdbcTemplate 自动装配 DataSourceAutoConfiguration DataSource 连接池多数实现是对DataSource的二次封装，提高执行效率\npackage javax.sql; /** * @since 1.4 */ public interface DataSource extends CommonDataSource, Wrapper { Connection getConnection() throws SQLException; Connection getConnection(String username, String password) throws SQLException; } 自动装配 JdbcTemplate自动装配，其依赖数据源配置项，故有@AutoConfigureAfter注解标识，该类执行于DataSourceAutoConfiguration之后，额外还有多个条件装配的标识\npackage org.springframework.boot.autoconfigure.jdbc; /** * {@link EnableAutoConfiguration Auto-configuration} for {@link JdbcTemplate} and * {@link NamedParameterJdbcTemplate}. * @since 1.4.0 */ @Configuration @ConditionalOnClass({ DataSource.class, JdbcTemplate.class }) @ConditionalOnSingleCandidate(DataSource.class) @AutoConfigureAfter(DataSourceAutoConfiguration.class) @EnableConfigurationProperties(JdbcProperties.class) public class JdbcTemplateAutoConfiguration {...} DataSourceAutoConfiguration\npackage org.springframework.boot.autoconfigure.jdbc; /** * {@link EnableAutoConfiguration Auto-configuration} for {@link DataSource}. */ @Configuration @ConditionalOnClass({ DataSource.class, EmbeddedDatabaseType.class }) @EnableConfigurationProperties(DataSourceProperties.class) @Import({ DataSourcePoolMetadataProvidersConfiguration.class, DataSourceInitializationConfiguration.class }) public class DataSourceAutoConfiguration {...} 该类属于自动装配，故必然在spring.factories中，而数据源下即JdbcTemplateAutoConfiguration\n此时直接启动会得到缺少数据源的错误报告\n*************************** APPLICATION FAILED TO START *************************** Description: Failed to configure a DataSource: \u0026#39;url\u0026#39; attribute is not specified and no embedded datasource could be configured. Reason: Failed to determine a suitable driver class Action: Consider the following: If you want an embedded database (H2, HSQL or Derby), please put it on the classpath. If you have database settings to be loaded from a particular profile you may need to activate it (no profiles are currently active). JPA 依赖 \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-data-jpa\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; 其属于SpringData项目，基于hibernate实现，同时也自动依赖JDBC\n实体映射关系 @javax.persistence.OneToOne @javax.persistence.OneToMany @javax.persistence.ManyToOne @javax.persistence.ManyToMany \u0026hellip; 实体操作 javax.persistence.EntityManager 属于Java类库的API\npackage javax.persistence; public interface EntityManager { /** * 存 */ public void persist(Object entity); /** * 更新 */ public \u0026lt;T\u0026gt; T merge(T entity); /** * 删除 */ public void remove(Object entity); /** * 查询以及多个重载方法 */ public \u0026lt;T\u0026gt; T find(Class\u0026lt;T\u0026gt; entityClass, Object primaryKey); public \u0026lt;T\u0026gt; T find(Class\u0026lt;T\u0026gt; entityClass, Object primaryKey, Map\u0026lt;String, Object\u0026gt; properties); ... } 自动装配 HibernateJpaAutoConfiguration 该类也同属于自动装配，且需要JDBC提供数据源，故其加载顺序标识为DataSourceAutoConfiguration之后\npackage org.springframework.boot.autoconfigure.orm.jpa; /** * {@link EnableAutoConfiguration Auto-configuration} for Hibernate JPA. */ @Configuration @ConditionalOnClass({ LocalContainerEntityManagerFactoryBean.class, EntityManager.class }) @Conditional(HibernateEntityManagerCondition.class) @EnableConfigurationProperties(JpaProperties.class) @AutoConfigureAfter({ DataSourceAutoConfiguration.class }) @Import(HibernateJpaConfiguration.class) public class HibernateJpaAutoConfiguration {...} 事务（Transaction） 依赖 \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-tx\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; Spring 事务抽象 PlatformTransactionManager 该接口属于事务根接口，不仅限于数据库事务，默认实现包含分布式事务JtaTransactionManager和数据库事务DataSourceTransactionManager\npackage org.springframework.transaction; import org.springframework.lang.Nullable; /** * \u0026lt;p\u0026gt;The default implementations of this strategy interface are * {@link org.springframework.transaction.jta.JtaTransactionManager} and * {@link org.springframework.jdbc.datasource.DataSourceTransactionManager} * @see org.springframework.transaction.support.TransactionTemplate * @see org.springframework.transaction.interceptor.TransactionInterceptor * @see org.springframework.transaction.interceptor.TransactionProxyFactoryBean */ public interface PlatformTransactionManager {...} JDBC 事务处理 DataSourceTransactionManager 内部通过IOC自动注入数据源接口\npackage org.springframework.jdbc.datasource; @SuppressWarnings(\u0026#34;serial\u0026#34;) public class DataSourceTransactionManager extends AbstractPlatformTransactionManager implements ResourceTransactionManager, InitializingBean { @Nullable private DataSource dataSource; ... } 自动装配 TransactionAutoConfiguration 自动装配的条件是Spring事务依赖存在PlatformTransactionManager\npackage org.springframework.boot.autoconfigure.transaction; /** * {@link org.springframework.boot.autoconfigure.EnableAutoConfiguration * Auto-configuration} for Spring transaction. * @since 1.3.0 */ @Configuration @ConditionalOnClass(PlatformTransactionManager.class) @AutoConfigureAfter({ JtaAutoConfiguration.class, HibernateJpaAutoConfiguration.class, DataSourceTransactionManagerAutoConfiguration.class, Neo4jDataAutoConfiguration.class }) @EnableConfigurationProperties(TransactionProperties.class) public class TransactionAutoConfiguration {...} 功能扩展介绍 Spring Boot 应用 Spring Application:失败分析、应用特性、事件监听等 Spring Boot配置:外部化配置、 Profile、配置属性 Spring Boot starter: Starter开发、最佳实践 SpringApplication 失败分析 FailureAnalysisReporter 该类在Spring Boot 1.4后开始支持\npackage org.springframework.boot.diagnostics; /** * Reports a {@code FailureAnalysis} to the user. * @since 1.4.0 */ @FunctionalInterface public interface FailureAnalysisReporter {...} 应用特性 SpringApplication Fluent API 原始默认启动方法为SpringApplication.run(SpringbootApplication.class, args)其实际返回可配置上下文对象ConfigurableApplicationContext，可通过Builder模式重写\npackage top.fjy8018.springboot; @SpringBootApplication public class SpringbootApplication { public static void main(String[] args) { // Fluent API // 等价建造者设计模式调用方法 new SpringApplicationBuilder(SpringbootApplication.class) // 不使用web模式 .web(WebApplicationType.NONE) // 自定义参数 .properties(\u0026#34;name=fjy\u0026#34;) .run(args); } } 2.0 版本下提供3种WEB容器类型\npackage org.springframework.boot; /** * @since 2.0.0 */ public enum WebApplicationType { NONE, SERVLET, REACTIVE } Spring Boot 配置 外部化配置 ConfigurationProperty @Profile 从SpringBoot 2.0 后引入，Origin类型支持其他配资源引入，非传统仅支持键值对，由此支持多数据源\npackage org.springframework.boot.context.properties.source; /** * @since 2.0.0 */ public final class ConfigurationProperty implements OriginProvider, Comparable\u0026lt;ConfigurationProperty\u0026gt; { private final ConfigurationPropertyName name; private final Object value; private final Origin origin; ... } 官方文档\n此处官方文档存在一定疏漏，部分场景配置无法实现\nSpring Boot uses a very particular PropertySource order that is designed to allow sensible overriding of values. Properties are considered in the following order:\nDevtools global settings properties on your home directory (~/.spring-boot-devtools.properties when devtools is active). @TestPropertySource annotations on your tests. @SpringBootTest#properties annotation attribute on your tests. Command line arguments. Properties from SPRING_APPLICATION_JSON (inline JSON embedded in an environment variable or system property). ServletConfig init parameters. ServletContext init parameters. JNDI attributes from java:comp/env. Java System properties (System.getProperties()). OS environment variables. A RandomValuePropertySource that has properties only in random.*. Profile-specific application properties outside of your packaged jar (application-{profile}.properties and YAML variants). Profile-specific application properties packaged inside your jar (application-{profile}.properties and YAML variants). Application properties outside of your packaged jar (application.properties and YAML variants). Application properties packaged inside your jar (application.properties and YAML variants). @PropertySource annotations on your @Configuration classes. Default properties (specified by setting SpringApplication.setDefaultProperties). Profile支持以注解方式条件开启配置，用于区分开发环境和生产环境等\npackage org.springframework.context.annotation; /** * @since 3.1 */ @Target({ElementType.TYPE, ElementType.METHOD}) @Retention(RetentionPolicy.RUNTIME) @Documented @Conditional(ProfileCondition.class) public @interface Profile { String[] value(); } 配置属性 PropertySources 提供两种编程方式：接口、注解，支持保存多个配置，按照17种顺序排列，该接口在SpringBoot中重新设计变为ConfigurationProperty\npackage org.springframework.core.env; /** * @since 3.1 * @see PropertySource */ public interface PropertySources extends Iterable\u0026lt;PropertySource\u0026lt;?\u0026gt;\u0026gt; {...} 运维管理介绍 官方文档\nSpring Boot Actuator 端点（Endpoints） Web Endpoints JMX Endpoints 依赖 \u0026lt;!--运维监控--\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-actuator\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; 启动效果\n访问健康检查接口/actuator/health返回应用状态\n{ \u0026#34;status\u0026#34;: \u0026#34;UP\u0026#34; } 访问根路径\n{ \u0026#34;_links\u0026#34;: { \u0026#34;self\u0026#34;: { \u0026#34;href\u0026#34;: \u0026#34;http://localhost:8080/actuator\u0026#34;, \u0026#34;templated\u0026#34;: false }, \u0026#34;health\u0026#34;: { \u0026#34;href\u0026#34;: \u0026#34;http://localhost:8080/actuator/health\u0026#34;, \u0026#34;templated\u0026#34;: false }, \u0026#34;info\u0026#34;: { \u0026#34;href\u0026#34;: \u0026#34;http://localhost:8080/actuator/info\u0026#34;, \u0026#34;templated\u0026#34;: false } } } 依据SpringBoot打印的进程号或者依据应用名称使用Jconsole监控，该操作使用的为JMX端点\n健康检查（Health Checks） Health HealthIndicator 使用JMX在默认情况下能获取比Web端点更多的信息，原因是JMX支持使用安全连接，具有更好的安全性\n官方文档指出默认状态只暴露health和info，可通过配置文件调整\n28.4 Actuator Security For security purposes, all actuators other than /health and /info are disabled by default. The management.endpoints.web.exposure.include property can be used to enable the actuators.\n在application.properties文件中加入配置重启应用\n# 暴露所有健康监控点 management.endpoints.web.exposure.include=* 从日志输出可知增加了更多的URL映射\n但尽管配置后，JMX显示的信息仍然比Web端更丰富，例如在JMX中不仅看到状态，还能看到磁盘、内存等信息，而Web端仅看到状态\n指标（Metrics） 内建 Metrics Web Endpoint : /actuator/metrics 自定义 Metrics Web端点列举支持指标\n{ \u0026#34;names\u0026#34;: [ \u0026#34;jvm.memory.max\u0026#34;, \u0026#34;jvm.gc.memory.promoted\u0026#34;, \u0026#34;jvm.memory.used\u0026#34;, \u0026#34;jvm.gc.max.data.size\u0026#34;, \u0026#34;jvm.gc.pause\u0026#34;, \u0026#34;jvm.memory.committed\u0026#34;, \u0026#34;system.cpu.count\u0026#34;, \u0026#34;logback.events\u0026#34;, \u0026#34;jvm.buffer.memory.used\u0026#34;, \u0026#34;jvm.threads.daemon\u0026#34;, \u0026#34;system.cpu.usage\u0026#34;, \u0026#34;jvm.gc.memory.allocated\u0026#34;, \u0026#34;jvm.threads.live\u0026#34;, \u0026#34;jvm.threads.peak\u0026#34;, \u0026#34;process.uptime\u0026#34;, \u0026#34;http.server.requests\u0026#34;, \u0026#34;process.cpu.usage\u0026#34;, \u0026#34;jvm.classes.loaded\u0026#34;, \u0026#34;jvm.classes.unloaded\u0026#34;, \u0026#34;jvm.gc.live.data.size\u0026#34;, \u0026#34;jvm.buffer.count\u0026#34;, \u0026#34;jvm.buffer.total.capacity\u0026#34;, \u0026#34;process.start.time\u0026#34; ] } 访问/actuator/metrics/jvm.memory.max可查看相关指标，转换后可知最大内存为5.2G\n{ \u0026#34;name\u0026#34;: \u0026#34;jvm.memory.max\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;The maximum amount of memory in bytes that can be used for memory management\u0026#34;, \u0026#34;baseUnit\u0026#34;: \u0026#34;bytes\u0026#34;, \u0026#34;measurements\u0026#34;: [ { \u0026#34;statistic\u0026#34;: \u0026#34;VALUE\u0026#34;, \u0026#34;value\u0026#34;: 5595725823 } ], \u0026#34;availableTags\u0026#34;: [ { \u0026#34;tag\u0026#34;: \u0026#34;area\u0026#34;, \u0026#34;values\u0026#34;: [ \u0026#34;heap\u0026#34;, \u0026#34;nonheap\u0026#34; ] }, { \u0026#34;tag\u0026#34;: \u0026#34;id\u0026#34;, \u0026#34;values\u0026#34;: [ \u0026#34;Compressed Class Space\u0026#34;, \u0026#34;PS Survivor Space\u0026#34;, \u0026#34;PS Old Gen\u0026#34;, \u0026#34;Metaspace\u0026#34;, \u0026#34;PS Eden Space\u0026#34;, \u0026#34;Code Cache\u0026#34; ] } ] } ","date":"2020-12-10T17:11:00+08:00","permalink":"/archives/248/","title":"SpringBoot源码分析1——总览"},{"content":"会计科目分类 按资产性质划分 资产类科目 资产类科目是金融企业各项经济资源的分类名称，用来核算金融企业的各种资金运用结果。\n此科目包括流动资产、固定资产、无形资产和其他资产等。\n负债类科目 负债类科目是金融企业各项债务的分类名称用来核算金融企 业的各种债务性资金来源状况。\n此科目包括短期负债、长期负债和其他负债等。\n所有者权益类科目 所有者权益类科目是金融企业资本项目的分类名称，用来核算所有者投入资金以及经营所得或亏损情况。\n前者包括实收资本和资本公积，后者包括盈余公积和未分配利润。\n损益类科目 损益类科目是金融企业各项经营收人和费用支出的分类名称用来核算金融企业的营业收支等情况。\n此科目包括利息收入、利息支出、投资收益、营 业费用等。(4)损益类科目。\n资产负债共同类科目 资产负债共同类科目既能反映商业银行的资产业务，又能反映商业银行的负债业务。\n此科目的余额在借方表现为资产，余额在贷方表现为负债。\n在编制会计报表时应根据余额方向，分别纳入资产类或负债类科目反映。\n按与资产负债表的关系分类 表内科目 表内科目是指反映金融企业资金实际增减变化而纳入资产负债表内的科目\n表外科目 表外科目是指反映金融企业确已发生而尚未涉及资金的实际增减变化,或不涉及资金增减变化而不列人资产负债表的科目,如“有价单证”、“空白重要凭证”等\n按反映经济业务内容详略程度分类 一级科目 二级科目 记账方法 复式记账法（借贷记账法） 借贷记账法的记账原理。借贷记账法以“资产=负债+所有者权益”的会计平衡公式为记账原理。这一平衡公式体现了资产总额与负债和所有者权益总额之间数量上的平衡关系。\n借贷记账法的记账符号。借贷记账法的记账符号是“借”和“贷”。“借”在账户的左方，“贷“在账户的右方。\n借贷记账法的记账规则。借贷记账法以“有借必有贷，借贷必相等”为记账规则。\n资产和费用的增加记入借方\n资产和费用的减少记入贷方\n余额反映在借方\n负债、权益和收入的增加记入贷方\n负债、权益和收入的减少记入借方\n余额反映在贷方。遵守记账规则\n关键是在把经济业务记入账户之前，正确确定账户对应关系，编制会计分录。\n借贷记账法的试算平衡\n发生额平衡公式\n各科目借方本期发生额 = 各科目贷方本期发生额\n余额平衡公式\n各科目借方余额合计=各科目贷方余额合计\n单式记账法 单式记账法就是对发生的每一项经济业务只在一个账户中进行登记的记账方法。\n单式记账设置收入、付出、余额三栏,业务发生或增加时记“收入”,销账或减少时记“付出”,“余额”表示结存。\n账簿 序时账簿 序时账簿通常称为日记账,它是按照全部经济业务完成时间的先后 顺序进行登记的账簿。\n在实际工作中,按照会计部门收到凭证的先后顺序即按照凭证的编号顺序进行登记\n分类账簿 分类账簿是全面、系统地对资金循环和收支活动所作的分类记录。\n按照分类的概括程度不同,分类账簿又可分为\n明细分类账 总分类账 明细分类账 明细分类账是按账户设置和登记的账簿。\n金融企业的明细分类账按每一会计科目下的开户单位或具体核算对象分户立账,并根据记账凭证的业务顺序逐笔连续记载\n总分类账 总分类账是按会计科目设置的，是汇总明细分类账的主要工具。\n总分类账于每日营业终了，根据各科目日结单借、贷方发生额和今日余额进行登记。\n备查账簿 备查账簿是对某些在日记账和分类账等主要账簿中未能记载的事项进行补充登记的账簿。\n这种账簿属于辅助性登记，没有固定的格式，可根据需要加以设计。\n账务组织 账务组织是指总的账务结构体系,它包括明细核算和综合核算两个系统。明细核算对综合核算起到补充作用,综合核算对明细核算起到统驭作用,二者构成了一套完整的、科学的、严密的账务组织体系\n明细核算 明细核算是按账户核算，反映各项资金增减变化及其结果的详细情况的核算系统。\n它的具体形式和所使用的账表包括分户账、登记簿、余额表等。\n它的核算程序为：根据会计凭证登记分户账或登记簿，再根据分户账或登记簿编制余额表，最后同总账进行核对。\n分户账 分户账是企业会计账簿中具体反映经济业务的明细分类账簿，是明细核算的主要形式。分户账的格式分为甲、乙、丙、丁四种。\n登记簿 登记簿又称登记卡，是为了适应某些业务需要而设置的专用账簿。登记簿的格式根据业务需要可自行设计，一般都采用收入、付出、余额三栏的通用格式，根据表外科目传票采取单式记账方法进行登记处理。\n余额表 余额表是核对总账与分户账余额和计算利息的重要工具。余额表分为计息余额表和一般余额表。\n综合核算 综合核算是按会计科目进行核算,综合、概括地反映各科目的资金增减变化情况,由各种总分类账组成的核算系统。\n综合核算所采用的主要形式包括科目 日结单、总账、日计表等。\n其记账程序为:通过业务传票的审核编制科目日结单,然后根 据科目日结单登记各科目总账,最后编制日计表\n科目日结单 科目日结单又称总账记账凭证，是每天会计科目的当日借贷方发生额和凭证、附件张数的汇总记录，是监督明细账发生额、轧平当日账务的重要工具，也是记载总账的依据。\n每日营业终了，应按同一科目的凭证区分现金和转账凭证、借方或贷方发生额，各自加计总数填入科目日结单的有关栏目，并注明凭证张数。\n总账 总账是综合核算与明细核算相互核对，统驭明细分户账的主要工具，是各科目的总括记录，也是编制日计表、月计表、业务状况报告表、资产负债表的依据。总账按科目设置，设有借、贷方发生额和借、贷方余额四栏，账页每月更换一次。\n总账的发生额，应于每日营业终了，根据各科目日结单的借贷方发生额合计数分别填记；余额根据本科目的上日余额加减本日发生额后求出。\n检验科目总账余额是否正确应视不同情况而定\n如果上日借方余额大于贷方余额，可使用以下公式进行检验：\n上日借方余额-上日贷方余额+本日借方发生额-本日贷方发生额=本日借方余额-本日贷方余额\n如果上日贷方余额大于借方余额，可使用以下公式进行检验：\n上日贷方余额-上日借方余额+本日贷方发生额-本日借方发生额=本日贷方余额-本日借方余额\n日计表 日计表是反映当日业务活动和轧平全部账务的主要工具。日计表应每日编制，日计表上当日发生额和余额应根据总账各科目当日发生额和余额填记。\n日计表上借、贷方发生额和借、贷方余额合计数必须各自平衡\n存款业务 存款业务的分类 按存款期限划分 活期存款 定期存款 按存款对象划分 单位存款 储蓄存款 按资金性质划分 财政性存款 一般存款 按存款产生的来源划分 原始存款 派生存款 按存款币种划分 人民币存款 外币存款 存款账户的种类 基本存款账户 它是存款人办理日常转账结算和现金收付的账户。\n一般存款账户 它是存款人在基本存款账户以外的银行借款转存、与基本存款账户不在同一地点的、附属非独立核算单位开立的账户。\n临时存款账户 它是存款人因临时经营活动需要开立的账户。\n专用存款账户 它是存款人因为特定用途需要开立的账户。\n单位活期存款业务的核算 现金存入 支票户现金存入核算 分录：\n借:现金 贷:吸收存款——活期存款——XX单位户\n存折户现金存入核算 分录：\n借:现金 贷:吸收存款——活期存款——XX单位户\n现金支取 支票户现金支取核算 分录：\n借:吸收存款——活期存款——XX单位户 贷:现金\n存折户现金支取核算 分录：\n借:吸收存款一一活期存款一一XX单位户 贷:现金\n活期储蓄存款业务的核算 开户 分录：\n借:现金 贷:吸收存款——活期储蓄存款——XX存款人户\n销户 分录：\n借:吸收存款——活期储蓄存款——XX存款人户 利息支出——活期储蓄存款利息支出户 贷:现金\n续存 分录：\n借:现金 贷:吸收存款——活期储蓄存款——XX存款人户\n支取 分录：\n借:吸收存款——活期储蓄存款——XX存款人户 贷:现金\n","date":"2020-08-13T11:20:00+08:00","permalink":"/archives/247/","title":"金融企业会计笔记"},{"content":"模板方法模式 介绍 定义:定义了一个算法的骨架,并允许子类为一个或多个步骤提供实现\n模板方法使得子类可以在不改变算法结构的情况下,重新定义算法的某些步骤\n类型：行为型\n适用场景\n一次性实现一个算法的不变的部分,并将可变的行为留给子类来实现 各子类中公共的行为被提取出来并集中到一个公共父类中, 从而避免代码重复 优点\n提高复用性 提高扩展性 符合开闭原则 缺点\n类数目增加 增加了系统实现的复杂度 继承关系自身缺点,如果父类添加新的抽象方法,所有子类都要改一遍 分类和扩展\n钩子方法 提供了默认的行为，子类可以在必要时进行扩展 相关设计模式\n模板方法模式和工厂方法模式 工厂方法是模板方法的一种特殊实现 模板方法模式和策略模式 策略模式在于实现算法可以互相替换而不影响应用层，不改变算法流程 模板方法定义了一个算法的骨架，不明确的交给子类实现，可以改变算法流程 代码样例 类图\n抽象课程\npackage top.fjy8018.designpattern.pattern.behavior.templatemethod; import lombok.extern.slf4j.Slf4j; import org.apache.ibatis.mapping.MappedStatement; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import java.util.Collection; /** * 抽象课程 * \u0026lt;p\u0026gt; * {@link java.util.AbstractList} 提供抽象List实现 * 例如 {@link java.util.AbstractList#addAll(int, Collection)} 定义通过模板 * {@link java.util.AbstractMap} 同理 * \u0026lt;p\u0026gt; * {@link javax.servlet.http.HttpServlet#doGet(HttpServletRequest, HttpServletResponse)} 也是模板定义 * {@link org.apache.ibatis.executor.BaseExecutor#doUpdate(MappedStatement, Object)} 等方法交由子类实现 * 例如 {@link org.apache.ibatis.executor.SimpleExecutor#doUpdate(MappedStatement, Object)} * * @author F嘉阳 * @date 2020/3/4 10:59 */ @Slf4j public abstract class ACourse { /** * 固定不变的行为 * 定义模板，不允许子类覆盖，防止影响流程执行顺序 */ protected final void makeCourse() { this.makePPT(); this.makeVideo(); // 调用可选方法 if (needWriteArticle()) { this.writeArticle(); } this.packageCourse(); } /** * 固定不变的行为 * 满足所有子类需求 */ final void makePPT() { log.info(\u0026#34;制作PPT\u0026#34;); } final void makeVideo() { log.info(\u0026#34;制作视频\u0026#34;); } /** * 可选行为 * \u0026lt;p\u0026gt; * 手记可写可不写，但写的手记的流程还是一致的，故也声明为final * 通过定义钩子方法，允许子类覆盖 */ final void writeArticle() { log.info(\u0026#34;编写手记\u0026#34;); } /** * 钩子方法，提供默认实现 * * @return */ protected boolean needWriteArticle() { return false; } /** * 必须实现的行为 * 课程内容打包 */ abstract void packageCourse(); } 具体课程实现\npackage top.fjy8018.designpattern.pattern.behavior.templatemethod; import lombok.extern.slf4j.Slf4j; /** * 具体课程实现 * * @author F嘉阳 * @date 2020/3/4 11:05 */ @Slf4j public class JavaCourse extends ACourse { /** * 必须实现的行为 * 课程内容打包 */ @Override void packageCourse() { log.info(\u0026#34;提供Java学习课程源码\u0026#34;); } } package top.fjy8018.designpattern.pattern.behavior.templatemethod; import lombok.extern.slf4j.Slf4j; /** * 具体课程实现 * @author F嘉阳 * @date 2020/3/4 11:06 */ @Slf4j public class VueCourse extends ACourse { private boolean needWriteArticleFlag = false; /** * 通过构造器开放给实例决定 * @param needWriteArticleFlag */ public VueCourse(boolean needWriteArticleFlag) { this.needWriteArticleFlag = needWriteArticleFlag; } /** * 必须实现的行为 * 课程内容打包 */ @Override void packageCourse() { log.info(\u0026#34;提供Vue学习课程源码\u0026#34;); log.info(\u0026#34;提供Vue学习课程多媒体素材\u0026#34;); } /** * 钩子方法，提供默认实现 * * @return */ @Override protected boolean needWriteArticle() { return this.needWriteArticleFlag; } } 测试\npackage top.fjy8018.designpattern.pattern.behavior.templatemethod; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; /** * @author F嘉阳 * @date 2020/3/4 11:08 */ @Slf4j class ACourseTest { @Test void makeJavaCourse() { ACourse javaCourse = new JavaCourse(); javaCourse.makeCourse(); } @Test void makeVueCourse() { ACourse vueCourse = new VueCourse(true); vueCourse.makeCourse(); } } 输出\n11:27:56.942 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作PPT 11:27:56.957 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作视频 11:27:56.957 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.JavaCourse - 提供Java学习课程源码 11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作PPT 11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 制作视频 11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.ACourse - 编写手记 11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.VueCourse - 提供Vue学习课程源码 11:27:56.969 [main] INFO top.fjy8018.designpattern.pattern.behavior.templatemethod.VueCourse - 提供Vue学习课程多媒体素材 迭代器模式 介绍 定义提供一种方法,顺序访问一个集合对象中的各个元素,而又不暴露 该对象的内部表示\n类型：行为型\n适用场景\n访问一个集合对象的内容而无需暴露它的内部表示 为遍历不同的集合结构提供一个统一的接口 优点\n分离了集合对象的遍历行为 缺点\n类的个数成对增加 相关设计模式\n迭代器模式和访问者模式 访问者模式扩展开放的部分，再作用于对象的操作上 迭代器模式扩展开放的部分，再作用于对象的种类上 代码样例 迭代器一般使用现成的，而不会自己实现\n类图\n课程接口\npackage top.fjy8018.designpattern.pattern.behavior.iterator; import lombok.AllArgsConstructor; import lombok.Getter; /** * @author F嘉阳 * @date 2020/3/4 14:49 */ @AllArgsConstructor @Getter public class Course { private String name; } 迭代器\npackage top.fjy8018.designpattern.pattern.behavior.iterator; import java.util.ArrayList; /** * JDK迭代器 * {@link java.util.Iterator} * 集合内部类实现 {@link ArrayList.Itr#hasNext()} * 同时做了进一步扩展 {@link ArrayList.ListItr} * * mybatis中 {@link org.apache.ibatis.cursor.defaults.DefaultCursor} * 其持有游标迭代器对象 {@link org.apache.ibatis.cursor.defaults.DefaultCursor.CursorIterator} * * @author F嘉阳 * @date 2020/3/4 14:49 */ public interface CourseIterator { /** * 获取下一个课程 * * @return */ Course nextCourse(); /** * 是否是最后一个课程 * @return */ boolean isLastCourse(); } 迭代器实现\npackage top.fjy8018.designpattern.pattern.behavior.iterator; import lombok.extern.slf4j.Slf4j; import java.util.List; /** * @author F嘉阳 * @date 2020/3/4 14:50 */ @Slf4j public class CourseIteratorImpl implements CourseIterator { private List courseList; private int position; private Course course; public CourseIteratorImpl(List courseList){ this.courseList=courseList; } @Override public Course nextCourse() { log.info(\u0026#34;返回课程,位置是: {}\u0026#34;,position); course=(Course)courseList.get(position); position++; return course; } @Override public boolean isLastCourse(){ if(position\u0026lt; courseList.size()){ return false; } return true; } } 对课程进行处理\npackage top.fjy8018.designpattern.pattern.behavior.iterator; /** * 对课程进行处理 * * @author F嘉阳 * @date 2020/3/4 14:48 */ public interface CourseAggregate { void addCourse(Course course); void removeCourse(Course course); /** * 获取课程迭代器 * @return */ CourseIterator getCourseIterator(); } 实现\npackage top.fjy8018.designpattern.pattern.behavior.iterator; import java.util.ArrayList; import java.util.List; /** * @author F嘉阳 * @date 2020/3/4 14:50 */ public class CourseAggregateImpl implements CourseAggregate { private List courseList; public CourseAggregateImpl() { this.courseList = new ArrayList(); } @Override public void addCourse(Course course) { courseList.add(course); } @Override public void removeCourse(Course course) { courseList.remove(course); } @Override public CourseIterator getCourseIterator() { return new CourseIteratorImpl(courseList); } } 测试\npackage top.fjy8018.designpattern.pattern.behavior.iterator; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; /** * @author F嘉阳 * @date 2020/3/4 14:52 */ @Slf4j class CourseIteratorTest { @Test void nextCourse() { Course course1 = new Course(\u0026#34;Java电商一期\u0026#34;); Course course2 = new Course(\u0026#34;Java电商二期\u0026#34;); Course course3 = new Course(\u0026#34;Java设计模式精讲\u0026#34;); Course course4 = new Course(\u0026#34;Python课程\u0026#34;); Course course5 = new Course(\u0026#34;算法课程\u0026#34;); Course course6 = new Course(\u0026#34;前端课程\u0026#34;); CourseAggregate courseAggregate = new CourseAggregateImpl(); courseAggregate.addCourse(course1); courseAggregate.addCourse(course2); courseAggregate.addCourse(course3); courseAggregate.addCourse(course4); courseAggregate.addCourse(course5); courseAggregate.addCourse(course6); log.info(\u0026#34;-----课程列表-----\u0026#34;); printCourses(courseAggregate); courseAggregate.removeCourse(course4); courseAggregate.removeCourse(course5); log.info(\u0026#34;-----删除操作之后的课程列表-----\u0026#34;); printCourses(courseAggregate); } public static void printCourses(CourseAggregate courseAggregate){ CourseIterator courseIterator= courseAggregate.getCourseIterator(); while(!courseIterator.isLastCourse()){ Course course=courseIterator.nextCourse(); log.info(course.getName()); } } } 输出\n15:18:23.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - -----课程列表----- 15:18:23.124 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 0 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java分布式 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 1 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java系统设计 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 2 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java设计模式精讲 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 3 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Python课程 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 4 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 算法课程 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 5 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 前端课程 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - -----删除操作之后的课程列表----- 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 0 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java分布式 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 1 15:18:23.126 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java系统设计 15:18:23.127 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 2 15:18:23.127 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - Java设计模式精讲 15:18:23.132 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorImpl - 返回课程,位置是: 3 15:18:23.132 [main] INFO top.fjy8018.designpattern.pattern.behavior.iterator.CourseIteratorTest - 前端课程 策略模式 介绍 定义：定义了算法家族,分别封装起来,让它们之间可以互相替换, 此模式让算法的变化不会影响到使用算法的用户。\n例如有大量if…else的情况\n类型：行为型\n适用场景\n系统有很多类，而他们的区别仅仅在于他们的行为不同 一个系统需要动态地在几种算法中选择一种 优点\n开闭原则 避免使用多重条件转移语句 提高算法的保密性和安全性 缺点\n客户端必须知道所有的策略类,并自行决定使用哪一个策略类。 产生很多策略类 相关设计模式\n策略模式和工厂模式 策略模式和状态模式 代码样例 改造前——初级策略模式 无法避免if…else\n类图\n促销接口\npackage top.fjy8018.designpattern.pattern.behavior.strategy; /** * 促销接口 * * @author F嘉阳 * @date 2020/3/4 15:37 */ public interface PromotionStrategy { void doPromotion(); } 满减促销策略\npackage top.fjy8018.designpattern.pattern.behavior.strategy; import lombok.extern.slf4j.Slf4j; /** * 满减促销策略 * * @author F嘉阳 * @date 2020/3/4 15:38 */ @Slf4j public class ManJianPromotionStrategy implements PromotionStrategy { @Override public void doPromotion() { log.info(\u0026#34;满减促销，满200-20\u0026#34;); } } 立减促销策略\npackage top.fjy8018.designpattern.pattern.behavior.strategy; import lombok.extern.slf4j.Slf4j; /** * 立减促销策略 * * @author F嘉阳 * @date 2020/3/4 15:39 */ @Slf4j public class LiJianPromotionStrategy implements PromotionStrategy{ @Override public void doPromotion() { log.info(\u0026#34;立减促销，课程价格直接减去配置的价格\u0026#34;); } } 返现促销策略\npackage top.fjy8018.designpattern.pattern.behavior.strategy; import lombok.extern.slf4j.Slf4j; /** * 返现促销策略 * * @author F嘉阳 * @date 2020/3/4 15:38 */ @Slf4j public class FanXianPromotionStrategy implements PromotionStrategy{ @Override public void doPromotion() { log.info(\u0026#34;返现促销，返回金额存放到用户余额中\u0026#34;); } } 测试\npackage top.fjy8018.designpattern.pattern.behavior.strategy; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang3.StringUtils; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; /** * @author F嘉阳 * @date 2020/3/4 15:46 */ @Slf4j class PromotionStrategyTest { @Test void existIfElse() { String promotionKey = \u0026#34;LIJIAN\u0026#34;; PromotionActivity activity = null; if (StringUtils.equals(promotionKey,\u0026#34;LIJIAN\u0026#34;)){ // 方法和类本身线程安全 activity= new PromotionActivity(new LiJianPromotionStrategy()); }else if(StringUtils.equals(promotionKey,\u0026#34;MANJIAN\u0026#34;)){ activity = new PromotionActivity(new FanXianPromotionStrategy()); } activity.executePromotionStrategy(); } } 输出\n16:08:28.582 [main] INFO top.fjy8018.designpattern.pattern.behavior.strategy.LiJianPromotionStrategy - 立减促销，课程价格直接减去配置的价格 改造后——结合工厂和策略模式 类图\n空促销策略\npackage top.fjy8018.designpattern.pattern.behavior.strategy; import lombok.extern.slf4j.Slf4j; /** * 空策略 * * @author F嘉阳 * @date 2020/3/4 16:16 */ @Slf4j public class EmptyPromotionStrategy implements PromotionStrategy { @Override public void doPromotion() { log.info(\u0026#34;空促销策略\u0026#34;); } } 促销策略工厂\npackage top.fjy8018.designpattern.pattern.behavior.strategy; import org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory; import org.springframework.beans.factory.support.CglibSubclassingInstantiationStrategy; import org.springframework.beans.factory.support.SimpleInstantiationStrategy; import org.springframework.core.io.FileSystemResource; import org.springframework.core.io.PathResource; import java.util.Arrays; import java.util.Comparator; import java.util.Map; import java.util.concurrent.ConcurrentHashMap; /** * 促销策略工厂 * \u0026lt;p\u0026gt; * {@link java.util.Comparator} 比较器，类似于抽象策略，有大量实现类，所有具体策略都要实现定义好的接口方法 * 例如 {@link com.google.common.collect.ExplicitOrdering} * 应用层可以逐级传递比较器 {@link Arrays#sort(Object[], Comparator)} * {@link java.util.TreeMap#compare(Object, Object)} 利用内部的比较器处理 * \u0026lt;p\u0026gt; * Spring资源策略抽象 {@link org.springframework.core.io.Resource} * 提供大量策略实现 {@link PathResource} {@link FileSystemResource} * \u0026lt;p\u0026gt; * {@link org.springframework.beans.factory.support.InstantiationStrategy} 初始化策略接口 * 初始化具体策略实现 {@link SimpleInstantiationStrategy} {@link CglibSubclassingInstantiationStrategy} * 初始化策略在 {@link AbstractAutowireCapableBeanFactory} 中被使用 * * @author F嘉阳 * @date 2020/3/4 16:09 */ public class PromotionStrategyFactory { private static Map\u0026lt;String, PromotionStrategy\u0026gt; STRATEGY_MAP = new ConcurrentHashMap\u0026lt;\u0026gt;(); /** * 无促销 */ private static final PromotionStrategy NON_PROMOTION = new EmptyPromotionStrategy(); static { STRATEGY_MAP.put(PromotionKey.FANXIAN, new FanXianPromotionStrategy()); STRATEGY_MAP.put(PromotionKey.MANJIAN, new ManJianPromotionStrategy()); STRATEGY_MAP.put(PromotionKey.LIJIAN, new LiJianPromotionStrategy()); } /** * 禁止实例化 */ private PromotionStrategyFactory() { } /** * 开放获取策略的方法 * * @param promotionKey * @return */ public static PromotionStrategy getPromotionStrategy(String promotionKey) { PromotionStrategy strategy = STRATEGY_MAP.get(promotionKey); return strategy == null ? NON_PROMOTION : strategy; } /** * 常量定义 */ private interface PromotionKey { String LIJIAN = \u0026#34;LIJIAN\u0026#34;; String FANXIAN = \u0026#34;FANXIAN\u0026#34;; String MANJIAN = \u0026#34;MANJIAN\u0026#34;; } } 测试\npackage top.fjy8018.designpattern.pattern.behavior.strategy; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang3.StringUtils; import org.junit.jupiter.api.Test; /** * @author F嘉阳 * @date 2020/3/4 15:46 */ @Slf4j class PromotionStrategyTest { @Test void nonIfElse() { String promotionKey = \u0026#34;LIJIAN\u0026#34;; PromotionActivity activity = new PromotionActivity(PromotionStrategyFactory.getPromotionStrategy(promotionKey)); activity.executePromotionStrategy(); } } 输出\n16:08:28.582 [main] INFO top.fjy8018.designpattern.pattern.behavior.strategy.LiJianPromotionStrategy - 立减促销，课程价格直接减去配置的价格 解释器模式 介绍 定义：给定一个语言，定义它的文法的一种表示，并定义一个解释器，这个解释器使用该表示来解释语言中的句子。 为了解释一种语言，而为语言创建的解释器 类型：行为型 适用场景（较少）\n某个特定类型问题发生频率足够高 优点\n语法由很多类表示,容易改变及扩展此语言 缺点\n当语法规则数目太多时,增加了系统复杂度 相关设计模式\n解释器模式和适配器模式 代码样例 类图\n解释器\npackage top.fjy8018.designpattern.pattern.behavior.interpreter; /** * 解释器 * \u0026lt;p\u0026gt; * JDK 源码 * {@link java.util.regex.Pattern} 正则解释器 * {@link org.springframework.expression.spel.standard.SpelExpressionParser} Spring解释器 * * @author F嘉阳 * @date 2020/3/5 14:45 */ public interface Interpreter { int interpret(); } 加法解释器\npackage top.fjy8018.designpattern.pattern.behavior.interpreter; /** * 加法解释器 * * @author F嘉阳 * @date 2020/3/5 14:46 */ public class AddInterpreter implements Interpreter { /** * 定义两个表达式 */ private Interpreter firstExpression, secondExpression; public AddInterpreter(Interpreter firstExpression, Interpreter secondExpression) { this.firstExpression = firstExpression; this.secondExpression = secondExpression; } @Override public int interpret() { // 此时传入为NumberInterpreter类型，调interpret方法转int类型 return this.firstExpression.interpret() + this.secondExpression.interpret(); } @Override public String toString() { return \u0026#34;+\u0026#34;; } } 乘法解释器\npackage top.fjy8018.designpattern.pattern.behavior.interpreter; /** * 乘法解释器 * * @author F嘉阳 * @date 2020/3/5 14:47 */ public class MultiInterpreter implements Interpreter { private Interpreter firstExpression, secondExpression; public MultiInterpreter(Interpreter firstExpression, Interpreter secondExpression) { this.firstExpression = firstExpression; this.secondExpression = secondExpression; } @Override public int interpret() { return this.firstExpression.interpret() * this.secondExpression.interpret(); } @Override public String toString() { return \u0026#34;*\u0026#34;; } } 类型解释器\npackage top.fjy8018.designpattern.pattern.behavior.interpreter; /** * @author F嘉阳 * @date 2020/3/5 14:47 */ public class NumberInterpreter implements Interpreter { private int number; public NumberInterpreter(int number) { this.number = number; } public NumberInterpreter(String number) { this.number = Integer.parseInt(number); } @Override public int interpret() { return this.number; } } 操作工具类\npackage top.fjy8018.designpattern.pattern.behavior.interpreter; /** * @author F嘉阳 * @date 2020/3/5 14:47 */ public class OperatorUtil { public static boolean isOperator(String symbol) { return (symbol.equals(\u0026#34;+\u0026#34;) || symbol.equals(\u0026#34;*\u0026#34;)); } public static Interpreter getExpressionObject(Interpreter firstExpression, Interpreter secondExpression, String symbol) { if (symbol.equals(\u0026#34;+\u0026#34;)) { return new AddInterpreter(firstExpression, secondExpression); } else if (symbol.equals(\u0026#34;*\u0026#34;)) { return new MultiInterpreter(firstExpression, secondExpression); } return null; } } 自定义解释器实现\npackage top.fjy8018.designpattern.pattern.behavior.interpreter; import lombok.extern.slf4j.Slf4j; import java.util.Stack; /** * @author F嘉阳 * @date 2020/3/5 14:48 */ @Slf4j public class CustomExpressionParser { private Stack\u0026lt;Interpreter\u0026gt; stack = new Stack\u0026lt;Interpreter\u0026gt;(); public int parse(String str) { String[] strItemArray = str.split(\u0026#34; \u0026#34;); for (String symbol : strItemArray) { if (!OperatorUtil.isOperator(symbol)) { Interpreter numberExpression = new NumberInterpreter(symbol); stack.push(numberExpression); log.info(String.format(\u0026#34;入栈: %d\u0026#34;, numberExpression.interpret())); } else { //是运算符号，可以计算 Interpreter firstExpression = stack.pop(); Interpreter secondExpression = stack.pop(); log.info(String.format(\u0026#34;出栈: %d 和 %d\u0026#34;, firstExpression.interpret(), secondExpression.interpret())); Interpreter operator = OperatorUtil.getExpressionObject(firstExpression, secondExpression, symbol); log.info(String.format(\u0026#34;应用运算符: %s\u0026#34;, operator)); int result = operator.interpret(); NumberInterpreter resultExpression = new NumberInterpreter(result); stack.push(resultExpression); log.info(String.format(\u0026#34;阶段结果入栈: %d\u0026#34;, resultExpression.interpret())); } } int result = stack.pop().interpret(); return result; } } 测试\npackage top.fjy8018.designpattern.pattern.behavior.interpreter; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import org.springframework.expression.Expression; import org.springframework.expression.spel.standard.SpelExpressionParser; /** * @author F嘉阳 * @date 2020/3/5 14:48 */ @Slf4j class InterpreterTest { @Test void interpret() { String myInterpreter = \u0026#34;6 100 11 + *\u0026#34;; CustomExpressionParser expressionParser = new CustomExpressionParser(); int result = expressionParser.parse(myInterpreter); log.info(\u0026#34;解释器计算结果: \u0026#34; + result); } @Test void springTest() { org.springframework.expression.ExpressionParser parser = new SpelExpressionParser(); Expression expression = parser.parseExpression(\u0026#34;100 * 2 + 400 * 1 + 66\u0026#34;); int result = (Integer) expression.getValue(); log.info(String.valueOf(result)); } } 输出\n14:52:28.086 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 6 14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 100 14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 入栈: 11 14:52:28.089 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 出栈: 11 和 100 14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 应用运算符: + 14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 阶段结果入栈: 111 14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 出栈: 111 和 6 14:52:28.090 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 应用运算符: * 14:52:28.091 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.GeelyExpressionParser - 阶段结果入栈: 666 14:52:28.091 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.InterpreterTest - 解释器计算结果: 666 14:52:28.550 [main] INFO top.fjy8018.designpattern.pattern.behavior.interpreter.InterpreterTest - 666 观察者模式 介绍 定义：定义了对象之间的一对多依赖，让多个观察者对象同时监听某一个主题对象，当主题对象发生变化时，它的所有依赖者（观察者）都会收到通知并更新\n类型：行为型\n适用场景\n关联行为场景，建立一套触发机制 优点\n观察者模式支持广播通信 观察者和被观察者之间建立一个抽象的耦合 缺点\n使用要得当,要避免循环调用 观察者之间有过多的细节依赖、提高时间消耗及程序复杂度 代码样例——JDK观察者 类图\n课程问题\npackage top.fjy8018.designpattern.pattern.behavior.observer; import lombok.AllArgsConstructor; import lombok.Getter; /** * 课程问题 * * @author F嘉阳 * @date 2020/3/5 15:23 */ @Getter @AllArgsConstructor public class Question { private String username; private String content; } 课程（被观察者）\npackage top.fjy8018.designpattern.pattern.behavior.observer; import lombok.AllArgsConstructor; import lombok.Getter; import lombok.extern.slf4j.Slf4j; import java.util.EventListener; import java.util.Observable; import java.util.Vector; /** * 课程 * 被观察者类 * * {@link Observable} 使用 {@link Vector} 保证线程安全 * * JDK源码 * {@link java.awt.Event} 事件监听器 * {@link javax.servlet.ServletRequestListener} HTTP请求事件监听器 * 继承 {@link EventListener}，有大量实现，例如{@link org.springframework.beans.factory.parsing.ReaderEventListener} * * @author F嘉阳 * @date 2020/3/5 15:23 */ @Slf4j @Getter @AllArgsConstructor public class Course extends Observable { private String courseName; /** * 产生问题 * @param course * @param question */ public void produceQuestion(Course course,Question question){ log.info(\u0026#34;{}在{}提交了一个问题\u0026#34;,question.getUsername(),course.getCourseName()); // 状态变化 setChanged(); // 通知观察者 notifyObservers(question); } } 老师实体（观察者）\npackage top.fjy8018.designpattern.pattern.behavior.observer; import lombok.AllArgsConstructor; import lombok.Getter; import lombok.extern.slf4j.Slf4j; import java.util.Observable; import java.util.Observer; /** * 老师实体 * * 老师观察课程是否产生了新的问题 * @author F嘉阳 * @date 2020/3/5 15:24 */ @Slf4j @Getter @AllArgsConstructor public class Teacher implements Observer { private String teacherName; /** * This method is called whenever the observed object is changed. An * application calls an \u0026lt;tt\u0026gt;Observable\u0026lt;/tt\u0026gt; object\u0026#39;s * \u0026lt;code\u0026gt;notifyObservers\u0026lt;/code\u0026gt; method to have all the object\u0026#39;s * observers notified of the change. * * @param o the observable object. * @param arg an argument passed to the \u0026lt;code\u0026gt;notifyObservers\u0026lt;/code\u0026gt; */ @Override public void update(Observable o, Object arg) { Course course = (Course) o; Question question = (Question) arg; log.info(\u0026#34;{}老师的{}课程接收到一个{}提交的问题，内容为：{}\u0026#34;, teacherName,course.getCourseName(),question.getUsername(),question.getContent()); } } 测试\npackage top.fjy8018.designpattern.pattern.behavior.observer; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; /** * @author F嘉阳 * @date 2020/3/5 15:36 */ @Slf4j class TeacherTest { @Test void update() { Course course = new Course(\u0026#34;Java设计模式\u0026#34;); Teacher teacher = new Teacher(\u0026#34;fjy8018\u0026#34;); course.addObserver(teacher); // 业务逻辑 Question question = new Question(\u0026#34;fjy\u0026#34;,\u0026#34;问题内容XXX\u0026#34;); course.produceQuestion(course,question); } } 输出\n15:38:34.437 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.Course - fjy在Java设计模式提交了一个问题 15:38:34.444 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.Teacher - fjy8018老师的Java设计模式课程接收到一个fjy提交的问题，内容为：问题内容XXX 代码样例——Guava监听器 package top.fjy8018.designpattern.pattern.behavior.observer.guava; import com.google.common.eventbus.Subscribe; import lombok.extern.slf4j.Slf4j; import java.util.concurrent.ConcurrentMap; import java.util.concurrent.CopyOnWriteArraySet; /** * Guava事件监听器 * {@link EventBus} 使用内部对象 {@link SubscriberRegistry} 内部使用 {@link ConcurrentMap} 存储订阅者 * 同时使用 {@link CopyOnWriteArraySet} 保证线程安全和订阅者唯一 * 故{@link com.google.common.eventbus.Subscriber} * 重写了 {@link com.google.common.eventbus.Subscriber#equals(Object)} * 和 {@link com.google.common.eventbus.Subscriber#hashCode()} 方法 * * @author F嘉阳 * @date 2020/3/5 15:56 */ @Slf4j public class GuavaEvent { @Subscribe public void subscribe(String str) { // 业务逻辑 log.info(\u0026#34;执行订阅方法，传入参数是：{}\u0026#34;, str); } } 测试\npackage top.fjy8018.designpattern.pattern.behavior.observer; import com.google.common.eventbus.EventBus; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import top.fjy8018.designpattern.pattern.behavior.observer.guava.Course; import top.fjy8018.designpattern.pattern.behavior.observer.guava.GuavaEvent; /** * @author F嘉阳 * @date 2020/3/5 15:58 */ @Slf4j class GuavaEventTest { @Test void subscribe() { EventBus eventBus = new EventBus(); GuavaEvent event = new GuavaEvent(); // 注册 eventBus.register(event); eventBus.post(\u0026#34;提交事件的内容\u0026#34;); } } 输出\n16:24:27.716 [main] INFO top.fjy8018.designpattern.pattern.behavior.observer.GuavaEvent - 执行订阅方法，传入参数是：提交事件的内容 备忘录模式 介绍 定义：保存一个对象的某个状态，以便在适当的时候恢复对象。（类似历史记录存储）\n“后悔药”\n类型：行为型\n适用场景\n保存及恢复数据相关业务场景 后悔的时候，即想恢复到之前的状态 优点\n为用户提供一种可恢复机制 存档信息的封装 缺点\n资源占用 相关设计模式\n备忘录模式和状态模式 代码样例 类图\n文章\npackage top.fjy8018.designpattern.pattern.behavior.memento; import lombok.AllArgsConstructor; import lombok.Data; /** * 文章，字段不宜过多，否则导致过大内存占用 * * @author F嘉阳 * @date 2020/3/6 17:00 */ @Data @AllArgsConstructor public class Article { private String title; private String content; private String imgs; /** * 保存 * @return */ public ArticleMemento saveToMemento() { ArticleMemento articleMemento = new ArticleMemento(this.title,this.content,this.imgs); return articleMemento; } /** * 撤销 * @param articleMemento */ public void undoFromMemento(ArticleMemento articleMemento) { this.title = articleMemento.getTitle(); this.content = articleMemento.getContent(); this.imgs = articleMemento.getImgs(); } } 文章备忘录（快照）\npackage top.fjy8018.designpattern.pattern.behavior.memento; import lombok.AllArgsConstructor; import lombok.Getter; import lombok.ToString; import org.springframework.binding.message.StateManageableMessageContext; /** * 文章备忘录（快照） * 快照不需要set方法，只能通过构造器注入 * * Spring工作流中源码 {@link StateManageableMessageContext#createMessagesMemento()} * * @author F嘉阳 * @date 2020/3/6 17:03 */ @Getter @ToString @AllArgsConstructor public class ArticleMemento { private String title; private String content; private String imgs; } 备忘录核心，栈实现管理者\npackage top.fjy8018.designpattern.pattern.behavior.memento; import java.util.Stack; /** * 保存一个对象的某个状态，以便在适当的时候恢复对象 * * @author F嘉阳 * @date 2020/3/6 17:04 */ public class ArticleMementoManager { /** * 使用栈特性实现动作撤销 */ private final Stack\u0026lt;ArticleMemento\u0026gt; ARTICLE_MEMENTO_STACK = new Stack\u0026lt;ArticleMemento\u0026gt;(); public ArticleMemento getMemento() { ArticleMemento articleMemento= ARTICLE_MEMENTO_STACK.pop(); return articleMemento; } public void addMemento(ArticleMemento articleMemento) { ARTICLE_MEMENTO_STACK.push(articleMemento); } } 测试\npackage top.fjy8018.designpattern.pattern.behavior.memento; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; /** * @author F嘉阳 * @date 2020/3/6 17:07 */ @Slf4j class ArticleMementoTest { @Test void getTitle() { ArticleMementoManager articleMementoManager = new ArticleMementoManager(); Article article= new Article(\u0026#34;如影随行的设计模式A\u0026#34;,\u0026#34;手记内容A\u0026#34;,\u0026#34;手记图片A\u0026#34;); // 存档 ArticleMemento articleMemento = article.saveToMemento(); articleMementoManager.addMemento(articleMemento); log.info(\u0026#34;标题:\u0026#34;+article.getTitle()+\u0026#34; 内容:\u0026#34;+article.getContent()+\u0026#34; 图片:\u0026#34;+article.getImgs()+\u0026#34; 暂存成功\u0026#34;); log.info(\u0026#34;手记完整信息:\u0026#34;+article); log.info(\u0026#34;修改手记start\u0026#34;); article.setTitle(\u0026#34;如影随行的设计模式B\u0026#34;); article.setContent(\u0026#34;手记内容B\u0026#34;); article.setImgs(\u0026#34;手记图片B\u0026#34;); log.info(\u0026#34;修改手记end\u0026#34;); log.info(\u0026#34;手记完整信息:\u0026#34;+article); articleMemento = article.saveToMemento(); articleMementoManager.addMemento(articleMemento); article.setTitle(\u0026#34;如影随行的设计模式C\u0026#34;); article.setContent(\u0026#34;手记内容C\u0026#34;); article.setImgs(\u0026#34;手记图片C\u0026#34;); log.info(\u0026#34;暂存回退start\u0026#34;); // 取档 log.info(\u0026#34;回退出栈1次\u0026#34;); articleMemento = articleMementoManager.getMemento(); article.undoFromMemento(articleMemento); log.info(\u0026#34;回退出栈2次\u0026#34;); articleMemento = articleMementoManager.getMemento(); article.undoFromMemento(articleMemento); log.info(\u0026#34;暂存回退end\u0026#34;); log.info(\u0026#34;手记完整信息:\u0026#34;+article); } } 输出\n17:33:51.154 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 标题:如影随行的设计模式A 内容:手记内容A 图片:手记图片A 暂存成功 17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式A, content=手记内容A, imgs=手记图片A) 17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 修改手记start 17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 修改手记end 17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式B, content=手记内容B, imgs=手记图片B) 17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 暂存回退start 17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 回退出栈1次 17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 回退出栈2次 17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 暂存回退end 17:33:51.169 [main] INFO top.fjy8018.designpattern.pattern.behavior.memento.ArticleMementoTest - 手记完整信息:Article(title=如影随行的设计模式A, content=手记内容A, imgs=手记图片A) 命令模式 介绍 定义：将请求封装成对象,以便使用不同的请求\n特点：命令模式解决了应用程序中对象的职责以及它们之间的通信方式\n类型：行为型\n适用场景\n请求调用者和请求接收者需要解耦,使得调用者和接收者不直接交互 需要抽象出等待执行的行为 优点\n降低耦合 容易扩展新命令或者一组命令 缺点\n命令的无限扩展会增加类的数量，提高系统实现复杂度 相关设计模式\n命令模式和备忘录模式 代码样例 类图\n命令抽象\npackage top.fjy8018.designpattern.pattern.behavior.command; /** * 命令抽象 * \u0026lt;p\u0026gt; * Java相关源码 {@link Runnable} 可理解为命令抽象 * 其实现类为具体的命令 * * @author F嘉阳 * @date 2020/7/21 15:58 */ public interface Command { void execute(); } 操作视频\npackage top.fjy8018.designpattern.pattern.behavior.command; import lombok.AllArgsConstructor; /** * 关闭视频命令 * * @author F嘉阳 * @date 2020/7/21 15:58 */ @AllArgsConstructor public class CloseCourseVideoCommand implements Command { private final CourseVideo courseVideo; @Override public void execute() { courseVideo.close(); } } 打开视频命令\npackage top.fjy8018.designpattern.pattern.behavior.command; import lombok.AllArgsConstructor; /** * 打开视频命令 * * @author F嘉阳 * @date 2020/7/21 15:58 */ @AllArgsConstructor public class OpenCourseVideoCommand implements Command { private final CourseVideo courseVideo; @Override public void execute() { courseVideo.open(); } } 关闭视频命令\npackage top.fjy8018.designpattern.pattern.behavior.command; import lombok.AllArgsConstructor; /** * 关闭视频命令 * * @author F嘉阳 * @date 2020/7/21 15:58 */ @AllArgsConstructor public class CloseCourseVideoCommand implements Command { private final CourseVideo courseVideo; @Override public void execute() { courseVideo.close(); } } 命令执行人\npackage top.fjy8018.designpattern.pattern.behavior.command; import java.util.ArrayList; import java.util.List; /** * 命令执行人 * * @author F嘉阳 * @date 2020/7/21 15:58 */ public class Staff { /** * 保证执行顺序 */ private final List\u0026lt;Command\u0026gt; commandList = new ArrayList\u0026lt;Command\u0026gt;(); public void addCommand(Command command) { commandList.add(command); } public void executeCommands() { for (Command command : commandList) { command.execute(); } commandList.clear(); } } 测试\npackage top.fjy8018.designpattern.pattern.behavior.command; import org.junit.jupiter.api.Test; /** * @author F嘉阳 * @date 2020/7/21 16:02 */ class CommandTest { @Test void doTest() { CourseVideo courseVideo = new CourseVideo(\u0026#34;Java设计模式精讲\u0026#34;); OpenCourseVideoCommand openCourseVideoCommand = new OpenCourseVideoCommand(courseVideo); CloseCourseVideoCommand closeCourseVideoCommand = new CloseCourseVideoCommand(courseVideo); Staff staff = new Staff(); staff.addCommand(openCourseVideoCommand); staff.addCommand(closeCourseVideoCommand); staff.executeCommands(); } } 输出\nJava设计模式精讲课程视频开放 Java设计模式精讲课程视频关闭 中介者模式 介绍 定义：定义一个 封装一组对象如何交互的对象\n特点：通过使对象明确地相互引用来促进松散耦合,并允许独立地改变它们的交互\n适用场景\n系统中对象之间存在复杂的引用关系，产生的相互依赖关系结构混乱且难以理解 交互的公共行为，如果需要改变行为则可以增加新的中介者类 优点\n将一对多转化成了一对一、降低程序复杂度 类之间解耦 缺点\n中介者过多,导致系统复杂 相关设计模式\n中介者模式和观察者模式 代码样例 类图\n学习群组（中介者）\npackage top.fjy8018.designpattern.pattern.behavior.mediator; import java.util.Date; import java.util.Timer; /** * 学习群组（中介者） * \u0026lt;p\u0026gt; * jdk源码 {@link Timer#schedule(java.util.TimerTask, Date)} 多个方法重载 * 都是调用{@link Timer#sched(java.util.TimerTask, long, long)} 实现的，由底层方法进行整体协调 * * @author F嘉阳 * @date 2020/7/21 16:20 */ public class StudyGroup { public static void showMessage(User user, String message) { System.out.println(new Date().toString() + \u0026#34; [\u0026#34; + user.getName() + \u0026#34;] : \u0026#34; + message); } } 用户\npackage top.fjy8018.designpattern.pattern.behavior.mediator; import lombok.AllArgsConstructor; import lombok.Data; /** * @author F嘉阳 * @date 2020/7/21 16:20 */ @Data @AllArgsConstructor public class User { private String name; public void sendMessage(String message) { // 只和中介者交互 StudyGroup.showMessage(this, message); } } 测试\npackage top.fjy8018.designpattern.pattern.behavior.mediator; import org.junit.jupiter.api.Test; /** * @author F嘉阳 * @date 2020/7/21 16:25 */ class MediatorTest { @Test void doTest() { User fjy = new User(\u0026#34;F嘉阳\u0026#34;); User tom = new User(\u0026#34;Tom\u0026#34;); fjy.sendMessage(\u0026#34; Hey! Tom! Let\u0026#39;s learn Design Pattern\u0026#34;); tom.sendMessage(\u0026#34;OK! F嘉阳\u0026#34;); } } 输出\nTue Jul 21 16:47:03 CST 2020 [F嘉阳] : Hey! Tom! Let\u0026#39;s learn Design Pattern Tue Jul 21 16:47:03 CST 2020 [Tom] : OK! F嘉阳 责任链模式 介绍 定义：为请求创建一个接收此次请求对象的链\n类型：行为型\n适用场景\n一个请求的处理需要多个对象当中的一个或几个协作处理 优点\n请求的发送者和接收者(请求的处理)解耦 责任链可以动态组合 缺点\n责任链太长或者处理时间过长,影响性能 责任链有可能过多 相关设计模式\n责任链模式和状态模式 代码样例 类图\n模拟二级审批场景\npackage top.fjy8018.designpattern.pattern.behavior.chainofresponsibility; import javax.servlet.FilterChain; import javax.servlet.ServletRequest; import javax.servlet.ServletResponse; /** * 模拟二级审批场景 * \u0026lt;p\u0026gt; * 审批抽象 * \u0026lt;p\u0026gt; * java 过滤器实现 {@link javax.servlet.Filter#doFilter(ServletRequest, ServletResponse, FilterChain)} * 实现 {@link FilterChain#doFilter(ServletRequest, ServletResponse)} 接口的组成一条过滤器链 * 所有子类通过调用 chain.doFilter(request, response) 实现链式传递 * \u0026lt;p\u0026gt; * SpringSecurity过滤器链实现也类似 * * @author F嘉阳 * @date 2020/7/21 16:59 */ public abstract class Approver { /** * 责任链核心：包含与自身相同的类型对象 */ protected Approver approver; /** * 设置下一个节点，组成链条 * * @param approver */ public void setNextApprover(Approver approver) { this.approver = approver; } /** * 交由子类实现 * * @param course */ public abstract void deploy(Course course); } 手记审批\npackage top.fjy8018.designpattern.pattern.behavior.chainofresponsibility; import org.apache.commons.lang3.StringUtils; /** * 手记审批 * * @author F嘉阳 * @date 2020/7/21 16:59 */ public class ArticleApprover extends Approver { @Override public void deploy(Course course) { if (StringUtils.isNotEmpty(course.getArticle())) { System.out.println(course.getName() + \u0026#34;含有手记,批准\u0026#34;); // 是否为末尾 if (approver != null) { // 执行下一个处理链 approver.deploy(course); } } else { System.out.println(course.getName() + \u0026#34;不含有手记,不批准,流程结束\u0026#34;); return; } } } 视频审批\npackage top.fjy8018.designpattern.pattern.behavior.chainofresponsibility; import org.apache.commons.lang3.StringUtils; /** * 视频审批 * * @author F嘉阳 * @date 2020/7/21 16:59 */ public class VideoApprover extends Approver { @Override public void deploy(Course course) { if (StringUtils.isNotEmpty(course.getVideo())) { System.out.println(course.getName() + \u0026#34;含有视频,批准\u0026#34;); if (approver != null) { approver.deploy(course); } } else { System.out.println(course.getName() + \u0026#34;不含有视频,不批准,流程结束\u0026#34;); return; } } } 课程\npackage top.fjy8018.designpattern.pattern.behavior.chainofresponsibility; import lombok.Data; /** * 课程 * * @author F嘉阳 * @date 2020/7/21 16:59 */ @Data public class Course { private String name; private String article; private String video; } 测试\npackage top.fjy8018.designpattern.pattern.behavior.chainofresponsibility; import org.junit.jupiter.api.Test; /** * @author F嘉阳 * @date 2020/7/21 15:06 */ class ChainofResponsibilityTest { @Test void doTest() { Approver articleApprover = new ArticleApprover(); Approver videoApprover = new VideoApprover(); Course course = new Course(); course.setName(\u0026#34;Java设计模式\u0026#34;); course.setArticle(\u0026#34;Java设计模式的手记\u0026#34;); course.setVideo(\u0026#34;Java设计模式的视频\u0026#34;); articleApprover.setNextApprover(videoApprover); articleApprover.deploy(course); } } 输出\nJava设计模式含有手记,批准 Java设计模式含有视频,批准 访问者模式 介绍 定义：封装作用于某数据结构(如List/set/Map等)中的各元素的操作\n特性：可以在不改变各元素的类的前提下定义作用于这些元素的操作\n类型：行为型\n适用场景\n一个数据结构如(List/set/Map等)包含很多类型对象 数据结构与数据操作分离 优点\n增加新的操作很容易即增加一个新的访问者 缺点\n增加新的数据结构困难 具体元素变更比较麻烦 相关设计模式\n访问者模式和迭代器模式 代码样例 类图\n课程抽象\npackage top.fjy8018.designpattern.pattern.behavior.visitor; import lombok.Data; /** * @author F嘉阳 * @date 2020/7/22 10:20 */ @Data public abstract class Course { private String name; public abstract void accept(IVisitor visitor); } 实战课\npackage top.fjy8018.designpattern.pattern.behavior.visitor; import lombok.Data; import lombok.EqualsAndHashCode; /** * 实战课，包含价格 * * @author F嘉阳 * @date 2020/7/22 10:20 */ @Data @EqualsAndHashCode(callSuper = true) public class CodingCourse extends Course { private int price; @Override public void accept(IVisitor visitor) { visitor.visit(this); } } 免费课\npackage top.fjy8018.designpattern.pattern.behavior.visitor; /** * 免费课 * * @author F嘉阳 * @date 2020/7/22 10:20 */ public class FreeCourse extends Course { /** * 接受策略 * * @param visitor */ @Override public void accept(IVisitor visitor) { visitor.visit(this); } } 访问者\npackage top.fjy8018.designpattern.pattern.behavior.visitor; /** * 封装操作数据结构的方法 * * @author F嘉阳 * @date 2020/7/22 10:20 */ public interface IVisitor { void visit(FreeCourse freeCourse); void visit(CodingCourse codingCourse); } 访问者核心实现\npackage top.fjy8018.designpattern.pattern.behavior.visitor; /** * 访问者 * 核心：对不同类型产生不同的操作行为 * \u0026lt;p\u0026gt; * jdk源码 {@link java.nio.file.FileVisitor} 实现对文件的访问 * spring源码 {@link org.springframework.beans.factory.config.BeanDefinitionVisitor} 遍历Bean具体属性并填充 * * @author F嘉阳 * @date 2020/7/22 10:20 */ public class Visitor implements IVisitor { /** * 访问免费课程，打印所有免费课程名称 * * @param freeCourse */ @Override public void visit(FreeCourse freeCourse) { System.out.println(\u0026#34;免费课程:\u0026#34; + freeCourse.getName()); } /** * 访问实战课程，打印所有实战课程名称及价格 * * @param codingCourse */ @Override public void visit(CodingCourse codingCourse) { System.out.println(\u0026#34;实战课程:\u0026#34; + codingCourse.getName() + \u0026#34; 价格:\u0026#34; + codingCourse.getPrice() + \u0026#34;元\u0026#34;); } } 测试\npackage top.fjy8018.designpattern.pattern.behavior.visitor; import org.junit.jupiter.api.Test; import java.util.ArrayList; import java.util.List; /** * @author F嘉阳 * @date 2020/7/22 16:19 */ class VisitorTest { @Test void visit() { List\u0026lt;Course\u0026gt; courseList = new ArrayList\u0026lt;\u0026gt;(); FreeCourse freeCourse = new FreeCourse(); freeCourse.setName(\u0026#34;SpringMVC数据绑定\u0026#34;); CodingCourse codingCourse = new CodingCourse(); codingCourse.setName(\u0026#34;Java设计模式精讲\u0026#34;); codingCourse.setPrice(299); // 数组组装 courseList.add(freeCourse); courseList.add(codingCourse); for (Course course : courseList) { course.accept(new Visitor()); } } } 输出\n免费课程:SpringMVC数据绑定 实战课程:Java设计模式精讲 价格:299元 状态模式 介绍 定义：允许一个对象在其内部状态改变时改变它的行为\n类型：行为型\n适用场景\n一个对象存在多个状态(不同状态下行为不同),且状态可相互转换 优点\n将不同的状态隔离 把各种状态的转换逻辑,分布到 State的子类中,减少相互间依赖 增加新的状态非常简单 缺点\n状态多的业务场景导致类数目增加,系统变复杂 相关设计模式\n状态模式和享元模式 代码样例 类图\n视频播放状态转换场景\npackage top.fjy8018.designpattern.pattern.behavior.state; import javax.faces.lifecycle.Lifecycle; /** * 视频播放状态转换场景 * \u0026lt;p\u0026gt; * {@link Lifecycle#execute(javax.faces.context.FacesContext)} 该方法通过外部状态控制改变行为 * {@link javax.faces.webapp.FacesServlet} 状态控制 * * @author F嘉阳 * @date 2020/7/22 15:20 */ public abstract class CourseVideoState { /** * 视频上下文 */ protected CourseVideoContext courseVideoContext; public void setCourseVideoContext(CourseVideoContext courseVideoContext) { this.courseVideoContext = courseVideoContext; } public abstract void play(); public abstract void speed(); public abstract void pause(); public abstract void stop(); } 视频上下文\npackage top.fjy8018.designpattern.pattern.behavior.state; /** * 视频上下文 * * @author F嘉阳 * @date 2020/7/22 15:20 */ public class CourseVideoContext { /** * 存储当前状态 */ private CourseVideoState courseVideoState; /** * 结合享元模式设置为内存共享 */ public final static PlayState PLAY_STATE = new PlayState(); public final static StopState STOP_STATE = new StopState(); public final static PauseState PAUSE_STATE = new PauseState(); public final static SpeedState SPEED_STATE = new SpeedState(); public CourseVideoState getCourseVideoState() { return courseVideoState; } public void setCourseVideoState(CourseVideoState courseVideoState) { this.courseVideoState = courseVideoState; // 把自己本身设置到实例上下文中 this.courseVideoState.setCourseVideoContext(this); } public void play(){ this.courseVideoState.play(); } public void speed(){ this.courseVideoState.speed(); } public void stop(){ this.courseVideoState.stop(); } public void pause(){ this.courseVideoState.pause(); } } 各状态实现\npackage top.fjy8018.designpattern.pattern.behavior.state; /** * @author F嘉阳 * @date 2020/7/22 15:20 */ public class PlayState extends CourseVideoState { @Override public void play() { System.out.println(\u0026#34;正常播放课程视频状态\u0026#34;); } @Override public void speed() { super.courseVideoContext.setCourseVideoState(CourseVideoContext.SPEED_STATE); } @Override public void pause() { super.courseVideoContext.setCourseVideoState(CourseVideoContext.PAUSE_STATE); } @Override public void stop() { super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE); } } package top.fjy8018.designpattern.pattern.behavior.state; /** * @author F嘉阳 * @date 2020/7/22 15:20 */ public class PauseState extends CourseVideoState { @Override public void play() { super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE); } @Override public void speed() { super.courseVideoContext.setCourseVideoState(CourseVideoContext.SPEED_STATE); } @Override public void pause() { System.out.println(\u0026#34;暂停播放课程视频状态\u0026#34;); } @Override public void stop() { super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE); } } package top.fjy8018.designpattern.pattern.behavior.state; /** * @author F嘉阳 * @date 2020/7/22 15:20 */ public class SpeedState extends CourseVideoState { @Override public void play() { super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE); } @Override public void speed() { System.out.println(\u0026#34;快进播放课程视频状态\u0026#34;); } @Override public void pause() { super.courseVideoContext.setCourseVideoState(CourseVideoContext.PAUSE_STATE); } @Override public void stop() { super.courseVideoContext.setCourseVideoState(CourseVideoContext.STOP_STATE); } } package top.fjy8018.designpattern.pattern.behavior.state; /** * @author F嘉阳 * @date 2020/7/22 15:20 */ public class StopState extends CourseVideoState { @Override public void play() { super.courseVideoContext.setCourseVideoState(CourseVideoContext.PLAY_STATE); } @Override public void speed() { System.out.println(\u0026#34;ERROR 停止状态不能快进!!\u0026#34;); } @Override public void pause() { System.out.println(\u0026#34;ERROR 停止状态不能暂停!!\u0026#34;); } @Override public void stop() { System.out.println(\u0026#34;停止播放课程视频状态\u0026#34;); } } 测试\npackage top.fjy8018.designpattern.pattern.behavior.state; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; /** * @author F嘉阳 * @date 2020/7/22 16:19 */ class PlayStateTest { @Test void play() { CourseVideoContext courseVideoContext = new CourseVideoContext(); courseVideoContext.setCourseVideoState(new PlayState()); System.out.println(\u0026#34;当前状态:\u0026#34;+courseVideoContext.getCourseVideoState().getClass().getSimpleName()); courseVideoContext.pause(); System.out.println(\u0026#34;当前状态:\u0026#34;+courseVideoContext.getCourseVideoState().getClass().getSimpleName()); courseVideoContext.speed(); System.out.println(\u0026#34;当前状态:\u0026#34;+courseVideoContext.getCourseVideoState().getClass().getSimpleName()); courseVideoContext.stop(); System.out.println(\u0026#34;当前状态:\u0026#34;+courseVideoContext.getCourseVideoState().getClass().getSimpleName()); courseVideoContext.speed(); } } 输出\n当前状态:PlayState 当前状态:PauseState 当前状态:SpeedState 当前状态:StopState ERROR 停止状态不能快进!! ","date":"2020-07-22T16:50:17+08:00","permalink":"/archives/246/","title":"设计模式学习——行为型"},{"content":"背景 吸取前两代系统的失败经验，对第二代密码管理微服务进行重新规划，并且在探索过程中见识到了cloud-platform、SpringBlade、pigx等优秀的微服务项目，考虑基于开源项目二次开发\n规划 为了尽快落地上线，第一期以实现最小功能为主，其他非必须功能尽可能往后放\n第一期 SpringCloud微服务集成 Nacos、Sentinel、graylog、zipkin、 Turbine集成 前后端密码管理功能对接 操作日志记录 邮件通知 Excel导入导出 v1迁移到v2工具包 第二期 dble读写分离 支持忘记密码、密码锁定等 集成微信推送、微信注册功能 支持上传前端日志 页面排序、检索优化 第三期 增加用户反馈功能 接入Seata分布式事务 支持配置导入导出权限 支持录入标签 … 技术体系 前端 前后端分离-采用前后端分离模式，前端基于Vue 采用Vuex进行状态管理，采用ES6作为语法支持 前后端基于开源项目SpringBlade二次开发 前后端代码生成-定制针对两套前端与后端的代码生成模板，轻松生成整个模块的前后端代码，减少重复工作量 多租户顶部菜单配置-提供给每个租户独立的顶部菜单配置模块，可以自定义顶部菜单切换 后端 基于SpringCloud的分布式架构 采用Mybatis与MybatisPlus作为ORM持久层框架 多租户系统-完整的SaaS多租户架构 借鉴OAuth2，通过Jwt令牌实现了多终端认证系统，可控制子系统的token权限互相隔离 权限管理-精心设计的权限管理方案，角色权限精确到按钮 集成邮件服务，通过事件传播实现不依赖消息队列通知发送服务 集成微信推送服务，实现微信注册、敏感操作消息推送 动态聚合文档-实现基于Nacos的Swagger SpringCloud聚合文档 实现动态网关，不重启就能设置网关配置项 集成Turbine聚合熔断监控 数据架构 采用MySQL进行数据存储，底层配置为一主两从的高可用集群 采用ElasticSearch三节点集群进行日志数据存储 分布式支持 Nacos-集成阿里巴巴的Nacos完成统一的服务注册与配置 Sentinel-集成Sentinel从流量控制、熔断降级、系统负载等多个维度保护服务的稳定性 采用Docker打包，部署于Kubernetes集群上 Seata分布式事务-定制集成Seata，支持分布式事务，无代码侵入，不失灵活与简洁 使用graylog进行分布式日志采集，集中监控 通过zipkin进行分布式链路追踪，耗时统计和服务调用情况 DevOps 配置管理采用单仓库模式，集成docker打包插件，自动推送到harbor仓库 采用GitLab Runner CICD流水线自动部署到测试环境和生产环境 结合Maven私服和Harbor私服为内网自动化部署提供支持 官方架构图 截图 管理员 操作日志记录\n错误日志系统内采集\nTurbine集成熔断监控\n通知发布\n普通用户 主题切换\n微信扫码注册\n补全信息\n主页资源权限控制\n菜单资源权限控制\nExcel导入导出\n密码管理\n运维 Nacos注册中心\nSentinel限流管控\nSpringBoot Admin应用内指标监控\nHarbor内网私服\nNexus内网私服，提供maven和npm包\ngraylog日志采集\n三节点ES集群支持日志存储\n耗时统计\n新UI风格\n动态服务依赖关系图\n新UI依赖图\nPrometheus指标采集\nGrafana资源指标监控\nKuberneties服务部署监控\n开发过程工具 Gitlab CICD流水线\n看板\n功能开发周期分析\ngitlab集成k8s部署环境\nSonarqube代码质量检查\n分支图\n工程模块图\n结果——成功 经过两个月的开发和调试，第一期工程符合预定计划，按期投产上线运行，并经过2周试运行和测试后，进入生产环境，迁移老版本数据，正式替换了运行一年半的第一代密码管理系统\n时间线 2020年3月开始基于SpringBlade 2.6.2编码 2020年4月20日第一个测试版本进入k8s测试环境 2020年4月28日第一个稳定版进行试运行阶段 2020年5月4日试运行阶段结束，第一个稳定版1.1.2发布，正式进行数据迁移并投入使用 ","date":"2020-05-07T21:38:00+08:00","permalink":"/archives/245/","title":"微服务平台开发总结（3）——第二代密码管理平台"},{"content":"背景 随着网站、工作使用需要，对密码存储的需求越来越大，初期采用本地数据库进行存储，但不适合多地访问，一旦离开了本地电脑，就无法查询密码信息，故计划搭建个人使用的密码信息管理系统\n概要 时间线 密码管理信息系统于2018年12月立项，2019年1月9日开始第一次提交 2019年2月7日第一个版本上线试运行 2019年4月7日进入维护阶段，不再支持新功能 2019年11月16日停止维护 架构 后端 JDK8 SpringBoot 2.1.1.RELEASE Jetty SpringData Jpa MySQL 5.7 Docker Kuberneties 1.14 Redis Graylog GitLab CICD 前端 Vue 2.5 element 2.4.11 vuex 3 vue-axios 2.1.4 eslint 5.8.0 nodejs nginx https Docker GitLab CICD 原理 后端存储密码数据字段和网站相关信息 采用SQL全字段模糊查询匹配结果 密码字段存储采用AES128结合SHA随机盐值加密存储，即使数据库被破解也无法查看对应的密码信息 密码加密秘钥不由服务端存储，只在用户登陆时录入并临时通过AES128加密后存储在Redis中，用户登出或者登陆超时后销毁 优势 单体项目，部署简单 容器化部署，迁移便利 Kuberneties云平台部署，支持灰度发布 无状态后端服务，支持集群化部署 安全审计 支持简单的安全审计\n登陆日志记录 登陆归属地记录 安全问题 通过实现的Spring拦截器进行登陆拦截，安全性不高 没有采用session会话方式，故多地登陆账号且加密秘钥未过期则不需要再次录入 支持所有跨域行为，没有对客户端进行识别 其他问题 独立用户控制，扩展性不高 没有基于RBAC模型的权限控制 没有管理后台 单模块、单体项目，不利于扩展 界面演示 登陆页\n若用户不存在则自动新增\n录入加密秘钥\n主页\n信息录入\n支持全字段模糊检索\n支持Excel模板信息导入\n登出\n效果 系统稳定运行一年，符合系统初期设计目的 同时兼容移动端渲染 ","date":"2020-05-07T21:34:00+08:00","permalink":"/archives/243/","title":"微服务平台开发总结（1）——第一代密码管理系统"},{"content":"背景 现有网上很多教程都是修改了源MySQL镜像，因此会存在安全隐患，也不容易提升版本。此处给出不修改源镜像的主从复制配置方案。\n现状 现有一台生产上运行的MySQL5.7服务器，由docker镜像提供支持\n原docker-compose文件\nversion: \u0026#39;3.7\u0026#39; services: mysql57: image: mysql:5.7 restart: always command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci ports: - 3306:3306 networks: - prod volumes: - ./mysql57/data:/var/lib/mysql environment: MYSQL_ROOT_PASSWORD: root networks: prod: 改造过程 配置主服务器 新增同步用户并授权 ##创建slave1用户 CREATE USER \u0026#39;slave1\u0026#39;@\u0026#39;%\u0026#39; IDENTIFIED BY \u0026#39;slavepass\u0026#39;; ##为slave1赋予REPLICATION SLAVE权限。 GRANT REPLICATION SLAVE ON *.* TO \u0026#39;slave1\u0026#39;@\u0026#39;%\u0026#39;; 开启binlog日志 由于原主服务器没有开启binlog日志，故需要挂载配置文件并开启binlog日志\n新增配置文件custom-master.cnf\n[mysqld] log-bin = mysql-bin log-bin-index = mysql-bin.index binlog_format = mixed server-id = 1 sync-binlog = 1 character-set-server = utf8 修改docker-compose文件为\nversion: \u0026#39;3.7\u0026#39; services: mysql57: image: mysql:5.7.29 restart: always command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci ports: - 3306:3306 networks: - prod volumes: - ./mysql57/data:/var/lib/mysql - ./mysql57/custom-master.cnf:/etc/mysql/conf.d/custom.cnf - ./mysql57/log:/var/log/mysql environment: MYSQL_ROOT_PASSWORD: root networks: prod: 更新服务（会短暂停机）\n$ docker-compose up -d 配置内容解释\nlog-bin：设置二进制日志文件的基本名； log-bin-index：设置二进制日志索引文件名； binlog_format：控制二进制日志格式，进而控制了复制类型，三个可选值 -STATEMENT：语句复制 -ROW：行复制 -MIXED：混和复制，默认选项 server-id：服务器设置唯一ID，默认为1，推荐取IP最后部分； sync-binlog：默认为0，为保证不会丢失数据，需设置为1，用于强制每次提交事务时，同步二进制日志到磁盘上。 备份主数据库数据 该步骤可以等到两台从服务器的docker环境都配置完毕后操作，以减少停机时间\n获取一个一致性的快照，需对所有表设置读锁： flush tables with read lock; 获取二进制日志的坐标： show master status; File Position Binlog_Do_DB Binlog_Ignore_DB Executed_Gtid_Set mysql-bin.000003 6165 备份数据 # 针对事务性引擎 mysqldump -u root -p --all-databases -e --single-transaction --flush-logs --max_allowed_packet=1048576 --net_buffer_length=16384 \u0026gt; /var/log/mysql/all_db.sql # 针对 MyISAM 引擎，或多引擎混合的数据库 mysqldump -uroot --all-databases -e -l --flush-logs --max_allowed_packet=1048576 --net_buffer_length=16384 \u0026gt; /var/log/mysql/all_db.sql 由于挂载了日志目录，故在容器外部可以看到该备份的数据文件\n配置从服务器 新增两台CentOS 7服务器，并安装docker和docker-compose服务 由于主从复制要求所有机器版本号包括小版本号都必须一致，故此处拉取mysql:5.7.29镜像 从主库拷贝数据文件all_db.sql到两台从库上 $ scp /root/mysql/log/all_db.sql root@192.168.1.33:/root/mysql/log/ 新增配置文件custom-slave1.cnf [mysqld] log-bin = mysql-bin binlog_format = mixed log-slave-updates = 0 server-id = 2 relay-log = mysql-relay-bin relay-log-index = mysql-relay-bin.index read-only = 1 slave_net_timeout = 10 新增docker-compose文件 version: \u0026#39;3\u0026#39; services: mysql57-master: image: mysql:5.7.29 restart: always command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci ports: - 3306:3306 networks: - mysql-net volumes: - ./data:/var/lib/mysql - ./custom-slave1.cnf:/etc/mysql/conf.d/custom.cnf - ./log:/var/log/mysql environment: MYSQL_ROOT_PASSWORD: root networks: mysql-net: 由于也挂载了log目录，启动后容器内就已经有从主库同步过来的数据文件 数据恢复 mysql -u root -p \u0026lt; /var/log/mysql/all_db.sql 使用工具连接数据库，配置主从连接 CHANGE MASTER TO MASTER_HOST=\u0026#39;10.168.1.30\u0026#39;, MASTER_USER=\u0026#39;slave1\u0026#39;, MASTER_PASSWORD=\u0026#39;slavepass\u0026#39;, MASTER_LOG_FILE=\u0026#39;mysql-bin.000003\u0026#39;, MASTER_LOG_POS=6165; 启动slave同步进程 start slave; 关闭主服务器全局读锁 UNLOCK TABLES; 查看从服务器同步状态，主要是Slave_IO_Running， Slave_SQL_Running状态必须为Yes Slave_IO_State Master_Host Master_User Master_Port Connect_Retry Master_Log_File Read_Master_Log_Pos Relay_Log_File Relay_Log_Pos Relay_Master_Log_File Slave_IO_Running Slave_SQL_Running Replicate_Do_DB Replicate_Ignore_DB Replicate_Do_Table Replicate_Ignore_Table Replicate_Wild_Do_Table Replicate_Wild_Ignore_Table Last_Errno Last_Error Skip_Counter Exec_Master_Log_Pos Relay_Log_Space Until_Condition Until_Log_File Until_Log_Pos Master_SSL_Allowed Master_SSL_CA_File Master_SSL_CA_Path Master_SSL_Cert Master_SSL_Cipher Master_SSL_Key Seconds_Behind_Master Master_SSL_Verify_Server_Cert Last_IO_Errno Last_IO_Error Last_SQL_Errno Last_SQL_Error Replicate_Ignore_Server_Ids Master_Server_Id Master_UUID Master_Info_File SQL_Delay SQL_Remaining_Delay Slave_SQL_Running_State Master_Retry_Count Master_Bind Last_IO_Error_Timestamp Last_SQL_Error_Timestamp Master_SSL_Crl Master_SSL_Crlpath Retrieved_Gtid_Set Executed_Gtid_Set Auto_Position Replicate_Rewrite_DB Channel_Name Master_TLS_Version Waiting for master to send event 192.168.1.30 slave1 3306 60 mysql-bin.000003 3722724 mysql-relay-bin.000002 3717604 mysql-bin.000003 Yes Yes 0 0 3722724 3717811 None 0 No 0 No 0 0 1 4c99130f-e664-11e9-987b-0242ac120003 /var/lib/mysql/master.info 0 Slave has read all relay log; waiting for more updates 86400 0 可能出现的问题 死锁——waiting for global read lock 原因在于flush tables with read lock语句和事务执行的过程都涉及到连个锁，而且不是原子的，再加上并行复制以及设置了从库binlog的顺序提交，最终导致多个线程形成死锁\n解决方法：\n重启主数据库 查看所有连接线程，分析死锁的原因，杀掉执行flush tables with read lock的线程 参考资料：\nMySQL备份导致的waiting for global read lock 由FTWRL导致的MySQL从库死锁分析及参数深究 ","date":"2020-04-17T15:37:02+08:00","permalink":"/archives/242/","title":"Docker MySQL主从复制方案"},{"content":"分离头指针 当前分支图\n切换到之前的某一次提交\n执行命令\n$ git checkout de11fa87ea 提示，当前位于“分离头指针”状态\n分支图\n注意\n在分离头指针状态下没有绑定分支 产生的commit不会被保存 分支切换后在分支图中看不到该commit 一段时间后会被git自动清除 使用场景\n尝试性做一些变更，可以随时丢弃 此处演示分离头指针丢失commit的情况和补救措施\n丢失commit 使用IDE修改文件并提交，忽略所有警告，修改后分支图如下\n此时接到其他需求，需要切换分支进行紧急修复\n执行命令\n$ git checkout master 分支切换成功，并弹出提示和告警\nWarning: you are leaving 1 commit behind, not connected to any of your branches:\n199ac20 游离状态修改文件\nIf you want to keep it by creating a new branch, this may be a good time to do so with:\ngit branch 199ac20\nSwitched to branch \u0026lsquo;master\u0026rsquo; Your branch is up to date with \u0026lsquo;origin/master\u0026rsquo;.\n此时查看分支图，刚才的commit已经不可见\n补救 此时发现刚才的commit十分重要，可根据git的提示进行补救\n执行命令\n$ git branch hot-fix 199ac20 再次查看分支图，可见commit已经恢复\n本地分支操作 以下操作仅适用于本地分支，无远程分支协同工作的情况\n修改当前commit的message——amend 当前分支图\n执行命令\n$ git commit --amend 自动弹出编辑器\n修改后保存并关闭编辑器即可，输出如下\n再次查看分支图\n修改前面某次commit的message——rebase 现计划修改如下message\n使用IDE拷贝其父提交的SHA值\n执行命令，-i表示交互式执行\n$ git rebase -i 199ac203c90f881024c6870d56517df9e2080841 自动弹出编辑器\n同时包含提示操作\nRebase 199ac20..bfe2b7f onto 199ac20 (3 commands)\nCommands: p, pick = use commit r, reword = use commit, but edit the commit message e, edit = use commit, but stop for amending s, squash = use commit, but meld into previous commit f, fixup = like \u0026ldquo;squash\u0026rdquo;, but discard this commit\u0026rsquo;s log message x, exec = run command (the rest of the line) using shell b, break = stop here (continue rebase later with \u0026lsquo;git rebase \u0026ndash;continue\u0026rsquo;) d, drop = remove commit l, label = label current HEAD with a name t, reset = reset HEAD to a label m, merge [-C | -c ] [# ] . create a merge commit using the original merge commit\u0026rsquo;s . message (or the oneline, if no original merge commit was . specified). Use -c to reword the commit message.\nThese lines can be re-ordered; they are executed from top to bottom.\nIf you remove a line here THAT COMMIT WILL BE LOST.\nHowever, if you remove everything, the rebase will be aborted.\nNote that empty commits are commented out\n由此可见“变基”支持很多操作，此处需要使用的是reword，修改目标提交的命令为r\n保存并关闭编辑器，会自动弹出新编辑器界面\n修改后保存并关闭即可，输出如下\n查看分支图\n整理连续多个commit——rebase 现计划合并3个commit\n使用IDE拷贝其父SHA值\n执行命令\n$ git rebase -i 199ac203c90f881024c6870d56517df9e2080841 此处使用squash，官方介绍如下\ns, squash = use commit, but meld into previous commit\n修改内容如下，说明将中间两个合并进第一个commit中\n保存后关闭，会弹出新的编辑器界面，提示可以输出合并commit的message\n编写message\n保存后关闭，提示修改成功\n查看分支图\n分支合并策略 演示不同分支合并策略效果，当前的分支图如下\n现在将hot-fix合并进master\nGitLab默认合并策略——merge commit gitlab默认合并请求使用的策略\n由于没有冲突，可以线上直接合并\n其对应执行的命令如下\nStep 1. Fetch and check out the branch for this merge request\n$ git fetch origin $ git checkout -b hot-fix origin/hot-fix Step 2. Review the changes locally\nStep 3. Merge the branch and fix any conflicts that come up\n$ git fetch origin $ git checkout origin/master $ git merge --no-ff hot-fix Step 4. Push the result of the merge to GitLab\n$ git push origin master 合并完成\n查看分支图，故当前分支会影响特性分支并产生新的提交\nGitLab合并提交策略 提交合并请求的时候勾选策略Squash commits when merge request is accepted\n其对应执行的命令如下（与默认策略一致）\nStep 1. Fetch and check out the branch for this merge request\n$ git fetch origin $ git checkout -b hot-fix origin/hot-fix Step 2. Review the changes locally\nStep 3. Merge the branch and fix any conflicts that come up\n$ git fetch origin $ git checkout origin/master $ git merge --no-ff hot-fix Step 4. Push the result of the merge to GitLab\n$ git push origin master 合并成功\n查看分支图，可见其影响特性分支，会产生新的提交，同时覆盖特性分支最后一次提交\n分支合并策略——squash merge GitHub支持squash merge合并策略，此处通过命令行执行测试\n$ git merge --squash origin/hot-fix $ git commit -m \u0026#34;squash commit\u0026#34; 查看分支图，特性分支不变，目标分支合并提交后产生一条提交，效果与GitLab合并提交类似，但命令使用方式不同\n注意：squash merge会变更提交者作者信息，这是一个很大的问题，后期问题追溯不好处理\n变基合并策略——rebase merge 该策略也是GitHub支持的合并策略，可能会产生较多冲突\n选择合并策略，最后一种即变基合并策略\n合并成功\n查看分支图，其产生的三次提交即变基合并的结果，不会影响特性分支，也不会变更提交人，类似cherry pick\n分支回滚策略 回滚策略分为两种，一种是合并后回滚，一种是普通提交回滚，不同回滚策略操作方法不同\n普通提交回滚 当前分支图，计划回滚最新提交\n直接在IDE中执行revert操作即可\n回滚会产生新的提交，回滚结束\n合并后回滚 此时分支图，使用merge commit方式进行过一次合并，此时不想要合并的结果，想恢复到未合并状态\n错误操作——直接执行revert 此时IDE已经不支持通过revert进行回滚合并操作，通过命令行强行执行\n查找SHA值\n回滚merge需要加上-m参数git就知道到底要revert哪一个merge\n$ git revert -n 25d7b405a4d50fe1b36019d90276973a8ed9160d -m 1 回滚后做一次commit即可，使用默认的message后如下\n验证回滚结果，使用两个提交进行差异比对\n无差异，回滚看似成功了\n此时在原分支做了修复后再进行合并\n报无法合并，因为分支已经进行过合并\n正确操作——reset 使用reset操作到上一次提交\n使用--hard参数，丢弃后续的commit\n此时继续进行操作，并支持重新合并\n禁止类操作 说明：禁止类说明是对开发人员禁止，而该命令在必要时候依然需要由配置管理员使用\npush -f 官方使用说明\n-f, \u0026ndash;force force updates\n原因：\n默认情况下远端分支若能顺利合并（fast forward），则会顺利合并 若远端分支产生冲突，则会停止push，同时提示错误，要pull到本地处理冲突后重新push 加-f之后，本地分支会强行推送到远端，若本地分支落后于远端分支也不会弹出提示，由此导致远程commit丢失 适用场景：\n远程分支已经混乱，本地分支符合需要，需要强制整理远程分支 公共分支禁止rebase 此处对场景进行模拟，正常情况下，多位开发者对公共分支common-branch进行修改，分支图如下\n另一开发者在本地修改后执行变基操作，此处以修改历史提交message为例\n变基\n录入变基信息\n变基成功\n此时查看本地分支图，可见产生了新的分支路径，同时远程message并没有变化，故rebase对存在远程分支的情况无效\n变基后推送到远端分支，提示需要合并，选择“合并”\n合并后分支图如下\n此时其他开发者修改文件后并fetch远程分支查看变更情况如下\n此时无法对远程分支进行push操作\n处理方法 pull远程分支在本地合并后重新推送，此时在本地再产生一次合并\n此时可以成功推送\n由此一来一回操作导致大量不必要的合并操作，并可能导致大量冲突，并且没有解决问题\n警告类操作 reset --hard 官方对reset操作用法说明\nusage: git reset [\u0026ndash;mixed | \u0026ndash;soft | \u0026ndash;hard | \u0026ndash;merge | \u0026ndash;keep] [-q] [] or: git reset [-q] [] [\u0026ndash;] \u0026hellip; or: EXPERIMENTAL: git reset [-q] [\u0026ndash;stdin [-z]] [] or: git reset \u0026ndash;patch [] [\u0026ndash;] [\u0026hellip;]\n-q, --quiet be quiet, only report errors --mixed reset HEAD and index --soft reset only HEAD --hard reset HEAD, index and working tree --merge reset HEAD, index and working tree --keep reset HEAD but keep local changes --recurse-submodules[=\u0026lt;reset\u0026gt;] control recursive updating of submodules -p, --patch select hunks interactively -N, --intent-to-add record only the fact that removed paths will be added later -z EXPERIMENTAL: paths are separated with NUL character --stdin EXPERIMENTAL: read paths from \u0026lt;stdin\u0026gt; 原因：\n在分支上执行该操作会丢失目标commit以后的所有commit 适用场景\n用于分支回滚且不打算保留其后的所有变更 ","date":"2020-03-13T14:08:00+08:00","image":"/images/recovered/7029ed877105a9b0efec.png","permalink":"/archives/240/","title":"Git工作使用笔记"},{"content":"外观模式 介绍 又叫门面模式,提供了一个统一的接口,用来访问子系统中的一群接口\n外观模式定义了一个高层接口,让子系统更容易使用\n类型：结构型\n适用场景：\n子系统越来越复杂,增加外观模式提供简单调用接口 构建多层系统结构,利用外观对象作为每层的入口,简化层间调用 优点：\n简化了调用过程，无需了解深入子系统，防止带来风险。 减少系统依赖、松散耦合 更好的划分访问层次 符合迪米特法则，即最少知道原则 缺点：\n增加子系统、扩展子系统行为容易引入风险 不符合开闭原则 相关的设计模式\n外观模式和中介者模式 外观模式和单例模式 外观模式和抽象工厂模式 代码样例 改造前 类图\npackage top.fjy8018.designpattern.pattern.structural.facade.before; import lombok.AllArgsConstructor; import lombok.Getter; /** * 外观模式又叫门面模式,提供了一个统一的接口,用来访问子系统中的一群接口 * 积分礼物 * * @author F嘉阳 * @date 2020/2/28 10:33 */ @Getter @AllArgsConstructor public class PointGift { private String name; } 子系统一：积分校验子系统\npackage top.fjy8018.designpattern.pattern.structural.facade.before; import lombok.extern.slf4j.Slf4j; /** * 积分校验子系统 * * @author F嘉阳 * @date 2020/2/28 10:34 */ @Slf4j public class QualifyService { /** * 模拟积分校验逻辑 * @param pointGift * @return */ public boolean isAvalible(PointGift pointGift){ log.info(\u0026#34;校验礼物 {} 通过，积分校验通过，库存通过\u0026#34;,pointGift.getName()); return true; } } 子系统二：积分支付校验子系统\npackage top.fjy8018.designpattern.pattern.structural.facade.before; import lombok.extern.slf4j.Slf4j; /** * 积分支付校验子系统 * * @author F嘉阳 * @date 2020/2/28 10:37 */ @Slf4j public class PointPaymentService { public boolean pay(PointGift pointGift){ // 扣减积分逻辑 log.info(\u0026#34;支付礼物 {} 积分成功\u0026#34;,pointGift.getName()); return true; } } 子系统三：物流子系统\npackage top.fjy8018.designpattern.pattern.structural.facade.before; import lombok.extern.slf4j.Slf4j; /** * 物流子系统 * * @author F嘉阳 * @date 2020/2/28 10:39 */ @Slf4j public class ShippingService { /** * 返回物流运单号 * * @param pointGift * @return */ public String shipGift(PointGift pointGift){ // 物流系统对接逻辑 log.info(\u0026#34;礼物 {} 进入物流系统\u0026#34;,pointGift.getName()); return \u0026#34;123456\u0026#34;; } } 外观类——礼物兑换逻辑\npackage top.fjy8018.designpattern.pattern.structural.facade.before; import lombok.extern.slf4j.Slf4j; /** * 外观类——礼物兑换逻辑 * * 该服务定义了与各个子系统交互的逻辑，业务系统无需知道与内部子系统的交互逻辑，符合迪米特法则 * @author F嘉阳 * @date 2020/2/28 10:44 */ @Slf4j public class GiftExchangeService { private PointPaymentService pointPaymentService; private QualifyService qualifyService; private ShippingService shippingService; public void setPointPaymentService(PointPaymentService pointPaymentService) { this.pointPaymentService = pointPaymentService; } public void setQualifyService(QualifyService qualifyService) { this.qualifyService = qualifyService; } public void setShippingService(ShippingService shippingService) { this.shippingService = shippingService; } /** * 系统交互逻辑 * * @param pointGift */ public void giftExchange(PointGift pointGift){ if (qualifyService.isAvalible(pointGift)){ // 资格校验通过 if (pointPaymentService.pay(pointGift)){ // 如果积分支付成功 String shippingOrderNo = shippingService.shipGift(pointGift); log.info(\u0026#34;物流系统下单成功，订单号为{}\u0026#34;,shippingOrderNo); } } } } 测试\npackage top.fjy8018.designpattern.pattern.structural.facade.before; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; /** * @author F嘉阳 * @date 2020/2/28 10:53 */ @Slf4j class GiftExchangeServiceTest { @Test void giftExchange() { PointGift pointGift = new PointGift(\u0026#34;手环\u0026#34;); // 注入服务 GiftExchangeService exchangeService = new GiftExchangeService(); exchangeService.setPointPaymentService(new PointPaymentService()); exchangeService.setQualifyService(new QualifyService()); exchangeService.setShippingService(new ShippingService()); // 兑换 exchangeService.giftExchange(pointGift); } } 执行结果\n10:56:55.639 [main] INFO top.fjy8018.designpattern.pattern.structural.facade.before.QualifyService - 校验礼物 手环 通过，积分校验通过，库存通过 10:56:55.652 [main] INFO top.fjy8018.designpattern.pattern.structural.facade.before.PointPaymentService - 支付礼物 手环 积分成功 10:56:55.652 [main] INFO top.fjy8018.designpattern.pattern.structural.facade.before.ShippingService - 礼物 手环 进入物流系统 10:56:55.652 [main] INFO top.fjy8018.designpattern.pattern.structural.facade.before.GiftExchangeService - 物流系统下单成功，订单号为123456 改造后 改造点很简单，只要将应用层注入的类在逻辑层注入即可\npackage top.fjy8018.designpattern.pattern.structural.facade; import lombok.extern.slf4j.Slf4j; /** * 外观类——礼物兑换逻辑 * \u0026lt;p\u0026gt; * 该服务定义了与各个子系统交互的逻辑，业务系统无需知道与内部子系统的交互逻辑，符合迪米特法则 * * @author F嘉阳 * @date 2020/2/28 10:44 */ @Slf4j public class GiftExchangeService { private PointPaymentService pointPaymentService = new PointPaymentService(); private QualifyService qualifyService = new QualifyService(); private ShippingService shippingService = new ShippingService(); /** * 系统交互逻辑 * * @param pointGift */ public void giftExchange(PointGift pointGift) { if (qualifyService.isAvalible(pointGift)) { // 资格校验通过 if (pointPaymentService.pay(pointGift)) { // 如果积分支付成功 String shippingOrderNo = shippingService.shipGift(pointGift); log.info(\u0026#34;物流系统下单成功，订单号为{}\u0026#34;, shippingOrderNo); } } } } 业务层\npackage top.fjy8018.designpattern.pattern.structural.facade; import lombok.extern.slf4j.Slf4j; import org.apache.catalina.connector.Request; import org.apache.catalina.connector.RequestFacade; /** * 外观类——礼物兑换逻辑 * \u0026lt;p\u0026gt; * 该服务定义了与各个子系统交互的逻辑，业务系统无需知道与内部子系统的交互逻辑，符合迪米特法则 * * 源码： * {@link org.springframework.jdbc.support.JdbcUtils} 对数据库相关的操作进行封装 * {@link org.apache.ibatis.session.Configuration#newMetaObject(Object)} 对创建对象的方法进行封装 * {@link RequestFacade} 其定义了操作HTTPRequest的公用方法 {@link Request} 在Request中都用该外观类操作 * 其他类似逻辑： * {@link org.apache.catalina.connector.ResponseFacade} * {@link org.apache.catalina.session.StandardSessionFacade} * * * @author F嘉阳 * @date 2020/2/28 10:44 */ @Slf4j public class GiftExchangeService { private PointPaymentService pointPaymentService = new PointPaymentService(); private QualifyService qualifyService = new QualifyService(); private ShippingService shippingService = new ShippingService(); /** * 系统交互逻辑 * * @param pointGift */ public void giftExchange(PointGift pointGift) { if (qualifyService.isAvalible(pointGift)) { // 资格校验通过 if (pointPaymentService.pay(pointGift)) { // 如果积分支付成功 String shippingOrderNo = shippingService.shipGift(pointGift); log.info(\u0026#34;物流系统下单成功，订单号为{}\u0026#34;, shippingOrderNo); } } } } 类图\n此时业务层无需关心子系统，只和外观类通信\n装饰器模式 介绍 定义：在不改变原有对象的基础之上，将功能附加到对象上\n提供了比继承更有弹性的替代方案（扩展原有对象功能）\n类型：结构型\n适用场景\n扩展一个类的功能或给一个类添加附加职责 动态的给一个对象添加功能，这些功能可以再动态的撤销 优点\n继承的有力补充,比继承灵活,不改变原有对象的情况下给一个对象 扩展功能 通过使用不同装饰类以及这些装饰类的排列组合,可以实现不同效果 符合开闭原则 缺点\n会出现更多的代码,更多的类,增加程序复杂性 动态装饰时,多层装饰时会更复杂 相关设计模式\n装饰者模式和代理模式 装饰者模式和适配器模式 代码样例 改造前 类图\nBattercake\npackage top.fjy8018.designpattern.pattern.structural.decorator.before; /** * @author F嘉阳 * @date 2018-09-27 16:56 */ public class Battercake { protected String getDesc() { return \u0026#34;煎饼\u0026#34;; } protected int cost() { return 8; } } BattercakeWithEgg\npackage top.fjy8018.designpattern.pattern.structural.decorator.before; /** * 不使用装饰器则只能通过类继承扩展功能，可能引起类爆炸 * * @author F嘉阳 * @date 2018-09-27 16:59 */ public class BattercakeWithEgg extends Battercake { @Override protected String getDesc() { return super.getDesc() + \u0026#34; 加一个鸡蛋\u0026#34;; } @Override protected int cost() { return super.cost() + 1; } } BattercakeWithEggSausage\npackage top.fjy8018.designpattern.pattern.structural.decorator.before; /** * @author F嘉阳 * @date 2018-09-27 17:00 */ public class BattercakeWithEggSausage extends BattercakeWithEgg { @Override protected String getDesc() { return super.getDesc() + \u0026#34; 加一根香肠\u0026#34;; } @Override protected int cost() { return super.cost() + 2; } } 测试\npackage top.fjy8018.designpattern.pattern.structural.decorator; @Slf4j class BattercakeTest { /** * 通过继承扩展灵活度低，无法自由组合 */ @Test void before() { Battercake battercake = new Battercake(); log.info(battercake.getDesc() + \u0026#34; 销售价格:\u0026#34; + battercake.cost()); Battercake battercakeWithEgg = new BattercakeWithEgg(); log.info(battercakeWithEgg.getDesc() + \u0026#34; 销售价格:\u0026#34; + battercakeWithEgg.cost()); Battercake battercakeWithEggSausage = new BattercakeWithEggSausage(); log.info(battercakeWithEggSausage.getDesc() + \u0026#34; 销售价格:\u0026#34; + battercakeWithEggSausage.cost()); } } 输出\n14:42:18.640 [main] INFO top.fjy8018.designpattern.pattern.structural.decorator.BattercakeTest - 煎饼 销售价格:8 14:42:18.652 [main] INFO top.fjy8018.designpattern.pattern.structural.decorator.BattercakeTest - 煎饼 加一个鸡蛋 销售价格:9 14:42:18.652 [main] INFO top.fjy8018.designpattern.pattern.structural.decorator.BattercakeTest - 煎饼 加一个鸡蛋 加一根香肠 销售价格:11 改造后 类图\n所有类的抽象\npackage top.fjy8018.designpattern.pattern.structural.decorator.after; /** * 装饰器模式按照业务场景一般有两层抽象， * 一层是所有类的抽象，一层是所有装饰器的抽象， * 所有装饰器的抽象可以为实现类提供实现额外功能的能力 * * @author F嘉阳 * @date 2018-09-27 17:02 */ public abstract class AbstractBattercake { public abstract String getDesc(); public abstract int cost(); } 所有装饰器的抽象\npackage top.fjy8018.designpattern.pattern.structural.decorator.after; /** * @author F嘉阳 * @date 2018-09-27 17:05 */ public abstract class AbstractBattercakeDecorator extends AbstractBattercake { private AbstractBattercake abstractBattercake; /** * 通过构造器注入待装饰对象 * * @param abstractBattercake */ public AbstractBattercakeDecorator(AbstractBattercake abstractBattercake) { this.abstractBattercake = abstractBattercake; } /** * 实现类可以提供额外能力 */ protected abstract void doSomething(); @Override public String getDesc() { return this.abstractBattercake.getDesc(); } @Override public int cost() { return this.abstractBattercake.cost(); } } package top.fjy8018.designpattern.pattern.structural.decorator.after; /** * Servlet源码：{@link javax.servlet.http.HttpServletRequestWrapper} * JDK源码：IO类：{@link java.io.BufferedReader} {@link java.io.BufferedInputStream} * Spring源码:{@link org.springframework.cache.transaction.TransactionAwareCacheDecorator} * mybatis源码：{@link org.apache.ibatis.cache.decorators.LruCache} * * @author F嘉阳 * @date 2018-09-27 17:04 */ public class ABattercake extends AbstractBattercake { @Override public String getDesc() { return \u0026#34;煎饼\u0026#34;; } @Override public int cost() { return 8; } } package top.fjy8018.designpattern.pattern.structural.decorator.after; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2018-09-27 17:08 */ @Slf4j public class EggBattercake extends AbstractBattercakeDecorator { public EggBattercake(AbstractBattercake abstractBattercake) { super(abstractBattercake); } @Override protected void doSomething() { log.info(\u0026#34;EggBattercake doSomething...\u0026#34;); } @Override public String getDesc() { return super.getDesc() + \u0026#34; 加一个鸡蛋\u0026#34;; } @Override public int cost() { return super.cost() + 1; } } package top.fjy8018.designpattern.pattern.structural.decorator.after; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2018-09-27 17:10 */ @Slf4j public class SausageBattercake extends AbstractBattercakeDecorator { public SausageBattercake(AbstractBattercake abstractBattercake) { super(abstractBattercake); } @Override protected void doSomething() { log.info(\u0026#34;SausageBattercake doSomething...\u0026#34;); } @Override public String getDesc() { return super.getDesc() + \u0026#34; 加一根香肠\u0026#34;; } @Override public int cost() { return super.cost() + 2; } } 测试\npackage top.fjy8018.designpattern.pattern.structural.decorator; @Slf4j class BattercakeTest { @Test void after() { // 自由组合装饰 AbstractBattercake battercake = new EggBattercake(new SausageBattercake(new ABattercake())); log.info(battercake.getDesc() + \u0026#34; 销售价格:\u0026#34; + battercake.cost()); AbstractBattercake battercake2 = new EggBattercake((new ABattercake())); log.info(battercake2.getDesc() + \u0026#34; 销售价格:\u0026#34; + battercake2.cost()); } } 输出\n14:42:18.660 [main] INFO top.fjy8018.designpattern.pattern.structural.decorator.BattercakeTest - 煎饼 加一根香肠 加一个鸡蛋 销售价格:11 14:42:18.660 [main] INFO top.fjy8018.designpattern.pattern.structural.decorator.BattercakeTest - 煎饼 加一个鸡蛋 销售价格:9 适配器模式 介绍 定义将一个类的接口转换成客户期望的另一个接口\n使原本接口不兼容的类可以一起工作\n类型：结构型\n适用场景\n不是软件设计阶段考虑的设计模式,是随着软件维护,由于 不同产品、不同厂家造成功能类似而接口不相同情况下的解决方案 已经存在的类,它的方法和需求不匹配时(方法结果相同或相似) 优点\n能提高类的透明性和复用,现有的类复用但不需要改变 目标类和适配器类解耦,提高程序扩展性 符合开闭原则 缺点\n适配器编写过程需要全面考虑,可能会增加系统的复杂性 增加系统代码可读的难度 分类与扩展\n对象适配器 类适配器 相关设计模式\n适配器模式和外观模式 代码样例——电源适配器 类图\n220V\npackage top.fjy8018.designpattern.pattern.structural.adapter.poweradapter; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2018-10-09 16:24 */ @Slf4j public class AC220 { public int output() { int output = 220; log.info(this.getClass().getSimpleName() + \u0026#34;output:\u0026#34; + output + \u0026#34;V\u0026#34;); return output; } } 5V\npackage top.fjy8018.designpattern.pattern.structural.adapter.poweradapter; /** * @author F嘉阳 * @date 2018-10-09 16:25 */ public interface DC5 { int output(); } 适配器\npackage top.fjy8018.designpattern.pattern.structural.adapter.poweradapter; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2018-10-09 16:25 */ @Slf4j public class Adapter implements DC5 { private AC220 ac220 = new AC220(); @Override public int output() { int output220 = ac220.output(); // 变压器模拟 int output5 = output220 / 44; log.info(\u0026#34;{} 输入{}V,输出{}V\u0026#34;, this.getClass().getSimpleName(), output220, output5); return output5; } } 测试\npackage top.fjy8018.designpattern.pattern.structural.adapter.poweradapter; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; class AdapterTest { @Test void output() { AC220 ac220 = new AC220(); ac220.output(); DC5 dc5 = new Adapter(); dc5.output(); } } 输出\n14:58:14.719 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.poweradapter.AC220 - AC220output:220V 14:58:14.729 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.poweradapter.AC220 - AC220output:220V 14:58:14.729 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.poweradapter.Adapter - Adapter 输入220V,输出5V 代码样例——类适配器 类图\n目标类实现接口\npackage top.fjy8018.designpattern.pattern.structural.adapter.classadapter; /** * 目标类实现接口 * * @author F嘉阳 * @date 2018-10-09 16:05 */ public interface Target { void request(); } 被适配对象\npackage top.fjy8018.designpattern.pattern.structural.adapter.classadapter; import lombok.extern.slf4j.Slf4j; /** * 适配器模式： * 类适配器模式 * 被适配对象 * * @author F嘉阳 * @date 2018-10-09 16:04 */ @Slf4j public class Adaptee { public void adapteeRequest() { log.info(\u0026#34;adapteeRequest...\u0026#34;); } } package top.fjy8018.designpattern.pattern.structural.adapter.classadapter; /** * 适配器模式：将一个类的接口转换成客户希望的另一个接口。适配器模式让那些接口不兼容的类可以一起工作 * * JDK源码：{@link javax.xml.bind.annotation.adapters.XMLAdapter} * spring源码：{@link jdk.internal.org.objectweb.asm.commons.AdviceAdapter} * {@link org.springframework.web.servlet.HandlerAdapter} * JPA源码：{@link org.springframework.orm.jpa.JpaVendorAdapter} * * @author F嘉阳 * @date 2018-10-09 16:07 */ public class Adapter extends Adaptee implements Target { @Override public void request() { // ... 新增业务逻辑 super.adapteeRequest(); } } 目标类具体实现\npackage top.fjy8018.designpattern.pattern.structural.adapter.classadapter; import lombok.extern.slf4j.Slf4j; /** * 目标类具体实现 * * @author F嘉阳 * @date 2018-10-09 16:06 */ @Slf4j public class ConcreteTarget implements Target { @Override public void request() { log.info(\u0026#34;ConcreteTarget...\u0026#34;); } } 测试\npackage top.fjy8018.designpattern.pattern.structural.adapter.classadapter; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; @Slf4j class AdapterTest { /** * 通过适配器实现调用新接口将具体实现转发到旧类中实现 */ @Test void request() { Target newTarget = new ConcreteTarget(); newTarget.request(); Target adapter = new Adapter(); adapter.request(); } } 输出\n15:05:50.583 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.classadapter.ConcreteTarget - ConcreteTarget... 15:05:50.601 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.classadapter.Adaptee - adapteeRequest... 代码样例——对象适配器 类图\nTarget、Adaptee、ConcreteTarget都与类适配器一致、Adapter修改如下\npackage top.fjy8018.designpattern.pattern.structural.adapter.objectadapter; /** * 适配器模式：将一个类的接口转换成客户希望的另一个接口。适配器模式让那些接口不兼容的类可以一起工作 * * @author F嘉阳 * @date 2018-10-09 16:07 */ public class Adapter implements Target { /** * 将继承改成组合方式，降低耦合度 */ private Adaptee adaptee = new Adaptee(); @Override public void request() { // ... 新增业务逻辑 adaptee.adapteeRequest(); } } 测试\npackage top.fjy8018.designpattern.pattern.structural.adapter.objectadapter; import org.junit.jupiter.api.Test; class AdapterTest { @Test void request() { Target newTarget = new ConcreteTarget(); newTarget.request(); Target adapter = new Adapter(); adapter.request(); } } 输出\n15:10:00.866 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.objectadapter.ConcreteTarget - ConcreteTarget... 15:10:00.876 [main] INFO top.fjy8018.designpattern.pattern.structural.adapter.objectadapter.Adaptee - adapteeRequest... 享元模式 介绍 定义：提供了减少对象数量从而改善应用所需的对象结构的方式\n运用共享技术有效地支持大量细粒度的对象\n类型：结构型\n适用场景\n常常应用于系统底层的开发,以便解决系统的性能问题 系统有大量相似对象、需要缓冲池的场景。 优点\n减少对象的创建，降低内存中对象的数量，降低系统的内存，提高效率 减少内存之外的其他资源占用 缺点\n关注内/外部状态、关注线程安全问题 使系统、程序的逻辑复杂化 分类与扩展\n内部状态 外部状态 相关设计模式\n享元模式和代理模式 享元模式和单例模式 代码样例 类图\npackage top.fjy8018.designpattern.pattern.structural.flyweight; /** * 享元模式：利用共享的方式来支持大量细粒度的对象，这些对象一部分内部状态是相同的。 * JDK源码：{@link Integer#valueOf(int)} {@link Integer.IntegerCache} * 其他类型同理 {@link Long#valueOf(long)} {@link Boolean#valueOf(boolean)} {@link Byte#valueOf(byte)} {@link Character#valueOf(char)} * Apache源码：连接池{@link org.apache.commons.pool2.impl.GenericObjectPool#returnObject(Object)},连接池配置 {@link org.apache.commons.pool2.impl.GenericObjectPoolConfig} * * @author F嘉阳 * @date 2018-10-14 16:13 */ public interface Employee { void report(); } 员工工厂类\npackage top.fjy8018.designpattern.pattern.structural.flyweight; import lombok.extern.slf4j.Slf4j; import java.util.Map; import java.util.concurrent.ConcurrentHashMap; /** * 员工工厂类 * * @author F嘉阳 * @date 2018-10-14 16:15 */ @Slf4j public class EmployeeFactory { private static final Map\u0026lt;String, Employee\u0026gt; EMPLOYEE_MAP = new ConcurrentHashMap\u0026lt;\u0026gt;(); public static Employee getEmployee(String department) { Manager manager = (Manager) EMPLOYEE_MAP.get(department); // 如果在对象池中不存在，则新建对象 if (manager == null) { manager = new Manager(department); log.warn(\u0026#34;创建{}部门经理\u0026#34;, department); String content = \u0026#34;报告内容……\u0026#34;; log.warn(\u0026#34;创建报告，内容为:{}\u0026#34;, content); manager.setContent(content); EMPLOYEE_MAP.put(department, manager); } return manager; } } package top.fjy8018.designpattern.pattern.structural.flyweight; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2018-10-14 16:16 */ @Slf4j public class Manager implements Employee { private String content; private String department; /** * 内部状态 */ private String title = \u0026#34;部门经理\u0026#34;; public Manager(String department) { this.department = department; } public void setContent(String content) { this.content = content; } @Override public void report() { log.info(\u0026#34;{}{}，汇报内容：{}\u0026#34;, department, title, content); } } 测试\npackage top.fjy8018.designpattern.pattern.structural.flyweight; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; @Slf4j class EmployeeTest { private static final String departments[] = {\u0026#34;RD\u0026#34;, \u0026#34;QA\u0026#34;, \u0026#34;PM\u0026#34;, \u0026#34;BD\u0026#34;}; @Test void report() { for (int i = 0; i \u0026lt; 10; i++) { String department = departments[(int) (Math.random() * departments.length)]; Employee employee = EmployeeFactory.getEmployee(department); employee.report(); } } } 输出\n15:47:51.177 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建PM部门经理 15:47:51.187 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建报告，内容为:报告内容…… 15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - PM部门经理，汇报内容：报告内容…… 15:47:51.188 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建BD部门经理 15:47:51.188 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建报告，内容为:报告内容…… 15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - BD部门经理，汇报内容：报告内容…… 15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - BD部门经理，汇报内容：报告内容…… 15:47:51.188 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建RD部门经理 15:47:51.188 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建报告，内容为:报告内容…… 15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - RD部门经理，汇报内容：报告内容…… 15:47:51.188 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建QA部门经理 15:47:51.188 [main] WARN top.fjy8018.designpattern.pattern.structural.flyweight.EmployeeFactory - 创建报告，内容为:报告内容…… 15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - QA部门经理，汇报内容：报告内容…… 15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - RD部门经理，汇报内容：报告内容…… 15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - QA部门经理，汇报内容：报告内容…… 15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - BD部门经理，汇报内容：报告内容…… 15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - RD部门经理，汇报内容：报告内容…… 15:47:51.188 [main] INFO top.fjy8018.designpattern.pattern.structural.flyweight.Manager - BD部门经理，汇报内容：报告内容…… 由此可见，在初次创建部门经理后都无需重新创建，后续的汇报可以直接从对象池里直接获取s\n组合模式 介绍 定义：将对象组合成树形结构以表示“部分-整体”的层次结构\n组合模式使客户端对单个对象和组合对象保持一致的方式处理\n类型：结构型\n适用场景\n希望客户端可以忽略组合对象与单个对象的差异时 处理一个树形结构时 优点\n清楚地定义分层次的复杂对象,表示对象的全部或部分层次 让客户端忽略了层次的差异,方便对整个层次结构进行控制 简化客户端代码 缺点\n使设计变得更加抽象 限制类型时会较为复杂 相关设计模式\n组合模式和访问者模式 代码样例 类图\n课程抽象类\npackage top.fjy8018.designpattern.pattern.structural.composite; import java.awt.*; import java.util.Collection; import java.util.Map; /** * 课程抽象类 * 抽象组件应当多加考虑，哪些参数需要，哪些不需要 * * JDK组合模式源码 * {@link java.awt.Container#add(Component)} 增加一个父类操作 * {@link java.util.HashMap#putAll(Map)} * {@link java.util.ArrayList#addAll(Collection)} * * Mybatis源码 * {@link org.apache.ibatis.scripting.xmltags.SqlNode} 公共接口 * 实现：{@link org.apache.ibatis.scripting.xmltags.MixedSqlNode} * * @author F嘉阳 * @date 2020/2/28 16:19 */ public abstract class CatalogComponent { public void add(CatalogComponent catalogComponent) { throw new UnsupportedOperationException(\u0026#34;不支持添加操作\u0026#34;); } public void remove(CatalogComponent catalogComponent) { throw new UnsupportedOperationException(\u0026#34;不支持删除操作\u0026#34;); } public String getName(CatalogComponent catalogComponent) { throw new UnsupportedOperationException(\u0026#34;不支持获取名称操作\u0026#34;); } public Double getPrice(CatalogComponent catalogComponent) { throw new UnsupportedOperationException(\u0026#34;不支持获取价格操作\u0026#34;); } public void print() { throw new UnsupportedOperationException(\u0026#34;不支持获取打印操作\u0026#34;); } } 课程\npackage top.fjy8018.designpattern.pattern.structural.composite; import lombok.AllArgsConstructor; import lombok.extern.slf4j.Slf4j; /** * 课程 * 重写抽象类需要充分考虑业务逻辑 * * @author F嘉阳 * @date 2020/2/28 16:24 */ @Slf4j @AllArgsConstructor public class Course extends CatalogComponent { private String name; private Double price; @Override public String getName(CatalogComponent catalogComponent) { return this.name; } @Override public Double getPrice(CatalogComponent catalogComponent) { return this.price; } @Override public void print() { log.info(\u0026#34;课程名称：{}，价格：{}\u0026#34;,name,price); } } 课程目录\npackage top.fjy8018.designpattern.pattern.structural.composite; import lombok.extern.slf4j.Slf4j; import java.util.ArrayList; import java.util.List; /** * 课程目录 * 可以添加目录节点 * * @author F嘉阳 * @date 2020/2/28 16:34 */ @Slf4j public class CourseCatalog extends CatalogComponent{ private List\u0026lt;CatalogComponent\u0026gt; items = new ArrayList\u0026lt;\u0026gt;(); private String name; public CourseCatalog(String name) { this.name = name; } @Override public void add(CatalogComponent catalogComponent) { items.add(catalogComponent); } @Override public void remove(CatalogComponent catalogComponent) { items.remove(catalogComponent); } @Override public void print() { log.info(\u0026#34;课程目录名称：{}\u0026#34;,name); items.forEach(CatalogComponent::print); } } 测试\npackage top.fjy8018.designpattern.pattern.structural.composite; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; /** * @author F嘉阳 * @date 2020/2/28 16:37 */ @Slf4j class CatalogComponentTest { @Test void add() { // 将课程与课程目录都认为是同一个类型，进行统一处理 CatalogComponent linuxCourse = new Course(\u0026#34;Linux课程\u0026#34;,35.0); CatalogComponent windowsCourse = new Course(\u0026#34;windows课程\u0026#34;,61.0); CatalogComponent macOsCourse = new Course(\u0026#34;macOs课程\u0026#34;,20.0); CatalogComponent springCourse = new Course(\u0026#34;spring课程\u0026#34;,55.0); CatalogComponent servletCourse = new Course(\u0026#34;servlet课程\u0026#34;,11.0); // 课程目录 CatalogComponent javaCourseCatalog = new CourseCatalog(\u0026#34;java课程目录\u0026#34;); CatalogComponent schoolCourseCatalog = new CourseCatalog(\u0026#34;学校课程主目录\u0026#34;); javaCourseCatalog.add(springCourse); javaCourseCatalog.add(servletCourse); // 既可以添加课程，也可以添加目录 schoolCourseCatalog.add(linuxCourse); schoolCourseCatalog.add(windowsCourse); schoolCourseCatalog.add(macOsCourse); schoolCourseCatalog.add(javaCourseCatalog); schoolCourseCatalog.print(); } } 输出\n09:24:29.687 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.CourseCatalog - 课程目录名称：学校课程主目录 09:24:29.691 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.Course - 课程名称：Linux课程，价格：35.0 09:24:29.693 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.Course - 课程名称：windows课程，价格：61.0 09:24:29.694 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.Course - 课程名称：macOs课程，价格：20.0 09:24:29.694 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.CourseCatalog - 课程目录名称：java课程目录 09:24:29.694 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.Course - 课程名称：spring课程，价格：55.0 09:24:29.694 [main] INFO top.fjy8018.designpattern.pattern.structural.composite.Course - 课程名称：servlet课程，价格：11.0 桥接模式 介绍 定义：将抽象部分与它的具体实现部分分离,使它们都可以独立地变化\n通过组合的方式建立两个类之间联系,而不是继承\n类型：结构型\n适用场景\n抽象和具体实现之间增加更多的灵活性 一个类存在两个(或多个)独立变化的维度,且这两个(或多个) 维度都需要独立进行扩展 不希望使用继承,或因为多层继承导致系统类的个数剧增 优点\n分离抽象部分及其具体实现部分 提高了系统的可扩展性 符合开闭原则 符合合成复用原则 缺点\n增加了系统的理解与设计难度 需要正确地识别出系统中两个独立变化的维度 相关设计模式\n桥接模式和组合模式 桥接模式和适配器模式 代码样例 类图\n桥接模式核心在于将抽象与实现通过组合模式连接，使抽象和实现都可以自由扩展\n银行账号——行为接口\npackage top.fjy8018.designpattern.pattern.structural.bridge; import java.sql.Driver; import java.sql.DriverManager; /** * 银行账号——行为接口 * * JDBC 为操作不同的数据库提供了同一个{@link java.sql.Connection} 连接 * 由MySQL或Oracle连接驱动实现相关接口操作数据库 * * {@link java.sql.Driver} * {@link DriverManager} 中的 {@link java.sql.DriverInfo} * 实现对 {@link java.sql.Driver} 的具体封装 * * JDBC 实现的 {@link com.mysql.cj.jdbc.Driver} 初始化时调 {@link DriverManager#registerDriver(Driver)} * * @author F嘉阳 * @date 2020/2/29 9:55 */ public interface Account { /** * 打开一个账号 * * @return */ Account openAccount(); /** * 查看账号类型 */ void showAccountType(); } 抽象银行\npackage top.fjy8018.designpattern.pattern.structural.bridge; import lombok.AllArgsConstructor; /** * 抽象银行 * 组合账户接口，交由子类实现具体的行为 * * @author F嘉阳 * @date 2020/2/29 10:05 */ @AllArgsConstructor public abstract class Bank { protected Account account; /** * 打开一个账户 * 该方法定义与接口相同，方法名称可以不同 * 当前抽象类的openAccount方法需要委托给Account接口openAccount方法实现 * 当前类为抽象类，委托给接口的实现类执行，体现抽象与实现分离 * @return */ abstract Account openAccount(); } 活期账户\npackage top.fjy8018.designpattern.pattern.structural.bridge; import lombok.extern.slf4j.Slf4j; /** * 活期账户 * * @author F嘉阳 * @date 2020/2/29 9:58 */ @Slf4j public class SavingAccount implements Account { /** * 打开一个账号 * * @return */ @Override public Account openAccount() { log.info(\u0026#34;打开活期账户\u0026#34;); return new SavingAccount(); } /** * 查看账号类型 */ @Override public void showAccountType() { log.info(\u0026#34;这是一个活期账户\u0026#34;); } } 定期账户\npackage top.fjy8018.designpattern.pattern.structural.bridge; import lombok.extern.slf4j.Slf4j; /** * 定期账户 * * @author F嘉阳 * @date 2020/2/29 9:57 */ @Slf4j public class DepositAccount implements Account{ /** * 打开一个账号 * * @return */ @Override public Account openAccount() { log.info(\u0026#34;打开定期账户\u0026#34;); return new DepositAccount(); } /** * 查看账号类型 */ @Override public void showAccountType() { log.info(\u0026#34;这是一个定期账户\u0026#34;); } } 广州银行实现\npackage top.fjy8018.designpattern.pattern.structural.bridge; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2020/2/29 10:09 */ @Slf4j public class GCBBank extends Bank { public GCBBank(Account account) { super(account); } /** * 打开一个账户 * 该方法定义与接口相同，方法名称可以不同 * 当前抽象类的openAccount方法需要委托给Account接口openAccount方法实现 * 当前类为抽象类，委托给接口的实现类执行，体现抽象与实现分离 * * @return */ @Override Account openAccount() { log.info(\u0026#34;打开广州银行账户\u0026#34;); // 委托 account.openAccount(); return account; } } 工商银行实现\npackage top.fjy8018.designpattern.pattern.structural.bridge; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2020/2/29 10:11 */ @Slf4j public class ICBCBank extends Bank { public ICBCBank(Account account) { super(account); } /** * 打开一个账户 * 该方法定义与接口相同，方法名称可以不同 * 当前抽象类的openAccount方法需要委托给Account接口openAccount方法实现 * 当前类为抽象类，委托给接口的实现类执行，体现抽象与实现分离 * * @return */ @Override Account openAccount() { log.info(\u0026#34;打开工商银行账户\u0026#34;); // 委托 account.openAccount(); return account; } } 测试\npackage top.fjy8018.designpattern.pattern.structural.bridge; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; /** * @author F嘉阳 * @date 2020/2/29 14:30 */ @Slf4j class AccountTest { @Test void openAccount() { Bank gcbBank = new GCBBank(new DepositAccount()); Account gcbAccount = gcbBank.openAccount(); gcbAccount.showAccountType(); Bank icbcBank = new ICBCBank(new SavingAccount()); Account icbcAccount = icbcBank.openAccount(); icbcAccount.showAccountType(); } } 输出\n14:36:13.478 [main] INFO top.fjy8018.designpattern.pattern.structural.bridge.GCBBank - 打开广州银行账户 14:36:13.488 [main] INFO top.fjy8018.designpattern.pattern.structural.bridge.DepositAccount - 打开定期账户 14:36:13.489 [main] INFO top.fjy8018.designpattern.pattern.structural.bridge.DepositAccount - 这是一个定期账户 14:36:13.489 [main] INFO top.fjy8018.designpattern.pattern.structural.bridge.ICBCBank - 打开工商银行账户 14:36:13.489 [main] INFO top.fjy8018.designpattern.pattern.structural.bridge.SavingAccount - 打开活期账户 14:36:13.489 [main] INFO top.fjy8018.designpattern.pattern.structural.bridge.SavingAccount - 这是一个活期账户 代理模式 介绍 定义：为其他对象提供一种代理,以控制对这个对象的访问\n代理对象在客户端和目标对象之间起到中介的作用\n类型：结构型\n适用场景\n保护目标对象 增强目标对象 优点\n代理模式能将代理对象与真实被调用的目标对象分离 一定程度上降低了系统的耦合度，扩展性好 保护目标对象 增强目标对象 缺点\n代理模式会造成系统设计中类的数目增加 在客户端和目标对象增加一个代理对象，会造成请求处理速度变慢 增加系统的复杂度 分类与扩展\n静态代理 显式在业务实现类新增代理类实现代理功能，在代理类中对同名的业务方法进行增强和包装 CGLib代理 可以代理类 代理类对象时，CGLib会生成代理类的子类，覆盖对应的方法 由于利用了继承和重写的特性，若类或者方法被标识为final，则无法进行代理 动态代理 无法代理类，可以代理接口 若增强的对象新增了一个方法，而接口中不包含该新增方法，则无法进行增强 对类的代理是在使用时动态创建了一个代理类的class文件，该class文件被字节码引擎执行，被代理类进行方法调用 Spring代理选择 当Bean有实现接口时, Spring就会用JDK的动态代理 当Bean没有实现接口时, Spring使用CGLib 可以强制使用 Cglib 在Spring配置中加入\u0026lt;aop: aspectj- autoproxy proxy-target-dass=\u0026quot;true\u0026quot;/\u0026gt; 参考资料： https://docs.spring.io/spring-framework/docs/current/spring-framework-reference/core.html 代理速度对比 CGLib：基于ASM字节码生成，比反射效率高 JDK动态代理 速度对比：万次执行的情况下，JDK7与JDK8的动态代理性能比CGLib速度快20%左右 相关设计模式\n代理模式和装饰者模式 代理模式和适配器模式 代码样例——Spring 分库代理实现 自定义数据源上下文\npackage top.fjy8018.designpattern.pattern.structural.proxy.db; /** * 数据源上下文 * * @author F嘉阳 * @date 2020/2/29 16:27 */ public class DataSourceContextHolder { /** * 存放数据源Bean Name */ private static final ThreadLocal\u0026lt;String\u0026gt; CONTEXT_HOLDER = new ThreadLocal\u0026lt;\u0026gt;(); /** * 配置哪个数据库，则动态数据源线程就会获取到哪个数据库 * @param dbType */ public static void setDBType(String dbType){ CONTEXT_HOLDER.set(dbType); } public static String getDBType(){ return CONTEXT_HOLDER.get(); } /** * 清除，防止内存泄漏 */ public static void clear(){ CONTEXT_HOLDER.remove(); } } 定义动态数据源\npackage top.fjy8018.designpattern.pattern.structural.proxy.db; import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource; /** * Spring分库实现 * * @author F嘉阳 * @date 2020/2/29 16:26 */ public class DynamicDataSource extends AbstractRoutingDataSource { /** * 动态获取数据库 * @return 当前线程所属数据源 */ @Override protected Object determineCurrentLookupKey() { return DataSourceContextHolder.getDBType(); } } 多数据源配置\n\u0026lt;bean id=\u0026#34;dataSource\u0026#34; class=\u0026#34;top.fjy8018.designpattern.pattern.structural.proxy.db.DynamicDataSource\u0026#34;\u0026gt; \u0026lt;property name=\u0026#34;targetDataSources\u0026#34;\u0026gt; \u0026lt;map key-type=\u0026#34;java.lang.String\u0026#34;\u0026gt; \u0026lt;entry value-ref=\u0026#34;db0\u0026#34; key=\u0026#34;db0\u0026#34;\u0026gt;\u0026lt;/entry\u0026gt; \u0026lt;entry value-ref=\u0026#34;db1\u0026#34; key=\u0026#34;db1\u0026#34;\u0026gt;\u0026lt;/entry\u0026gt; \u0026lt;/map\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;defaultTargetDataSource\u0026#34; ref=\u0026#34;db0\u0026#34;\u0026gt;\u0026lt;/property\u0026gt; \u0026lt;/bean\u0026gt; \u0026lt;bean id=\u0026#34;sqlSessionFactory\u0026#34; class=\u0026#34;org.mybatis.spring.SqlSessionFactoryBean\u0026#34;\u0026gt; \u0026lt;property name=\u0026#34;dataSource\u0026#34; ref=\u0026#34;dataSource\u0026#34; /\u0026gt; \u0026lt;/bean\u0026gt; 代码样例——静态代理 类图\n订单\npackage top.fjy8018.designpattern.pattern.structural.proxy; import lombok.Getter; import lombok.Setter; /** * 订单 * * @author F嘉阳 * @date 2020/2/29 16:12 */ @Getter @Setter public class Order { private Object orderInfo; private Integer userId; } DAO层\npackage top.fjy8018.designpattern.pattern.structural.proxy; /** * DAO层——与业务无关 * 仿mybatis * * @author F嘉阳 * @date 2020/2/29 16:15 */ public interface IOrderDao { int insert(Order order); } package top.fjy8018.designpattern.pattern.structural.proxy; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2020/2/29 16:17 */ @Slf4j public class IOrderDaoImpl implements IOrderDao { @Override public int insert(Order order) { log.info(\u0026#34;DAO层添加订单记录成功\u0026#34;); return 1; } } 订单服务层\npackage top.fjy8018.designpattern.pattern.structural.proxy; /** * 订单服务层 * * @author F嘉阳 * @date 2020/2/29 16:14 */ public interface IOrderService { /** * 保存订单 * @param order * @return 保存的行数 */ int save(Order order); } package top.fjy8018.designpattern.pattern.structural.proxy; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2020/2/29 16:16 */ @Slf4j public class IOrderServiceImpl implements IOrderService { /** * 模拟Spring注入 */ private IOrderDao orderDao = new IOrderDaoImpl(); /** * 保存订单 * * @param order * @return 保存的行数 */ @Override public int save(Order order) { log.info(\u0026#34;Service层调用DAO层添加订单记录成功\u0026#34;); return orderDao.insert(order); } } 静态代理类\npackage top.fjy8018.designpattern.pattern.structural.proxy.staticproxy; import lombok.extern.slf4j.Slf4j; import top.fjy8018.designpattern.pattern.structural.proxy.IOrderService; import top.fjy8018.designpattern.pattern.structural.proxy.IOrderServiceImpl; import top.fjy8018.designpattern.pattern.structural.proxy.Order; import top.fjy8018.designpattern.pattern.structural.proxy.db.DataSourceContextHolder; /** * 静态代理实现分库 * * @author F嘉阳 * @date 2020/2/29 16:20 */ @Slf4j public class OrderServiceStaticProxy { /** * 模拟Spring注入 */ private IOrderService orderService = new IOrderServiceImpl(); private static final String DB = \u0026#34;db\u0026#34;; /** * 对源方法增强 * * @param order * @return */ public int save(Order order) { beforeMethod(); // 通过用户ID对2取模分库 Integer userId = order.getUserId(); int dbRouter = userId % 2; log.info(\u0026#34;静态代理分配到【db{}】处理数据\u0026#34;, dbRouter); // 设置DataSource DataSourceContextHolder.setDBType(DB + String.valueOf(dbRouter)); afterMethod(); return orderService.save(order); } private void beforeMethod() { log.info(\u0026#34;静态代理before code\u0026#34;); } private void afterMethod() { log.info(\u0026#34;静态代理after code\u0026#34;); } } 测试\npackage top.fjy8018.designpattern.pattern.structural.proxy.staticproxy; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import top.fjy8018.designpattern.pattern.structural.proxy.Order; import static org.junit.jupiter.api.Assertions.*; /** * @author F嘉阳 * @date 2020/2/29 17:17 */ @Slf4j class OrderServiceStaticProxyTest { @Test void save() { Order order1 = new Order(); order1.setUserId(1); Order order2 = new Order(); order2.setUserId(2); OrderServiceStaticProxy proxy = new OrderServiceStaticProxy(); proxy.save(order1); proxy.save(order2); } } 输出\n17:20:36.012 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.staticproxy.OrderServiceStaticProxy - 静态代理before code 17:20:36.022 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.staticproxy.OrderServiceStaticProxy - 静态代理分配到【db1】处理数据 17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.staticproxy.OrderServiceStaticProxy - 静态代理after code 17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderServiceImpl - Service层调用DAO层添加订单记录成功 17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderDaoImpl - DAO层添加订单记录成功 17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.staticproxy.OrderServiceStaticProxy - 静态代理before code 17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.staticproxy.OrderServiceStaticProxy - 静态代理分配到【db0】处理数据 17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.staticproxy.OrderServiceStaticProxy - 静态代理after code 17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderServiceImpl - Service层调用DAO层添加订单记录成功 17:20:36.024 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderDaoImpl - DAO层添加订单记录成功 代码样例——动态代理 类图\n动态代理类\npackage top.fjy8018.designpattern.pattern.structural.proxy.dynamic; import lombok.extern.slf4j.Slf4j; import org.apache.ibatis.binding.MapperProxy; import org.springframework.aop.framework.ProxyFactoryBean; import top.fjy8018.designpattern.pattern.structural.proxy.Order; import top.fjy8018.designpattern.pattern.structural.proxy.db.DataSourceContextHolder; import java.lang.reflect.InvocationHandler; import java.lang.reflect.Method; import java.lang.reflect.Proxy; /** * 动态代理实现分库 * 动态代理适用场景更加广泛，可以实现通用分库逻辑 * \u0026lt;p\u0026gt; * {@link ProxyFactoryBean#getObject()} 代理工厂类 * {@link org.springframework.aop.framework.JdkDynamicAopProxy} spring AOP实现，对动态代理封装 * {@link org.springframework.aop.framework.CglibAopProxy} Cglib 对类进行代理增强 * {@link org.apache.ibatis.binding.MapperProxyFactory#newInstance(MapperProxy)} 实现mapper代理 * 实际封装在 {@link MapperProxy#invoke(Object, Method, Object[])} * 其获取的方法利用了享元模式 {@link MapperProxy#cachedMapperMethod(Method)} * * @author F嘉阳 * @date 2020/3/3 11:00 */ @Slf4j public class OrderServiceDynamicProxy implements InvocationHandler { private Object target; private static final String DB = \u0026#34;db\u0026#34;; public OrderServiceDynamicProxy(Object target) { this.target = target; } /** * 绑定目标对象 * * @return */ public Object bind() { Class clazz = target.getClass(); return Proxy.newProxyInstance(clazz.getClassLoader(), clazz.getInterfaces(), this); } /** * 代理 * * @param proxy 代理对象，一般不调用该参数 * @param method 要被增强的方法对象 * @param args 方法参数 * @return * @throws Throwable */ @Override public Object invoke(Object proxy, Method method, Object[] args) throws Throwable { // 参数传递 Object argObject = args[0]; beforeMethod(argObject); // object是要被增强的方法的返回值 Object object = method.invoke(target, args); afterMethod(); return object; } /** * 前置增强方法 * * @param args 方法参数 */ private void beforeMethod(Object args) { int userId = 0; // 对参数取模 log.info(\u0026#34;动态代理，before code\u0026#34;); if (args instanceof Order) { Order order = (Order) args; userId = order.getUserId(); } int dbRouter = userId % 2; log.info(\u0026#34;动态代理分配到【db{}】处理数据\u0026#34;, dbRouter); // 设置DataSource DataSourceContextHolder.setDBType(DB + dbRouter); } private void afterMethod() { log.info(\u0026#34;动态代理after code\u0026#34;); } } 测试类\npackage top.fjy8018.designpattern.pattern.structural.proxy.dynamic; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import top.fjy8018.designpattern.pattern.structural.proxy.IOrderService; import top.fjy8018.designpattern.pattern.structural.proxy.IOrderServiceImpl; import top.fjy8018.designpattern.pattern.structural.proxy.Order; /** * @author F嘉阳 * @date 2020/3/3 15:54 */ @Slf4j class OrderServiceDynamicProxyTest { @Test void invoke() { Order order1 = new Order(); order1.setUserId(1); Order order2 = new Order(); order2.setUserId(2); // 动态代理直接代理接口 IOrderService proxy = (IOrderService) new OrderServiceDynamicProxy(new IOrderServiceImpl()).bind(); proxy.save(order1); proxy.save(order2); } } 输出\n15:57:57.234 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.dynamic.OrderServiceDynamicProxy - 动态代理，before code 15:57:57.238 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.dynamic.OrderServiceDynamicProxy - 动态代理分配到【db1】处理数据 15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderServiceImpl - Service层调用DAO层添加订单记录成功 15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderDaoImpl - DAO层添加订单记录成功 15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.dynamic.OrderServiceDynamicProxy - 动态代理after code 15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.dynamic.OrderServiceDynamicProxy - 动态代理，before code 15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.dynamic.OrderServiceDynamicProxy - 动态代理分配到【db0】处理数据 15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderServiceImpl - Service层调用DAO层添加订单记录成功 15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.IOrderDaoImpl - DAO层添加订单记录成功 15:57:57.248 [main] INFO top.fjy8018.designpattern.pattern.structural.proxy.dynamic.OrderServiceDynamicProxy - 动态代理after code 动态代理源码分析 Proxy类\npackage java.lang.reflect; public class Proxy implements java.io.Serializable { // 反射调用开始 @CallerSensitive public static Object newProxyInstance(ClassLoader loader, Class\u0026lt;?\u0026gt;[] interfaces, InvocationHandler h) throws IllegalArgumentException { Objects.requireNonNull(h); final Class\u0026lt;?\u0026gt;[] intfs = interfaces.clone(); final SecurityManager sm = System.getSecurityManager(); if (sm != null) { // 合法性检查 checkProxyAccess(Reflection.getCallerClass(), loader, intfs); } /* * 动态生成代理类 */ Class\u0026lt;?\u0026gt; cl = getProxyClass0(loader, intfs); /* * 实例化并返回新创建的类实例对象 */ try { if (sm != null) { checkNewProxyPermission(Reflection.getCallerClass(), cl); } final Constructor\u0026lt;?\u0026gt; cons = cl.getConstructor(constructorParams); final InvocationHandler ih = h; if (!Modifier.isPublic(cl.getModifiers())) { AccessController.doPrivileged(new PrivilegedAction\u0026lt;Void\u0026gt;() { public Void run() { cons.setAccessible(true); return null; } }); } return cons.newInstance(new Object[]{h}); } catch (IllegalAccessException|InstantiationException e) { throw new InternalError(e.toString(), e); } catch (InvocationTargetException e) { Throwable t = e.getCause(); if (t instanceof RuntimeException) { throw (RuntimeException) t; } else { throw new InternalError(t.toString(), t); } } catch (NoSuchMethodException e) { throw new InternalError(e.toString(), e); } } private static void checkProxyAccess(Class\u0026lt;?\u0026gt; caller, ClassLoader loader, Class\u0026lt;?\u0026gt;... interfaces) { // 安全校验 SecurityManager sm = System.getSecurityManager(); if (sm != null) { ClassLoader ccl = caller.getClassLoader(); if (VM.isSystemDomainLoader(loader) \u0026amp;\u0026amp; !VM.isSystemDomainLoader(ccl)) { sm.checkPermission(SecurityConstants.GET_CLASSLOADER_PERMISSION); } ReflectUtil.checkProxyPackageAccess(ccl, interfaces); } } // 生成代理类 private static Class\u0026lt;?\u0026gt; getProxyClass0(ClassLoader loader, Class\u0026lt;?\u0026gt;... interfaces) { // 若一个类中的接口数量超过限制则报错 if (interfaces.length \u0026gt; 65535) { throw new IllegalArgumentException(\u0026#34;interface limit exceeded\u0026#34;); } // 从代理类缓存中读取 return proxyClassCache.get(loader, interfaces); } } 从代理类缓存中读取实现\npackage java.lang.reflect; final class WeakCache\u0026lt;K, P, V\u0026gt; { public V get(K key, P parameter) { Objects.requireNonNull(parameter); expungeStaleEntries(); Object cacheKey = CacheKey.valueOf(key, refQueue); // 使用并发包 ConcurrentMap\u0026lt;Object, Supplier\u0026lt;V\u0026gt;\u0026gt; valuesMap = map.get(cacheKey); if (valuesMap == null) { ConcurrentMap\u0026lt;Object, Supplier\u0026lt;V\u0026gt;\u0026gt; oldValuesMap = map.putIfAbsent(cacheKey, valuesMap = new ConcurrentHashMap\u0026lt;\u0026gt;()); if (oldValuesMap != null) { valuesMap = oldValuesMap; } } Object subKey = Objects.requireNonNull(subKeyFactory.apply(key, parameter)); Supplier\u0026lt;V\u0026gt; supplier = valuesMap.get(subKey); Factory factory = null; while (true) { if (supplier != null) { // 循环获取，直到获取成功才返回 V value = supplier.get(); if (value != null) { return value; } } // 若首次创建则通过工厂类创建对应的类 if (factory == null) { factory = new Factory(key, parameter, subKey, valuesMap); } if (supplier == null) { supplier = valuesMap.putIfAbsent(subKey, factory); if (supplier == null) { supplier = factory; } } else { if (valuesMap.replace(subKey, supplier, factory)) { supplier = factory; } else { supplier = valuesMap.get(subKey); } } } } } 断点查看执行情况\n","date":"2020-03-04T10:44:04+08:00","permalink":"/archives/239/","title":"设计模式学习——结构型"},{"content":"简单工厂 介绍 定义由一个工厂对象决定创建出哪一种产品类的实例\n类型：创建型,但不属于GOF23种设计模式\n工厂类负责创建的对象比较少 客户端(应用层)只知道传入工厂类的参数，对于如何创建对象(逻辑)不关心 只需要传入一个正确的参数,就可以获取你所需要的对象 而无须知道其创建细节 缺点\n工厂类的职责相对过重，增加新的产品需要修改工厂类的判断逻辑，违背开闭原则 代码样例 类图\npackage top.fjy8018.designpattern.pattern.creational.simplefactory; import ch.qos.logback.classic.LoggerContext; import java.util.Calendar; /** * 简单工厂 * JDK样例 {@link Calendar#getInstance()} 其调用的 {@link Calendar#createCalendar} 就是简单工厂的体现 * 日志框架样例 {@link org.slf4j.LoggerFactory#getLogger(String)} -\u0026gt; {@link LoggerContext#getLogger(String)} * * @author F嘉阳 * @date 2018-09-23 16:13 */ public abstract class AbstractVideo { public abstract void produce(); } package top.fjy8018.designpattern.pattern.creational.simplefactory; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2018-09-23 16:13 */ @Slf4j public class JavaAbstractVideo extends AbstractVideo { @Override public void produce() { log.info(\u0026#34;生产Java视频\u0026#34;); } } package top.fjy8018.designpattern.pattern.creational.simplefactory; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2018-09-23 16:14 */ @Slf4j public class PythonAbstractVideo extends AbstractVideo { @Override public void produce() { log.info(\u0026#34;生产Python视频\u0026#34;); } } package top.fjy8018.designpattern.pattern.creational.simplefactory; /** * @author F嘉阳 * @date 2018-09-23 16:16 */ public class VideoFactory { /** * 由工厂负责创建类，业务层无需关心具体实现 * 其缺点是不符合开闭原则，即如果具体实现类扩展后，工厂类也必须修改 * * @param type * @return */ public AbstractVideo getVideo(String type) { switch (type) { case \u0026#34;Java\u0026#34;: return new JavaAbstractVideo(); case \u0026#34;Python\u0026#34;: return new PythonAbstractVideo(); default: return null; } } /** * 通过反射在一定程度弥补简单工厂不足（开闭原则） * * @param c * @return */ public AbstractVideo getVideo(Class c) { AbstractVideo video = null; try { video = (AbstractVideo) Class.forName(c.getName()).newInstance(); } catch (ClassNotFoundException | IllegalAccessException | InstantiationException e) { e.printStackTrace(); } return video; } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.simplefactory; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; class AbstractVideoTest { /** * 不使用简单工厂，则业务层需要引入具体的实现类 */ @Test void before() { AbstractVideo video = new JavaAbstractVideo(); video.produce(); } /** * 使用简单工厂后业务层无需导入具体的实现类，即不关心具体实现过程 */ @Test void after() { VideoFactory factory = new VideoFactory(); AbstractVideo video = factory.getVideo(\u0026#34;Java\u0026#34;); video.produce(); } /** * 通过传递类名方式改进扩展性问题 */ @Test void improve() { VideoFactory factory = new VideoFactory(); AbstractVideo video = factory.getVideo(JavaAbstractVideo.class); video.produce(); } } 工厂方法 介绍 定义：定义一个创建对象的接口，但让实现这个接口的类来决定实例化哪个类工厂方法让类的实例化推迟到子类中进行\n类型：创建型\n特点：\n创建对象需要大量重复的代码 客户端(应用层)不依赖于产品类实例如何被创建、实现等细节 一个类通过其子类来指定创建哪个对象 优点：\n用户只需要关心所需产品对应的工厂,无须关心创建细节 加入新产品符合开闭原则,提高可扩展性 缺点：\n类的个数容易过多,增加复杂度 增加了系统的抽象性和理解难度 代码样例 类图\npackage top.fjy8018.designpattern.pattern.creational.factorymethod; /** * 工厂方法 * * @author F嘉阳 * @date 2018-09-23 16:13 */ public abstract class AbstractVideo { public abstract void produce(); } package top.fjy8018.designpattern.pattern.creational.factorymethod; import sun.misc.Launcher; import java.util.Collection; /** * 定义一个创建产品对象的工厂接口，将实际创建工作推迟到子类当中 * 工厂方法抽象类，只提供契约，对于已知的逻辑可以提供默认实现 * \u0026lt;p\u0026gt; * 提高产品与产品族的拓展性 * \u0026lt;p\u0026gt; * JDK源码样例1：{@link java.net.URLStreamHandlerFactory#createURLStreamHandler(String)} -\u0026gt; {@link Launcher.Factory#createURLStreamHandler(String)} * JDK源码样例2：{@link Collection#iterator()} * * @author F嘉阳 * @date 2018-09-23 16:16 */ public abstract class AbstractVideoFactory { public abstract AbstractVideo getVideo(); } package top.fjy8018.designpattern.pattern.creational.factorymethod; /** * @author F嘉阳 * @date 2018-09-23 20:11 */ public class JavaAbstractVideoFactory extends AbstractVideoFactory { @Override public AbstractVideo getVideo() { return new JavaVideo(); } } package top.fjy8018.designpattern.pattern.creational.factorymethod; import lombok.extern.slf4j.Slf4j; /** * 具体的产品 * * @author F嘉阳 * @date 2018-09-23 16:13 */ @Slf4j public class JavaVideo extends AbstractVideo { @Override public void produce() { log.info(\u0026#34;生产Java视频\u0026#34;); } } package top.fjy8018.designpattern.pattern.creational.factorymethod; /** * @author F嘉阳 * @date 2018-09-23 20:11 */ public class PythonAbstractVideoFactory extends AbstractVideoFactory { @Override public AbstractVideo getVideo() { return new PythonVideo(); } } package top.fjy8018.designpattern.pattern.creational.factorymethod; import lombok.extern.slf4j.Slf4j; /** * 具体的产品 * * @author F嘉阳 * @date 2018-09-23 16:14 */ @Slf4j public class PythonVideo extends AbstractVideo { @Override public void produce() { log.info(\u0026#34;生产Python视频\u0026#34;); } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.factorymethod; import org.junit.jupiter.api.Test; class AbstractVideoFactoryTest { @Test void getVideo() { AbstractVideoFactory factory = new JavaAbstractVideoFactory(); factory.getVideo().produce(); } } 抽象工厂 介绍 定义:抽象工厂模式提供一个创建一系列相关或相互依赖对象的接口\n无须指定它们具体的类\n类型：创建型\n适用场景\n客户端（应用层）不依赖于产品类实例如何被创建、实现等细节 强调一系列相关的产品对象（属于同一产品族）一起使用创建对象需要大量重复的代码 提供一个产品类的库，所有的产品以同样的接口出现，从而使客户端不依赖于具体实现 优点：\n具体产品在应用层代码隔离,无须关心创建细节 将一个系列的产品族统一到一起创建 缺点：\n规定了所有可能被创建的产品集合，产品族中扩展新的产品困难，需要修改抽象工厂的接口 增加了系统的抽象性和理解难度 抽象工程与普通工厂区分 使用抽象工厂易于扩展产品族\n其与工厂方法区别为：抽象工厂关心产品族，工厂方法关系产品等级\n代码样例 类图\npackage top.fjy8018.designpattern.pattern.creational.abstractfactory; /** * 工厂方法 * * @author F嘉阳 * @date 2018-09-23 16:13 */ public abstract class AbstractVideo { public abstract void produce(); } package top.fjy8018.designpattern.pattern.creational.abstractfactory; /** * 工厂方法 * * @author F嘉阳 * @date 2018-09-23 16:13 */ public abstract class AbstractArticle { public abstract void produce(); } package top.fjy8018.designpattern.pattern.creational.abstractfactory; /** * 抽象工厂：抽象工厂模式可以向客户端提供一个接口，使客户端在不必指定产品的具体的情况下，创建多个\u0026lt;strong\u0026gt;产品族\u0026lt;/strong\u0026gt;中的产品对象 * 使用抽象工厂易于扩展产品族 * 但抽象工厂不适于新增产品等级，即新增video同级的产品 * 其与工厂方法区别为：抽象工厂关心产品族，工厂方法关系产品等级 * * JDK源码样例：{@link java.sql.Connection} 该类的实现类都属于同一个产品族 * * @author F嘉阳 * @date 2018-09-23 22:43 */ public interface CourseFactory { AbstractVideo video(); AbstractArticle article(); } package top.fjy8018.designpattern.pattern.creational.abstractfactory; import lombok.extern.slf4j.Slf4j; /** * 具体的产品 * * @author F嘉阳 * @date 2018-09-23 16:13 */ @Slf4j public class JavaArticle extends AbstractArticle { @Override public void produce() { log.info(\u0026#34;生产Java文章\u0026#34;); } } package top.fjy8018.designpattern.pattern.creational.abstractfactory; import lombok.extern.slf4j.Slf4j; /** * 具体的产品 * * @author F嘉阳 * @date 2018-09-23 16:13 */ @Slf4j public class JavaVideo extends AbstractVideo { @Override public void produce() { log.info(\u0026#34;生产Java视频\u0026#34;); } } package top.fjy8018.designpattern.pattern.creational.abstractfactory; import lombok.extern.slf4j.Slf4j; /** * 具体的产品 * * @author F嘉阳 * @date 2018-09-23 16:13 */ @Slf4j public class PythonArticle extends AbstractArticle { @Override public void produce() { log.info(\u0026#34;生产Python文章\u0026#34;); } } package top.fjy8018.designpattern.pattern.creational.abstractfactory; import lombok.extern.slf4j.Slf4j; /** * 具体的产品 * * @author F嘉阳 * @date 2018-09-23 16:14 */ @Slf4j public class PythonVideo extends AbstractVideo { @Override public void produce() { log.info(\u0026#34;生产Python视频\u0026#34;); } } package top.fjy8018.designpattern.pattern.creational.abstractfactory; /** * 从一个产品工厂生产出来的产品一定属于同一个族 * * @author F嘉阳 * @date 2018-09-23 22:46 */ public class JavaCourseFactory implements CourseFactory { @Override public AbstractVideo video() { return new JavaVideo(); } @Override public AbstractArticle article() { return new JavaArticle(); } } package top.fjy8018.designpattern.pattern.creational.abstractfactory; /** * 从一个产品工厂生产出来的产品一定属于同一个族 * * @author F嘉阳 * @date 2018-09-23 22:46 */ public class PythonCourseFactory implements CourseFactory { @Override public AbstractVideo video() { return new PythonVideo(); } @Override public AbstractArticle article() { return new PythonArticle(); } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.abstractfactory; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; class CourseFactoryTest { @Test void test() { CourseFactory factory = new JavaCourseFactory(); factory.article().produce(); factory.video().produce(); } } 建造者 介绍 定义:将一个复杂对象的构建与它的表示分离,使得同样的构建过程可以创建不同的表示\n用户只需指定需要建造的类型就可以得到它们,建造过程及细节不需要知道\n类型：创建型\n适用场景\n如果一个对象有非常复杂的内部结构(很多属性) 想把复杂对象的创建和使用分离 优点：\n封装性好,刨建和使用分离 扩展性好、建造类之间独立、一定程度上解耦 缺点：\n产生多余的 Builder对象 产品内部发生变化,建造者都要修改,成本较大 代码样例 类图\npackage top.fjy8018.designpattern.pattern.creational.builder; /** * 建造者抽象类 * * @author F嘉阳 * @date 2018-09-25 17:15 */ public abstract class AbstractCourseBuilder { public abstract void courseName(String courseName); public abstract void coursePPT(String coursePPT); public abstract void courseVideo(String courseVideo); /** * question \u0026amp; answer */ public abstract void courseQA(String courseQA); public abstract BCourse build(); } package top.fjy8018.designpattern.pattern.creational.builder; /** * 实体类 * * @author F嘉阳 * @date 2018-09-25 17:13 */ public class BCourse { private String courseName; private String coursePPT; private String courseVideo; /** * question \u0026amp; answer */ private String courseQA; public String getCourseName() { return courseName; } public void setCourseName(String courseName) { this.courseName = courseName; } public String getCoursePPT() { return coursePPT; } public void setCoursePPT(String coursePPT) { this.coursePPT = coursePPT; } public String getCourseVideo() { return courseVideo; } public void setCourseVideo(String courseVideo) { this.courseVideo = courseVideo; } public String getCourseQA() { return courseQA; } public void setCourseQA(String courseQA) { this.courseQA = courseQA; } @Override public String toString() { return \u0026#34;BCourse{\u0026#34; + \u0026#34;courseName=\u0026#39;\u0026#34; + courseName + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, coursePPT=\u0026#39;\u0026#34; + coursePPT + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, courseVideo=\u0026#39;\u0026#34; + courseVideo + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, courseQA=\u0026#39;\u0026#34; + courseQA + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#39;}\u0026#39;; } } package top.fjy8018.designpattern.pattern.creational.builder; /** * 教练-\u0026gt;建造课程 * * @author F嘉阳 * @date 2018-09-25 18:54 */ public class Coach { private AbstractCourseBuilder builder; public Coach(AbstractCourseBuilder builder) { this.builder = builder; } public BCourse createCourse(String courseName, String coursePPT, String courseVideo, String courseQA) { builder.courseName(courseName); builder.coursePPT(coursePPT); builder.courseVideo(courseVideo); builder.courseQA(courseQA); return builder.build(); } } package top.fjy8018.designpattern.pattern.creational.builder; /** * 建造者实现类 * * @author F嘉阳 * @date 2018-09-25 17:17 */ public class CourseBuilder extends AbstractCourseBuilder { private BCourse course = new BCourse(); @Override public void courseName(String courseName) { course.setCourseName(courseName); } @Override public void coursePPT(String coursePPT) { course.setCoursePPT(coursePPT); } @Override public void courseVideo(String courseVideo) { course.setCourseVideo(courseVideo); } @Override public void courseQA(String courseQA) { course.setCourseQA(courseQA); } @Override public BCourse build() { return course; } } 链式调用 package top.fjy8018.designpattern.pattern.creational.builder.chain; import com.google.common.collect.ImmutableSet; import org.apache.ibatis.builder.xml.XMLConfigBuilder; import org.apache.ibatis.parsing.XNode; import java.io.Reader; /** * 实体类 * 当只需要构建一个对象时，可以使用建造者，如果构建多个对象或者一类对象，则使用工厂方法或者抽象工厂 * * 用静态内部类建造者，实现链式调用（推荐） * * JDK源码样例：{@link StringBuilder#append(int)} 和 {@link StringBuffer#append(int)} 区别在于后者对所有append方法加上同步关键字 * guava源码样例：不可变集合 {@link ImmutableSet.Builder#build()} * Spring源码样例：{@link org.springframework.beans.factory.support.BeanDefinitionBuilder} * mybatis源码样例：{@link org.apache.ibatis.session.SqlSessionFactoryBuilder#build(Reader)} 封装了读XML的操作 * 实际调用的是{@link XMLConfigBuilder#parseConfiguration(XNode)} 解析XML节点，将复杂的XML解析过程一次建造完成 * * * @author F嘉阳 * @date 2018-09-25 17:13 */ public class BuCourse { private String courseName; private String coursePPT; private String courseVideo; /** * question \u0026amp; answer */ private String courseQA; /** * 构造器接受静态内部类 * * @param builder */ public BuCourse(BuCourseBuilder builder) { this.courseName = builder.courseName; this.coursePPT = builder.coursePPT; this.courseVideo = builder.courseVideo; this.courseQA = builder.courseQA; } @Override public String toString() { return \u0026#34;BCourse{\u0026#34; + \u0026#34;courseName=\u0026#39;\u0026#34; + courseName + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, coursePPT=\u0026#39;\u0026#34; + coursePPT + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, courseVideo=\u0026#39;\u0026#34; + courseVideo + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, courseQA=\u0026#39;\u0026#34; + courseQA + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#39;}\u0026#39;; } public static class BuCourseBuilder { private String courseName; private String coursePPT; private String courseVideo; private String courseQA; public BuCourseBuilder courseName(String courseName) { this.courseName = courseName; return this; } public BuCourseBuilder coursePPT(String coursePPT) { this.coursePPT = coursePPT; return this; } public BuCourseBuilder courseVideo(String courseVideo) { this.courseVideo = courseVideo; return this; } public BuCourseBuilder courseQA(String courseQA) { this.courseQA = courseQA; return this; } /** * 传递静态内部类返回目标对象 * * @return */ public BuCourse build() { return new BuCourse(this); } } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.builder; import com.google.common.collect.ImmutableSet; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import top.fjy8018.designpattern.pattern.creational.builder.chain.BuCourse; import java.util.Set; @Slf4j class CourseBuilderTest { @Test void coachBuild() { Coach coach = new Coach(new CourseBuilder()); BCourse course = coach.createCourse(\u0026#34;Java课程\u0026#34;, \u0026#34;课程PPT\u0026#34;, \u0026#34;课程视频\u0026#34;, \u0026#34;课程问答\u0026#34;); log.info(course.toString()); } @Test void chainBuild() { BuCourse course = new BuCourse.BuCourseBuilder() .courseName(\u0026#34;Java课程\u0026#34;) .coursePPT(\u0026#34;课程PPT\u0026#34;) .courseVideo(\u0026#34;课程视频\u0026#34;) .courseQA(\u0026#34;课程问答\u0026#34;) .build(); log.info(course.toString()); } /** * 不可变集合源码测试 */ @Test void ImmutableSetBuild() { Set\u0026lt;String\u0026gt; set = ImmutableSet.\u0026lt;String\u0026gt;builder().add(\u0026#34;f\u0026#34;).add(\u0026#34;j\u0026#34;).add(\u0026#34;y\u0026#34;).build(); log.info(set.toString()); } } 单例模式 介绍 定义:保证一个类仅有一个实例,并提供一个全局访问点\n适用场景\n想确保任何情况下都绝对只有一个实例 优点\n在內存里只有一个实例,减少了内存开销 可以避免对资源的多重占用 设置全局访问点,严格控制访问 缺点\n没有接口，扩展困难 重点\n私有构造器 线程安全 延迟加载 序列化和反序列化安全 反射 分类 恶汉单例 懒汉单例 双重校验锁 静态内部类 枚举 容器单例 ThreadLocal单例 代码样例 懒汉单例——线程不安全 package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; /** * 懒汉式单例：延迟加载（线程不安全） * * @author F嘉阳 * @date 2018-09-24 10:29 */ @Slf4j public class LazySingleton { private static LazySingleton lazySingleton = null; /** * 单例模式构造器必须是屏蔽的 */ private LazySingleton() { } /** * 线程不安全 * * @return */ public static LazySingleton getInstance() { if (lazySingleton == null) { log.debug(\u0026#34;LazySingleton实例化\u0026#34;); lazySingleton = new LazySingleton(); } return lazySingleton; } /** * 线程不安全验证 * * @return */ public static LazySingleton getInstance2() throws InterruptedException { if (lazySingleton == null) { // 使用线程等待模拟复杂实例化过程，让多线程同时进入该方法 Thread.sleep(1000); log.debug(\u0026#34;LazySingleton实例化\u0026#34;); lazySingleton = new LazySingleton(); } return lazySingleton; } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; @Slf4j class LazySingletonTest { /** * 懒汉式注重延迟加载，当使用时才实例化 * 可以使用多线程测试方法或者通过多线程Debug方式测试 */ @Test void getInstanceOneThread() { LazySingleton lazySingleton = LazySingleton.getInstance(); } @Test void getInstanceMutilThread() throws InterruptedException { Thread thread1 = new Thread(new MyRunnable()); Thread thread2 = new Thread(new MyRunnable()); thread1.start(); thread2.start(); Thread.sleep(3000); log.debug(\u0026#34;finish\u0026#34;); } private class MyRunnable implements Runnable { @Override public void run() { LazySingleton lazySingleton = null; try { lazySingleton = LazySingleton.getInstance2(); } catch (InterruptedException e) { e.printStackTrace(); } log.info(\u0026#34; \u0026#34; + lazySingleton); } } } 懒汉单例——线程安全 package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; /** * 懒汉式单例：延迟加载（线程安全） * * @author F嘉阳 * @date 2018-09-24 10:29 */ @Slf4j public class LazySingletonSynchronized { private static LazySingletonSynchronized lazySingleton = null; /** * 单例模式构造器必须是屏蔽的 */ private LazySingletonSynchronized() { log.debug(LazySingletonSynchronized.class.getSimpleName() + \u0026#34;构造器实例化\u0026#34;); } /** * synchronized确保线程安全 * synchronized用于静态方法是对class文件加锁，用于普通方法则是对堆内存中的实例方法加锁 * synchronized(LazySingletonSynchronized.class)与在方法名上加锁效果一致 * * @return */ public static LazySingletonSynchronized getInstance() { synchronized (LazySingletonSynchronized.class) { if (lazySingleton == null) { log.debug(\u0026#34;LazySingletonSynchronized实例化\u0026#34;); lazySingleton = new LazySingletonSynchronized(); } } return lazySingleton; } /** * 线程安全验证 * * @return */ public synchronized static LazySingletonSynchronized getInstance2() throws InterruptedException { if (lazySingleton == null) { // 使用线程等待模拟复杂实例化过程，让多线程同时进入该方法 Thread.sleep(1000); log.debug(\u0026#34;LazySingletonSynchronized实例化\u0026#34;); lazySingleton = new LazySingletonSynchronized(); } return lazySingleton; } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; @Slf4j class LazySingletonSynchronizedTest { @Test void getInstance() { LazySingletonSynchronized.getInstance(); } @Test void getInstance2() throws InterruptedException { Thread thread1 = new Thread(new LazySingletonSynchronizedTest.MyRunnable()); Thread thread2 = new Thread(new LazySingletonSynchronizedTest.MyRunnable()); thread1.start(); thread2.start(); Thread.sleep(3000); log.debug(\u0026#34;finish\u0026#34;); } private class MyRunnable implements Runnable { @Override public void run() { LazySingletonSynchronized lazySingleton = null; try { lazySingleton = LazySingletonSynchronized.getInstance2(); } catch (InterruptedException e) { e.printStackTrace(); } log.info(\u0026#34; \u0026#34; + lazySingleton); } } } 懒汉单例——双重校验锁（线程安全） package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; /** * 懒汉式单例：延迟加载（双重校验锁保证线程安全） * * @author F嘉阳 * @date 2018-09-24 10:29 */ @Slf4j public class LazyDoubleCheckSingleton { /** * volatile确保内存可见性，同时禁止指令重排 * volatile会增加一定的汇编代码： * 1. 将当前处理器缓存行的数据写回系统内存 * 2. 写回系统内存后使其他处理器缓存的地址无效，迫使其他处理器从共享内存中同步最新数据 */ private volatile static LazyDoubleCheckSingleton lazySingleton = null; /** * 单例模式构造器必须是屏蔽的 */ private LazyDoubleCheckSingleton() { } /** * 双重校验锁确保线程安全 * 存在指令重排序导致的隐患 * * @return */ public static LazyDoubleCheckSingleton getInstance() { // 首次检查 if (lazySingleton == null) { synchronized (LazyDoubleCheckSingleton.class) { // 再次检查，在此处加锁比对方法加锁开销更小，因为如果首次检查不为空则直接返回 if (lazySingleton == null) { log.debug(\u0026#34;LazyDoubleCheckSingleton实例化\u0026#34;); lazySingleton = new LazyDoubleCheckSingleton(); // 实例化包含三个过程，这三个过程可能由于指令重排序导致执行顺序打乱 // 但Java底层只保证三个过程在单线程内顺序一致 // 1.分配内存 // 2.初始化 // 3.LazyDoubleCheckSingleton对象指向刚分配的内存地址 } } } return lazySingleton; } /** * 线程安全验证 * * @return */ public static LazyDoubleCheckSingleton getInstance2() throws InterruptedException { if (lazySingleton == null) { // 使用线程等待模拟复杂实例化过程，让多线程同时进入该方法 Thread.sleep(1000); synchronized (LazyDoubleCheckSingleton.class) { // 再次检查，在此处加锁比对方法加锁开销更小，因为如果首次检查不为空则直接返回 if (lazySingleton == null) { log.debug(\u0026#34;LazyDoubleCheckSingleton实例化\u0026#34;); lazySingleton = new LazyDoubleCheckSingleton(); } } } return lazySingleton; } } 静态内部类——延迟加载 原理 JVM在类初始化时会加锁，使用初始化锁可以同步多个线程对一个类的初始化，确保了线程安全类初始化时机\n一. 主动引用 虚拟机规范中并没有强制约束何时进行加载，但是规范严格规定了有且只有下列五种情况必须对类进行初始化（加载、验证、准备都会随之发生）：\n遇到 new、getstatic、putstatic、invokestatic 这四条字节码指令时，如果类没有进行过初始化，则必须先触发其初始化。 最常见的生成这 4 条指令的场景是： (1) 使用 new 关键字实例化对象的时候； (2) 读取或赋值一个类的静态字段（被 final 修饰、已在编译期把结果放入常量池的静态字段除外）的时候； (3) 以及调用一个类的静态方法的时候。 (4) 使用一个类的非常量静态成员 (5) 若一个类是顶级类，且该类有嵌套的断言语句（少见）\n使用java.lang.reflect包的方法对类进行反射调用的时候，如果类没有进行初始化，则需要先触发其初始化。\n当初始化一个类的时候，如果发现其父类还没有进行过初始化，则需要先触发其父类的初始化。\n当虚拟机启动时，用户需要指定一个要执行的主类（包含main()方法的那个类），虚拟机会先初始化这个主类；\n当使用JDK 1.7 的动态语言支持时，如果一个java.lang.invoke.MethodHandle 实例最后的解析结果为 REF_getStatic, REF_putStatic, REF_invokeStatic 的方法句柄，并且这个方法句柄所对应的类没有进行过初始化，则需要先触发其初始化；\n二. 被动引用 以上 5 种场景中的行为称为对一个类进行主动引用。除此之外，所有引用类的方式都不会触发初始化，称为被动引用。被动引用的常见例子包括：\n通过子类引用父类的静态字段，不会导致子类初始化。 System.out.println(SubClass.value); // value 字段在 SuperClass 中定义 通过数组定义来引用类，不会触发此类的初始化。该过程会对数组类进行初始化，数组类是一个由虚拟机自动生成的、直接继承自 Object的子类，其中包含了数组的属性和方法。 SuperClass[] sca = new SuperClass[10]; 常量在编译阶段会存入调用类的常量池中，本质上并没有直接引用到定义常量的类，因此不会触发定义常量的类的初始化。 System.out.println(ConstClass.HELLOWORLD); package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; /** * 懒汉单例：静态内部类实现（线程安全） * 使用静态内部类可以屏蔽类实例化的重排序,因为非构造线程不允许看到指令重排序 * * @author F嘉阳 * @date 2018-09-24 13:11 */ @Slf4j public class StaticInnerClassSingleton { /** * 私有构造器是单例必须的 */ private StaticInnerClassSingleton() { log.debug(StaticInnerClassSingleton.class.getSimpleName() + \u0026#34;构造器实例化\u0026#34;); } /** * 开发单例入口 * * @return */ public static StaticInnerClassSingleton getInstance() { log.debug(StaticInnerClassSingleton.class.getSimpleName() + \u0026#34;内部类实例化\u0026#34;); return InnerClass.staticInnerClassSingleton; } /** * 静态内部类 */ private static class InnerClass { private static StaticInnerClassSingleton staticInnerClassSingleton = new StaticInnerClassSingleton(); } } 恶汉单例 package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import java.io.Serializable; /** * 单例模式：饿汉式（类加载时初始化） * 优点：写法简单，类加载时初始化，线程安全 * 缺点：若类不被使用则会造成内存浪费 * * JDK源码样例：{@link Runtime} 其实例在类加载时实例化并通过{@link Runtime#getRuntime()} 获取 * * @author F嘉阳 * @date 2018-09-24 15:38 */ @Slf4j public class HungrySingleton implements Serializable { /** * 类加载时初始化 * 声明final（可选），只有在类加载时初始化才能声明为final，故懒汉式不能声明为final */ private static final HungrySingleton HUNGRYSINGLETON; static { log.debug(HungrySingleton.class.getSimpleName() + \u0026#34;静态块实例化\u0026#34;); HUNGRYSINGLETON = new HungrySingleton(); } /** * 该构造器会在反射攻击时调用 */ private HungrySingleton() { log.debug(HungrySingleton.class.getSimpleName() + \u0026#34;构造器实例化\u0026#34;); } public static HungrySingleton getInstance() { return HUNGRYSINGLETON; } } 恶汉单例——克隆攻击防御 package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import java.io.Serializable; /** * 单例模式：饿汉式 * 克隆攻击改进 * 要么不要实现克隆接口，要么不要调用父类的克隆实现，转而自己实现克隆方法 * * @author F嘉阳 * @date 2018-09-24 15:38 */ @Slf4j public class HungrySingletonCloneableImprove implements Serializable, Cloneable { private static final HungrySingletonCloneableImprove HUNGRYSINGLETON; static { log.debug(HungrySingletonCloneableImprove.class.getSimpleName() + \u0026#34;静态块实例化\u0026#34;); HUNGRYSINGLETON = new HungrySingletonCloneableImprove(); } private HungrySingletonCloneableImprove() { if (HUNGRYSINGLETON != null) { throw new RuntimeException(\u0026#34;单例模式禁止反射调用\u0026#34;); } log.debug(HungrySingletonCloneableImprove.class.getSimpleName() + \u0026#34;构造器实例化\u0026#34;); } public static HungrySingletonCloneableImprove getInstance() { return HUNGRYSINGLETON; } @Override protected Object clone() throws CloneNotSupportedException { return HUNGRYSINGLETON; } /** * 解决序列化攻击问题 * * @return */ private Object readResolve() { log.debug(\u0026#34;序列化获取对象\u0026#34;); return HUNGRYSINGLETON; } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import java.lang.reflect.Method; /** * 单例模式克隆攻击 * * @author F嘉阳 * @date 2018-09-25 22:40 */ @Slf4j class SingletonCloneAttack { /** * 改进前 */ @Test void before() throws Exception { // 正常方式获取单例 HungrySingletonReflectImprove instance = HungrySingletonReflectImprove.getInstance(); Method method = instance.getClass().getDeclaredMethod(\u0026#34;clone\u0026#34;); // 由于原本的访问权限为protect，故要改变访问权限 method.setAccessible(true); // 克隆实例 HungrySingletonReflectImprove instance2 = (HungrySingletonReflectImprove) method.invoke(instance); // 验证 log.info(Integer.toHexString(instance.hashCode())); log.info(Integer.toHexString(instance2.hashCode())); // 返回false，克隆攻击成功 log.info(String.valueOf(instance == instance2)); } /** * 改进后 */ @Test void after() throws Exception { // 正常方式获取单例 HungrySingletonCloneableImprove instance = HungrySingletonCloneableImprove.getInstance(); Method method = instance.getClass().getDeclaredMethod(\u0026#34;clone\u0026#34;); // 由于原本的访问权限为protect，故要改变访问权限 method.setAccessible(true); // 克隆实例 HungrySingletonCloneableImprove instance2 = (HungrySingletonCloneableImprove) method.invoke(instance); // 验证 log.info(Integer.toHexString(instance.hashCode())); log.info(Integer.toHexString(instance2.hashCode())); // 返回false，克隆攻击成功 log.info(String.valueOf(instance == instance2)); } } 单例模式序列化攻击 package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import java.io.*; /** * 单例模式序列化攻击 */ @Slf4j class SingletonSerializableAttackTest { private static final String FILE_NAME = \u0026#34;tmp/singleton_obj.dat\u0026#34;; @BeforeEach void init() { File dir = new File(\u0026#34;tmp\u0026#34;); boolean mkdir = false; if (!dir.exists()) { mkdir = dir.mkdir(); if (mkdir) { log.debug(\u0026#34;文件夹新建成功\u0026#34;); } else { log.error(\u0026#34;文件夹新建失败\u0026#34;); } } File file = new File(FILE_NAME); file.deleteOnExit(); } /** * 枚举类序列化攻击 * 原因：{@link ObjectInputStream#readEnum(boolean)} 中对枚举的执行过程是2000 - 2007行 * 通过枚举的类和名称来获取对象实例，因为枚举的名称在枚举中唯一，所以没有实例化的动作，保证了单例可靠性 * * @throws IOException */ @Test void EnumInstanceGetInstance() throws Exception { EnumInstance singleton = EnumInstance.getInstance(); singleton.setData(new Object()); ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(FILE_NAME)); oos.writeObject(singleton); ObjectInputStream ois = new ObjectInputStream(new FileInputStream(FILE_NAME)); EnumInstance newInstance = (EnumInstance) ois.readObject(); log.info(\u0026#34;写入前：\u0026#34; + singleton.getData()); log.info(\u0026#34;读出后：\u0026#34; + newInstance.getData()); // 此处结果为false，说明是两个不同的对象 log.info(String.valueOf(newInstance.getData() == singleton.getData())); oos.close(); ois.close(); } /** * 通过序列化测试写入和读出后是否属于同一个对象 * 根据单例模式原则，永远只有一个实例化的对象 * * @throws IOException */ @Test void getInstanceBefore() throws IOException, ClassNotFoundException { HungrySingleton hungrySingleton = HungrySingleton.getInstance(); ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(FILE_NAME)); oos.writeObject(hungrySingleton); ObjectInputStream ois = new ObjectInputStream(new FileInputStream(FILE_NAME)); HungrySingleton newInstance = (HungrySingleton) ois.readObject(); log.info(\u0026#34;写入前：\u0026#34; + hungrySingleton); log.info(\u0026#34;读出后：\u0026#34; + newInstance); // 此处结果为false，说明是两个不同的对象 log.info(String.valueOf(newInstance == hungrySingleton)); oos.close(); ois.close(); } @Test void getInstanceAfter() throws IOException, ClassNotFoundException { HungrySingletonSerializableImprove before = HungrySingletonSerializableImprove.getInstance(); ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(FILE_NAME)); oos.writeObject(before); ObjectInputStream ois = new ObjectInputStream(new FileInputStream(FILE_NAME)); HungrySingletonSerializableImprove after = (HungrySingletonSerializableImprove) ois.readObject(); log.info(\u0026#34;写入前：\u0026#34; + before); log.info(\u0026#34;读出后：\u0026#34; + after); // 此处结果为true，说明是同一个对象 log.info(String.valueOf(before == after)); oos.close(); ois.close(); } } 懒汉单例——反射攻击 测试类\npackage top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import java.lang.reflect.Constructor; import java.lang.reflect.Field; import java.lang.reflect.InvocationTargetException; /** * 单例模式反射攻击 */ @Slf4j class SingletonReflectAttack { /** * 修复前懒汉式反射攻击 * 由于反射攻击足够强大，对于非类加载时实例化的单例模式，是无法彻底防止反射攻击的 * * @throws NoSuchMethodException * @throws IllegalAccessException * @throws InvocationTargetException * @throws InstantiationException */ @Test void lazySingletonGetInstanceBefore() throws Exception { Class clazz = LazySingletonSynchronized.class; Constructor constructor = clazz.getDeclaredConstructor(); // 打开访问权限 constructor.setAccessible(true); // 通过反射获取实例 LazySingletonSynchronized reflectInstance = (LazySingletonSynchronized) constructor.newInstance(); // 通过正常方式获取 LazySingletonSynchronized instance = LazySingletonSynchronized.getInstance(); // 判断是否同一个对象 log.info(\u0026#34;反射方式：\u0026#34; + reflectInstance); log.info(\u0026#34;正常方式：\u0026#34; + instance); // 此处结果为false，说明是两个不同的对象 log.info(String.valueOf(reflectInstance == instance)); } } 懒汉单例——线程安全、反射攻击防御1 package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; /** * 懒汉式单例：延迟加载（线程安全） * * @author F嘉阳 * @date 2018-09-24 10:29 */ @Slf4j public class LazySingletonSynchronizedReflectImprove1 { private static LazySingletonSynchronizedReflectImprove1 lazySingleton = null; /** * 单例模式构造器必须是屏蔽的 * 对于非类加载时实例化的单例该方法防止反射攻击无效 * 因为当反射先执行，则会实例化一个或者N个对象 * 当正常方式获取单例时，又会实例化多一个对象 */ private LazySingletonSynchronizedReflectImprove1() { if (lazySingleton != null) { throw new RuntimeException(\u0026#34;单例模式禁止反射调用\u0026#34;); } log.debug(LazySingletonSynchronizedReflectImprove1.class.getSimpleName() + \u0026#34;构造器实例化\u0026#34;); } /** * synchronized确保线程安全 * synchronized用于静态方法是对class文件加锁，用于普通方法则是对堆内存中的实例方法加锁 * synchronized(LazySingletonSynchronized.class)与在方法名上加锁效果一致 * * @return */ public static LazySingletonSynchronizedReflectImprove1 getInstance() { synchronized (LazySingletonSynchronizedReflectImprove1.class) { if (lazySingleton == null) { log.debug(LazySingletonSynchronizedReflectImprove1.class.getSimpleName() + \u0026#34;实例化\u0026#34;); lazySingleton = new LazySingletonSynchronizedReflectImprove1(); } } return lazySingleton; } /** * 线程安全验证 * * @return */ public synchronized static LazySingletonSynchronizedReflectImprove1 getInstance2() throws InterruptedException { if (lazySingleton == null) { // 使用线程等待模拟复杂实例化过程，让多线程同时进入该方法 Thread.sleep(1000); log.debug(LazySingletonSynchronizedReflectImprove1.class.getSimpleName() + \u0026#34;实例化\u0026#34;); lazySingleton = new LazySingletonSynchronizedReflectImprove1(); } return lazySingleton; } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import java.lang.reflect.Constructor; import java.lang.reflect.Field; import java.lang.reflect.InvocationTargetException; /** * 单例模式反射攻击 */ @Slf4j class SingletonReflectAttack { /** * 用构造器防止单例模式被反射攻击 * 失败！ * * @throws NoSuchMethodException * @throws IllegalAccessException * @throws InvocationTargetException * @throws InstantiationException */ @Test void lazySingletonGetInstanceAfter1() throws Exception { Class clazz = LazySingletonSynchronizedReflectImprove1.class; Constructor constructor = clazz.getDeclaredConstructor(); // 打开访问权限 constructor.setAccessible(true); // 通过反射获取实例 LazySingletonSynchronizedReflectImprove1 reflectInstance = (LazySingletonSynchronizedReflectImprove1) constructor.newInstance(); // 通过正常方式获取 LazySingletonSynchronizedReflectImprove1 instance = LazySingletonSynchronizedReflectImprove1.getInstance(); // 判断是否同一个对象 log.info(\u0026#34;反射方式：\u0026#34; + reflectInstance); log.info(\u0026#34;正常方式：\u0026#34; + instance); // 此处结果为false，说明是两个不同的对象 log.info(String.valueOf(reflectInstance == instance)); } } 懒汉单例——线程安全、反射攻击防御2 package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; /** * 懒汉式单例：延迟加载（线程安全） * * @author F嘉阳 * @date 2018-09-24 10:29 */ @Slf4j public class LazySingletonSynchronizedReflectImprove2 { private static LazySingletonSynchronizedReflectImprove2 lazySingleton = null; /** * 是否能实例化标志 * 若能实例化，则为{@code true} */ private static boolean flag = true; /** * 单例模式构造器必须是屏蔽的 */ private LazySingletonSynchronizedReflectImprove2() { if (!flag) { throw new RuntimeException(\u0026#34;单例模式禁止反射调用\u0026#34;); } flag = false; log.debug(LazySingletonSynchronizedReflectImprove2.class.getSimpleName() + \u0026#34;构造器实例化\u0026#34;); } /** * synchronized确保线程安全 * synchronized用于静态方法是对class文件加锁，用于普通方法则是对堆内存中的实例方法加锁 * synchronized(LazySingletonSynchronized.class)与在方法名上加锁效果一致 * * @return */ public static LazySingletonSynchronizedReflectImprove2 getInstance() { synchronized (LazySingletonSynchronizedReflectImprove2.class) { if (lazySingleton == null) { log.debug(LazySingletonSynchronizedReflectImprove2.class.getSimpleName() + \u0026#34;实例化\u0026#34;); lazySingleton = new LazySingletonSynchronizedReflectImprove2(); } } return lazySingleton; } /** * 线程安全验证 * * @return */ public synchronized static LazySingletonSynchronizedReflectImprove2 getInstance2() throws InterruptedException { if (lazySingleton == null) { // 使用线程等待模拟复杂实例化过程，让多线程同时进入该方法 Thread.sleep(1000); log.debug(LazySingletonSynchronizedReflectImprove2.class.getSimpleName() + \u0026#34;实例化\u0026#34;); lazySingleton = new LazySingletonSynchronizedReflectImprove2(); } return lazySingleton; } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import java.lang.reflect.Constructor; import java.lang.reflect.Field; import java.lang.reflect.InvocationTargetException; /** * 单例模式反射攻击 */ @Slf4j class SingletonReflectAttack { /** * 用标识位防止单例模式被反射攻击 * 失败！ * * @throws NoSuchMethodException * @throws IllegalAccessException * @throws InvocationTargetException * @throws InstantiationException */ @Test void lazySingletonGetInstanceAfter2() throws Exception { Class clazz = LazySingletonSynchronizedReflectImprove2.class; Constructor constructor = clazz.getDeclaredConstructor(); // 打开访问权限 constructor.setAccessible(true); // 首先通过正常方式获取 LazySingletonSynchronizedReflectImprove2 instance = LazySingletonSynchronizedReflectImprove2.getInstance(); // 通过反射修改成员变量 Field flag = clazz.getDeclaredField(\u0026#34;flag\u0026#34;); // 修改访问权限 flag.setAccessible(true); flag.set(clazz, true); // 通过反射获取实例 LazySingletonSynchronizedReflectImprove2 reflectInstance = (LazySingletonSynchronizedReflectImprove2) constructor.newInstance(); // 判断是否同一个对象 log.info(\u0026#34;反射方式：\u0026#34; + reflectInstance); log.info(\u0026#34;正常方式：\u0026#34; + instance); // 此处结果为false，说明是两个不同的对象 log.info(String.valueOf(reflectInstance == instance)); } } 恶汉单例——反射攻击防御 package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import java.io.Serializable; /** * 单例模式：饿汉式 * 反射攻击改进 * 对于在类加载时进行初始化的单例，可以用该方法解决反射攻击问题 * * @author F嘉阳 * @date 2018-09-24 15:38 */ @Slf4j public class HungrySingletonReflectImprove implements Serializable, Cloneable { private static final HungrySingletonReflectImprove HUNGRYSINGLETON; static { log.debug(HungrySingletonReflectImprove.class.getSimpleName() + \u0026#34;静态块实例化\u0026#34;); HUNGRYSINGLETON = new HungrySingletonReflectImprove(); } private HungrySingletonReflectImprove() { if (HUNGRYSINGLETON != null) { throw new RuntimeException(\u0026#34;单例模式禁止反射调用\u0026#34;); } log.debug(HungrySingletonReflectImprove.class.getSimpleName() + \u0026#34;构造器实例化\u0026#34;); } @Override protected Object clone() throws CloneNotSupportedException { return super.clone(); } public static HungrySingletonReflectImprove getInstance() { return HUNGRYSINGLETON; } /** * 解决序列化攻击问题 * * @return */ private Object readResolve() { log.debug(\u0026#34;序列化获取对象\u0026#34;); return HUNGRYSINGLETON; } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import java.lang.reflect.Constructor; import java.lang.reflect.Field; import java.lang.reflect.InvocationTargetException; /** 单例模式反射攻击 */ @Slf4j class SingletonReflectAttack { /** * 修复前饿汉式攻击 * * @throws NoSuchMethodException * @throws IllegalAccessException * @throws InvocationTargetException * @throws InstantiationException */ @Test void hungrySingletonGetInstanceBefore() throws Exception { Class clazz = HungrySingletonSerializableImprove.class; Constructor constructor = clazz.getDeclaredConstructor(); // 打开访问权限 constructor.setAccessible(true); // 通过反射获取实例 HungrySingletonSerializableImprove reflectInstance = (HungrySingletonSerializableImprove) constructor.newInstance(); // 通过正常方式获取 HungrySingletonSerializableImprove instance = HungrySingletonSerializableImprove.getInstance(); // 判断是否同一个对象 log.info(\u0026#34;反射方式：\u0026#34; + reflectInstance); log.info(\u0026#34;正常方式：\u0026#34; + instance); // 此处结果为false，说明是两个不同的对象 log.info(String.valueOf(reflectInstance == instance)); } /** * 修复后饿汉式攻击 * * @throws NoSuchMethodException * @throws IllegalAccessException * @throws InvocationTargetException * @throws InstantiationException */ @Test void hungrySingletonGetInstanceAfter() throws Exception { Class clazz = HungrySingletonReflectImprove.class; Constructor constructor = clazz.getDeclaredConstructor(); // 打开访问权限 constructor.setAccessible(true); // 通过反射获取实例，此处抛出异常 HungrySingletonReflectImprove reflectInstance = (HungrySingletonReflectImprove) constructor.newInstance(); // 通过正常方式获取 HungrySingletonReflectImprove instance = HungrySingletonReflectImprove.getInstance(); // 判断是否同一个对象 log.info(\u0026#34;反射方式：\u0026#34; + reflectInstance); log.info(\u0026#34;正常方式：\u0026#34; + instance); log.info(String.valueOf(reflectInstance == instance)); } } 静态内部类——反射攻击防御 package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; /** * 懒汉单例：静态内部类实现（线程安全） * 反射攻击改进 * * @author F嘉阳 * @date 2018-09-24 13:11 */ @Slf4j public class StaticInnerClassSingletonReflectImprove { /** * 私有构造器是单例必须的 */ private StaticInnerClassSingletonReflectImprove() { if (InnerClass.staticInnerClassSingleton != null) { throw new RuntimeException(\u0026#34;单例模式禁止反射调用\u0026#34;); } log.debug(StaticInnerClassSingletonReflectImprove.class.getSimpleName() + \u0026#34;构造器实例化\u0026#34;); } /** * 开发单例入口 * * @return */ public static StaticInnerClassSingletonReflectImprove getInstance() { log.debug(StaticInnerClassSingletonReflectImprove.class.getSimpleName() + \u0026#34;内部类实例化\u0026#34;); return InnerClass.staticInnerClassSingleton; } /** * 静态内部类 * \u0026lt;p\u0026gt; * 原理： * \u0026lt;strong\u0026gt;JVM在类初始化时会加锁，使用初始化锁可以同步多个线程对一个类的初始化，确保了线程安全\u0026lt;/strong\u0026gt; * 类初始化时机 * 一. 主动引用 * 虚拟机规范中并没有强制约束何时进行加载，但是规范严格规定了有且只有下列五种情况必须对类进行初始化（加载、验证、准备都会随之发生）： * 1. 遇到 new、getstatic、putstatic、invokestatic 这四条字节码指令时，如果类没有进行过初始化，则必须先触发其初始化。 * 最常见的生成这 4 条指令的场景是： * (1) 使用 new 关键字实例化对象的时候； * (2) 读取或赋值一个类的静态字段（被 final 修饰、已在编译期把结果放入常量池的静态字段除外）的时候； * (3) 以及调用一个类的静态方法的时候。 * (4) 使用一个类的非常量静态成员 * (5) 若一个类是顶级类，且该类有嵌套的断言语句（少见） * \u0026lt;p\u0026gt; * 2. 使用 {@link java.lang.reflect} 包的方法对类进行反射调用的时候，如果类没有进行初始化，则需要先触发其初始化。 * 3. 当初始化一个类的时候，如果发现其父类还没有进行过初始化，则需要先触发其父类的初始化。 * 4. 当虚拟机启动时，用户需要指定一个要执行的主类（包含 main() 方法的那个类），虚拟机会先初始化这个主类； * 5. 当使用 JDK 1.7 的动态语言支持时，如果一个 {@link java.lang.invoke.MethodHandle} * 实例最后的解析结果为 REF_getStatic, REF_putStatic, REF_invokeStatic 的方法句柄，并且这个方法句柄所对应的类没有进行过初始化，则需要先触发其初始化； * \u0026lt;p\u0026gt; * 二. 被动引用 * 以上 5 种场景中的行为称为对一个类进行主动引用。除此之外，所有引用类的方式都不会触发初始化，称为被动引用。被动引用的常见例子包括： * \u0026lt;p\u0026gt; * 1. 通过子类引用父类的静态字段，不会导致子类初始化。 * System.out.println(SubClass.value); // value 字段在 SuperClass 中定义 * 2. 通过数组定义来引用类，不会触发此类的初始化。该过程会对数组类进行初始化，数组类是一个由虚拟机自动生成的、直接继承自 {@link Object} 的子类，其中包含了数组的属性和方法。 * SuperClass[] sca = new SuperClass[10]; * 3. 常量在编译阶段会存入调用类的常量池中，本质上并没有直接引用到定义常量的类，因此不会触发定义常量的类的初始化。 * System.out.println(ConstClass.HELLOWORLD); */ private static class InnerClass { private static StaticInnerClassSingletonReflectImprove staticInnerClassSingleton = new StaticInnerClassSingletonReflectImprove(); } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import java.lang.reflect.Constructor; import java.lang.reflect.Field; import java.lang.reflect.InvocationTargetException; /** 单例模式反射攻击 */ @Slf4j class SingletonReflectAttack { /** * 修复前懒汉式攻击 * * @throws NoSuchMethodException * @throws IllegalAccessException * @throws InvocationTargetException * @throws InstantiationException */ @Test void staticInnerClassSingletonGetInstanceBefore() throws Exception { Class clazz = StaticInnerClassSingleton.class; Constructor constructor = clazz.getDeclaredConstructor(); // 打开访问权限 constructor.setAccessible(true); // 通过反射获取实例 StaticInnerClassSingleton reflectInstance = (StaticInnerClassSingleton) constructor.newInstance(); // 通过正常方式获取 StaticInnerClassSingleton instance = StaticInnerClassSingleton.getInstance(); // 判断是否同一个对象 log.info(\u0026#34;反射方式：\u0026#34; + reflectInstance); log.info(\u0026#34;正常方式：\u0026#34; + instance); // 此处结果为false，说明是两个不同的对象 log.info(String.valueOf(reflectInstance == instance)); } /** * 修复后懒汉式攻击 * * @throws NoSuchMethodException * @throws IllegalAccessException * @throws InvocationTargetException * @throws InstantiationException */ @Test void staticInnerClassSingletonGetInstanceAfter() throws Exception { Class clazz = StaticInnerClassSingletonReflectImprove.class; Constructor constructor = clazz.getDeclaredConstructor(); // 打开访问权限 constructor.setAccessible(true); // 通过反射获取实例 StaticInnerClassSingletonReflectImprove reflectInstance = (StaticInnerClassSingletonReflectImprove) constructor.newInstance(); // 通过正常方式获取 StaticInnerClassSingletonReflectImprove instance = StaticInnerClassSingletonReflectImprove.getInstance(); // 判断是否同一个对象 log.info(\u0026#34;反射方式：\u0026#34; + reflectInstance); log.info(\u0026#34;正常方式：\u0026#34; + instance); log.info(String.valueOf(reflectInstance == instance)); } } 枚举单例（effective java最佳实践） package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; /** * 枚举单例模式（effective Java推荐） * 枚举类天生可序列化，有效防止序列化攻击 * \u0026lt;p\u0026gt; * 通过反编译可知： * 1. 枚举类的通过final修饰，不能被继承 * 2. 其构造器私有，不能被调用，如果通过反射强制调用，在newInstance方法中会抛出异常 * 3. 枚举变量声明为final和static * 4. 枚举变量通过静态块实例化（类似饿汉式），所以线程安全 * * @author F嘉阳 * @date 2018-09-24 20:30 */ @Slf4j public enum EnumInstance { /** * 单例枚举 */ INSTANCE { @Override protected void logTest() { log.info(EnumInstance.class.getSimpleName() + \u0026#34;打印输出\u0026#34;); } }; public static EnumInstance getInstance() { return INSTANCE; } /** * 目标实例化对象 */ private Object data; public Object getData() { return data; } public void setData(Object data) { this.data = data; } // 提供外部访问方法入口 protected abstract void logTest(); } 测试类\npackage top.fjy8018.designpattern.pattern.creational.singleton; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; class EnumInstanceTest { @Test void logTest() { EnumInstance instance = EnumInstance.getInstance(); instance.logTest(); } } 反射攻击测试\npackage top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import java.lang.reflect.Constructor; import java.lang.reflect.Field; import java.lang.reflect.InvocationTargetException; /** 单例模式反射攻击 */ @Slf4j class SingletonReflectAttack { /** * 枚举单例反射攻击 * 枚举类只有一个构造器 {@link Enum#Enum(String, int)} * 由于{@link Constructor#newInstance(Object...)} 416行指明枚举类不允许反射实例化，强制实例化会抛{@link IllegalArgumentException} * 从底层保证了枚举类无法被反射攻击 * * @throws NoSuchMethodException * @throws IllegalAccessException * @throws InvocationTargetException * @throws InstantiationException */ @Test void enumInstanceGetInstance() throws Exception { Class clazz = EnumInstance.class; // 由于枚举类没有无参构造器，需要指定参数 Constructor constructor = clazz.getDeclaredConstructor(String.class, int.class); // 打开访问权限 constructor.setAccessible(true); // 通过反射获取实例 EnumInstance reflectInstance = (EnumInstance) constructor.newInstance(\u0026#34;fjy\u0026#34;, 123); // 通过正常方式获取 EnumInstance instance = EnumInstance.getInstance(); // 判断是否同一个对象 log.info(\u0026#34;反射方式：\u0026#34; + reflectInstance.getData()); log.info(\u0026#34;正常方式：\u0026#34; + instance.getData()); // 此处结果为false，说明是两个不同的对象 log.info(String.valueOf(reflectInstance.getData() == instance.getData())); } } 容器单例——HashMap package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang3.StringUtils; import org.springframework.beans.factory.config.AbstractFactoryBean; import java.awt.*; import java.util.HashMap; import java.util.Map; /** * 基于容器（集合）实现单例 * 适用于类加载时有多个对象需要实例化场景 * * JDK源码样例：{@link Desktop#getDesktop()} * 其{@link sun.awt.AppContext} 通过{@link HashMap}实现，使用同步保证单例线程安全 * Spring 源码样例：{@link AbstractFactoryBean#getObject()} 在156行判断当前类是否已经实例化 * 若没有实例化则在{@link AbstractFactoryBean#getEarlySingletonInstance()}中通过代理方式实例化对象 * * @author F嘉阳 * @date 2018-09-24 22:31 */ @Slf4j public class ContainerSingleton { private static Map\u0026lt;String, Object\u0026gt; instancesMap = new HashMap\u0026lt;\u0026gt;(); private ContainerSingleton() { } public static void putInstance(String key, Object instance) { if (StringUtils.isNotBlank(key) \u0026amp;\u0026amp; instance != null) { if (!instancesMap.containsKey(key)) { instancesMap.put(key, instance); } } } public static Object getInstance(String key) { return instancesMap.get(key); } /** * 线程安全测试 * * @param key * @param instance */ public static void putInstance2(String key, Object instance) throws InterruptedException { if (StringUtils.isNotBlank(key) \u0026amp;\u0026amp; instance != null) { if (!instancesMap.containsKey(key)) { Thread.sleep(1000); log.debug(\u0026#34;放入实例：\u0026#34; + instance); instancesMap.put(key, instance); } } } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; @Slf4j class ContainerSingletonTest { /** * 直接使用{@link java.util.HashMap} 只会放入一个值，但后面放的值会覆盖首次放入的值 * * @throws InterruptedException */ @Test void getInstanceHashMapUnsafe() throws InterruptedException { Thread thread1 = new Thread(new MyHashMapRunnable()); Thread thread2 = new Thread(new MyHashMapRunnable()); thread1.start(); thread2.start(); Thread.sleep(2000); log.debug(\u0026#34;finish\u0026#34;); } private class MyHashMapRunnable implements Runnable { @Override public void run() { try { ContainerSingleton.putInstance2(\u0026#34;object\u0026#34;, new Object()); // 等待其他线程放入对象后取值 Thread.sleep(100); } catch (InterruptedException e) { e.printStackTrace(); } Object singleton = ContainerSingleton.getInstance(\u0026#34;object\u0026#34;); log.info(\u0026#34; \u0026#34; + singleton); } } private class MyHashtableRunnable implements Runnable { @Override public void run() { try { ContainerSingletonHashtable.putInstance2(\u0026#34;object\u0026#34;, new Object()); // 等待其他线程放入对象后取值 Thread.sleep(100); } catch (InterruptedException e) { e.printStackTrace(); } Object singleton = ContainerSingletonHashtable.getInstance(\u0026#34;object\u0026#34;); log.info(\u0026#34; \u0026#34; + singleton); } } } 容器单例——Hashtable（不推荐） package top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang3.StringUtils; import java.util.Hashtable; import java.util.Map; /** * 基于容器（集合）实现单例（线程不安全） * {@link Hashtable} 实现（不建议） * * @author F嘉阳 * @date 2018-09-24 22:31 */ @Slf4j public class ContainerSingletonHashtable { private static Map\u0026lt;String, Object\u0026gt; instancesMap = new Hashtable\u0026lt;\u0026gt;(); private ContainerSingletonHashtable() { } public static void putInstance(String key, Object instance) { if (StringUtils.isNotBlank(key) \u0026amp;\u0026amp; instance != null) { if (!instancesMap.containsKey(key)) { instancesMap.put(key, instance); } } } public static Object getInstance(String key) { return instancesMap.get(key); } /** * 线程安全测试 * * @param key * @param instance */ public static void putInstance2(String key, Object instance) throws InterruptedException { if (StringUtils.isNotBlank(key) \u0026amp;\u0026amp; instance != null) { if (!instancesMap.containsKey(key)) { Thread.sleep(1000); log.debug(\u0026#34;放入实例：\u0026#34; + instance); instancesMap.put(key, instance); } } } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; @Slf4j class ContainerSingletonTest { /** * 使用{@link java.util.Hashtable} {@link java.util.concurrent.ConcurrentHashMap}可以控制并发，只会放入一个值，但后面放的值会覆盖首次放入的值，效果实际同{@link java.util.HashMap} * 对于Hashtable，在一定程度保证线程安全，但影响了性能 * 对于ConcurrentHashMap，由于使用了静态的ConcurrentHashMap，并直接操作了map，也不是绝对线程安全 * * @throws InterruptedException */ @Test void getInstanceHashtableUnsafe() throws InterruptedException { Thread thread1 = new Thread(new MyHashtableRunnable()); Thread thread2 = new Thread(new MyHashtableRunnable()); thread1.start(); thread2.start(); Thread.sleep(2000); log.debug(\u0026#34;finish\u0026#34;); } private class MyHashMapRunnable implements Runnable { @Override public void run() { try { ContainerSingleton.putInstance2(\u0026#34;object\u0026#34;, new Object()); // 等待其他线程放入对象后取值 Thread.sleep(100); } catch (InterruptedException e) { e.printStackTrace(); } Object singleton = ContainerSingleton.getInstance(\u0026#34;object\u0026#34;); log.info(\u0026#34; \u0026#34; + singleton); } } private class MyHashtableRunnable implements Runnable { @Override public void run() { try { ContainerSingletonHashtable.putInstance2(\u0026#34;object\u0026#34;, new Object()); // 等待其他线程放入对象后取值 Thread.sleep(100); } catch (InterruptedException e) { e.printStackTrace(); } Object singleton = ContainerSingletonHashtable.getInstance(\u0026#34;object\u0026#34;); log.info(\u0026#34; \u0026#34; + singleton); } } } ThreadLocal（伪）单例 package top.fjy8018.designpattern.pattern.creational.singleton; import org.apache.ibatis.executor.ErrorContext; /** * 基于{@link ThreadLocalInstance} 的（伪）单例模式，只能保证线程内单例唯一（空间换时间思想） * * mybatis源码样例：{@link ErrorContext#instance()} 返回通过ThreadLocal实现的单例对象LOCAL * * @author F嘉阳 * @date 2018-09-25 16:24 */ public class ThreadLocalInstance { private static ThreadLocal\u0026lt;ThreadLocalInstance\u0026gt; instanceThreadLocal = new ThreadLocal\u0026lt;ThreadLocalInstance\u0026gt;() { @Override protected ThreadLocalInstance initialValue() { return new ThreadLocalInstance(); } }; private ThreadLocalInstance() { } /** * 其基于{@link java.lang.ThreadLocal.ThreadLocalMap}实现，维护线程隔离，故不用指定key * * @return */ public static ThreadLocalInstance getInstance() { return instanceThreadLocal.get(); } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.singleton; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; @Slf4j class ThreadLocalInstanceTest { /** * {@link ThreadLocal} 只能保证在同一个线程中单例安全 * * @throws InterruptedException */ @Test void getInstance() throws InterruptedException { // 在主线程中取 for (int i = 0; i \u0026lt; 5; i++) { ThreadLocalInstance instance = ThreadLocalInstance.getInstance(); log.info(\u0026#34; \u0026#34; + instance); } Thread thread1 = new Thread(new ThreadLocalInstanceTest.MyRunnable()); Thread thread2 = new Thread(new ThreadLocalInstanceTest.MyRunnable()); thread1.start(); thread2.start(); Thread.sleep(2000); log.debug(\u0026#34;finish\u0026#34;); } private class MyRunnable implements Runnable { @Override public void run() { ThreadLocalInstance instance = null; try { instance = ThreadLocalInstance.getInstance(); } catch (Exception e) { e.printStackTrace(); } log.info(\u0026#34; \u0026#34; + instance); } } } 原型模式 介绍 定义：指原型实例指定创建对象的种类，并且通过拷贝这些原型创建新的对象\n不需要知道任何创建的细节，不调用构造函数\n类型：创建型\n适用场景\n类初始化消耗较多资源 new产生的一个对象需要非常繁琐的过程（数据准备、访问权限等) 构造函数比较复杂 循环体中生产大量对象时 优点\n原型模式性能比直接new一个对象性能高 简化创建过程 缺点\n必须配备克隆方法 对克隆复杂对象或对克隆出的对象进行复杂改造时，容易引入风险 深拷贝、浅拷贝要运用得当 代码样例 改进前 类图\npackage top.fjy8018.designpattern.pattern.creational.prototype.before; import lombok.Getter; import lombok.Setter; import lombok.extern.slf4j.Slf4j; /** * 原型模式：使用原型实例指定待创建对象的类型，并且通过\u0026lt;strong\u0026gt;复制\u0026lt;/strong\u0026gt;这个原型来创建新的对象 * 适用于初始化对象复杂的场景 * \u0026lt;p\u0026gt; * 邮件 * * @author F嘉阳 * @date 2018-09-25 20:39 */ @Slf4j @Getter @Setter public class BMail { private String name; private String emailAddress; private String content; public BMail() throws InterruptedException { // 模拟调用构造器需要复杂过程 Thread.sleep(100); log.debug(this.getClass().getSimpleName() + \u0026#34;构造器调用\u0026#34;); } @Override public String toString() { return \u0026#34;AMail{\u0026#34; + \u0026#34;name=\u0026#39;\u0026#34; + name + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, emailAddress=\u0026#39;\u0026#34; + emailAddress + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, content=\u0026#39;\u0026#34; + content + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#39;}\u0026#39;; } } package top.fjy8018.designpattern.pattern.creational.prototype.before; import lombok.extern.slf4j.Slf4j; import top.fjy8018.designpattern.pattern.creational.prototype.before.BMail; import java.text.MessageFormat; /** * 邮件发送类 * * @author F嘉阳 * @date 2018-09-25 20:42 */ @Slf4j public class BMailUtil { public static void sendEmail(BMail mail) { String content = \u0026#34;向{0}同学,邮件地址:{1},邮件内容:{2}发送邮件成功\u0026#34;; log.info(MessageFormat.format(content, mail.getName(), mail.getEmailAddress(), mail.getContent())); } public static void saveOriginMail(BMail mail) { log.info(\u0026#34;存储Mail记录，originMailContent：{}\u0026#34;, mail.getContent()); } } 改进后 类图\npackage top.fjy8018.designpattern.pattern.creational.prototype.after; import lombok.Getter; import lombok.Setter; import lombok.extern.slf4j.Slf4j; /** * 原型模式：使用原型实例指定待创建对象的类型，并且通过\u0026lt;strong\u0026gt;复制\u0026lt;/strong\u0026gt;这个原型来创建新的对象 * 适用于初始化对象复杂的场景 * \u0026lt;p\u0026gt; * 邮件 * * @author F嘉阳 * @date 2018-09-25 20:39 */ @Slf4j @Getter @Setter public class AMail implements Cloneable { private String name; private String emailAddress; private String content; public AMail() throws InterruptedException { // 模拟调用构造器需要复杂过程 Thread.sleep(100); log.debug(this.getClass().getSimpleName() + \u0026#34;构造器调用\u0026#34;); } /** * 带上父类toString方法查看是否为同一个对象 * * @return */ @Override public String toString() { return \u0026#34;AMail{\u0026#34; + \u0026#34;name=\u0026#39;\u0026#34; + name + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, emailAddress=\u0026#39;\u0026#34; + emailAddress + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, content=\u0026#39;\u0026#34; + content + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#39;}\u0026#39;; } /** * 调用克隆函数不需要调用构造器以此加快实例化速度 * * @return * @throws CloneNotSupportedException */ @Override public Object clone() throws CloneNotSupportedException { return super.clone(); } } package top.fjy8018.designpattern.pattern.creational.prototype.after; import lombok.extern.slf4j.Slf4j; import java.text.MessageFormat; /** * 邮件发送类 * * @author F嘉阳 * @date 2018-09-25 20:42 */ @Slf4j public class AMailUtil { public static void sendEmail(AMail mail) { String content = \u0026#34;向{0}同学,邮件地址:{1},邮件内容:{2}发送邮件成功\u0026#34;; log.info(MessageFormat.format(content, mail.getName(), mail.getEmailAddress(), mail.getContent())); } public static void saveOriginMail(AMail mail) { log.info(\u0026#34;存储Mail记录，originMailContent：{}\u0026#34;, mail.getContent()); } } 测试类\npackage top.fjy8018.designpattern.pattern.creational.prototype.before; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import top.fjy8018.designpattern.pattern.creational.prototype.after.AMail; import top.fjy8018.designpattern.pattern.creational.prototype.after.AMailUtil; import java.util.Date; @Slf4j class BMailTest { private long start; @BeforeEach void init() { start = System.currentTimeMillis(); } @AfterEach void destory() { log.debug(\u0026#34;耗时{}ms\u0026#34;, System.currentTimeMillis() - start); } /** * 每次实例化对象，耗时长——耗时613ms * * @throws Exception */ @Test void before() throws Exception { BMail mail = new BMail(); mail.setContent(\u0026#34;原始正文\u0026#34;); for (int i = 0; i \u0026lt; 5; i++) { // 初始化临时发送邮件对象，防止原始正文被最后一个对象覆盖 BMail tempMail = new BMail(); tempMail.setName(\u0026#34;学员\u0026#34; + i); tempMail.setContent(\u0026#34;正文\u0026#34; + i); tempMail.setEmailAddress(\u0026#34;地址\u0026#34; + i); BMailUtil.sendEmail(tempMail); } // 本意希望在发送完所有邮件后保存开始的正文 BMailUtil.saveOriginMail(mail); } /** * 调用克隆方法（浅拷贝，因为没有对每个属性实现克隆方法），显然速度更快——耗时110ms * * @throws Exception */ @Test void after() throws Exception { AMail mail = new AMail(); mail.setContent(\u0026#34;原始正文\u0026#34;); // 通过hashCode查看是否为同一个对象 log.debug(Integer.toHexString(mail.hashCode())); for (int i = 0; i \u0026lt; 5; i++) { // 初始化临时发送邮件对象，防止原始正文被最后一个对象覆盖 AMail tempMail = (AMail) mail.clone(); tempMail.setName(\u0026#34;学员\u0026#34; + i); tempMail.setContent(\u0026#34;正文\u0026#34; + i); tempMail.setEmailAddress(\u0026#34;地址\u0026#34; + i); AMailUtil.sendEmail(tempMail); log.debug(Integer.toHexString(tempMail.hashCode())); } // 本意希望在发送完所有邮件后保存开始的正文 AMailUtil.saveOriginMail(mail); } } 深克隆实现 改进前 package top.fjy8018.designpattern.pattern.creational.prototype.clone; import lombok.Getter; import lombok.Setter; import lombok.extern.slf4j.Slf4j; import java.util.Date; /** * 原型模式：使用原型实例指定待创建对象的类型，并且通过\u0026lt;strong\u0026gt;复制\u0026lt;/strong\u0026gt;这个原型来创建新的对象 * 适用于初始化对象复杂的场景 * \u0026lt;p\u0026gt; * 邮件 * * @author F嘉阳 * @date 2018-09-25 20:39 */ @Slf4j @Getter @Setter public class CMailBefore implements Cloneable { private String name; private Date date; private String content; public CMailBefore() throws InterruptedException { // 模拟调用构造器需要复杂过程 Thread.sleep(100); log.debug(this.getClass().getSimpleName() + \u0026#34;构造器调用\u0026#34;); } /** * 带上父类toString方法查看是否为同一个对象 * * @return */ @Override public String toString() { return \u0026#34;CMailBefore{\u0026#34; + \u0026#34;name=\u0026#39;\u0026#34; + name + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, date=\u0026#34; + date + \u0026#34;, content=\u0026#39;\u0026#34; + content + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#39;}\u0026#39;; } /** * 调用克隆函数不需要调用构造器以此加快实例化速度 * 通过调用每个属性的克隆方法实现深克隆 * * @return * @throws CloneNotSupportedException */ @Override public Object clone() throws CloneNotSupportedException { return super.clone(); } } 改进后 package top.fjy8018.designpattern.pattern.creational.prototype.clone; import lombok.Getter; import lombok.Setter; import lombok.extern.slf4j.Slf4j; import org.apache.ibatis.cache.CacheKey; import java.util.ArrayList; import java.util.Date; /** * 原型模式：使用原型实例指定待创建对象的类型，并且通过\u0026lt;strong\u0026gt;复制\u0026lt;/strong\u0026gt;这个原型来创建新的对象 * 适用于初始化对象复杂的场景 * \u0026lt;p\u0026gt; * 邮件——深克隆实现 * * JDK源码样例：{@link ArrayList#clone()} 通过元素拷贝实现 * mybatis源码样例：{@link CacheKey#clone()} 通过重新初始化一个对象再进行数组拷贝实现 * * @author F嘉阳 * @date 2018-09-25 20:39 */ @Slf4j @Getter @Setter public class CMailAfter implements Cloneable { private String name; /** * String类型本身是不可变，不需要重新实现深克隆 */ private Date date; private String content; public CMailAfter() throws InterruptedException { // 模拟调用构造器需要复杂过程 Thread.sleep(100); log.debug(this.getClass().getSimpleName() + \u0026#34;构造器调用\u0026#34;); } /** * 带上父类toString方法查看是否为同一个对象 * * @return */ @Override public String toString() { return \u0026#34;CMailAfter{\u0026#34; + \u0026#34;name=\u0026#39;\u0026#34; + name + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, date=\u0026#34; + date + \u0026#34;, content=\u0026#39;\u0026#34; + content + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#39;}\u0026#39;; } /** * 调用克隆函数不需要调用构造器以此加快实例化速度 * 通过调用每个属性的克隆方法实现深克隆 * * @return * @throws CloneNotSupportedException */ @Override public Object clone() throws CloneNotSupportedException { CMailAfter mail = (CMailAfter) super.clone(); // 单独对Date进行克隆 mail.date = (Date) date.clone(); return mail; } } 集合类实现 改进前 package top.fjy8018.designpattern.pattern.creational.prototype.clone; import lombok.Getter; import lombok.Setter; import java.util.Date; import java.util.List; /** * 对集合类原型模式实现 * * @author F嘉阳 * @date 2018-09-25 22:59 */ @Getter @Setter public class ListCloneBefore implements Cloneable { private String name; private List\u0026lt;Date\u0026gt; dateList; @Override public String toString() { return \u0026#34;ListCloneBefore{\u0026#34; + \u0026#34;name=\u0026#39;\u0026#34; + name + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, dateList=\u0026#34; + dateList + \u0026#39;}\u0026#39;; } @Override protected Object clone() throws CloneNotSupportedException { return super.clone(); } } 改进后 package top.fjy8018.designpattern.pattern.creational.prototype.clone; import lombok.Getter; import lombok.Setter; import org.apache.ibatis.cache.CacheKey; import java.util.ArrayList; import java.util.Date; import java.util.List; /** * 对集合类原型模式实现 * * @author F嘉阳 * @date 2018-09-25 22:59 */ @Getter @Setter public class ListCloneAfter implements Cloneable { private String name; private List\u0026lt;Date\u0026gt; dateList; @Override public String toString() { return \u0026#34;ListCloneBefore{\u0026#34; + \u0026#34;name=\u0026#39;\u0026#34; + name + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, dateList=\u0026#34; + dateList + \u0026#39;}\u0026#39;; } /** * 重新实现集合类属性值拷贝方法，模仿{@link CacheKey#clone()} * * @return * @throws CloneNotSupportedException */ @Override protected Object clone() throws CloneNotSupportedException { ListCloneAfter newInstance = (ListCloneAfter) super.clone(); newInstance.setDateList(new ArrayList\u0026lt;\u0026gt;(this.dateList)); return newInstance; } } ","date":"2020-02-28T15:26:43+08:00","permalink":"/archives/238/","title":"设计模式学习——创建型"},{"content":"开闭原则 介绍 定义：一个软件实体如类、模块和函数应该对扩展开放,对修改关闭\n用抽象构建框架,用实现扩展细节\n优点：提高软件系统的可复用性及可维护性\n代码样例 类图\n课程接口\npackage top.fjy8018.designpattern.principle.openclose; import java.math.BigDecimal; /** * 课程接口 * 对于面向接口编程，接口应当是稳定的 * 根据开闭原则，在此体现为对接口的增加和修改时关闭的，因为会导致所有实现类的变化 * 而对课程类的扩展（extend）是开放的 * * @author F嘉阳 * @date 2018-09-18 16:28 */ public interface Course { /** * 获取课程ID * * @return */ Integer getId(); /** * 获取课程名称 * * @return */ String getName(); /** * 获取课程价格 * * @return */ BigDecimal getPrice(); } 课程实现类\npackage top.fjy8018.designpattern.principle.openclose; import java.math.BigDecimal; /** * 课程实现类 * * @author F嘉阳 * @date 2018-09-18 16:30 */ public class JavaCourse implements Course { private Integer id; private String name; private BigDecimal price; public JavaCourse(Integer id, String name, BigDecimal price) { this.id = id; this.name = name; this.price = price; } @Override public Integer getId() { return this.id; } @Override public String getName() { return this.name; } @Override public BigDecimal getPrice() { return this.price; } @Override public String toString() { return \u0026#34;JavaCourse{\u0026#34; + \u0026#34;id=\u0026#34; + id + \u0026#34;, name=\u0026#39;\u0026#34; + name + \u0026#39;\\\u0026#39;\u0026#39; + \u0026#34;, price=\u0026#34; + price.setScale(2, BigDecimal.ROUND_HALF_UP) + \u0026#39;}\u0026#39;; } } Java课程促销类\npackage top.fjy8018.designpattern.principle.openclose; import java.math.BigDecimal; /** * Java课程促销 * 根据开闭原则：对于扩展是开放的，对于修改是关闭的 * 故通过继承方式实现 * * @author F嘉阳 * @date 2018-09-18 16:43 */ public class JavaDiscountCourse extends JavaCourse { public JavaDiscountCourse(Integer id, String name, BigDecimal price) { super(id, name, price); } @Override public Integer getId() { return super.getId(); } @Override public String getName() { return super.getName(); } /** * 实现打折价格逻辑 * * @return */ public BigDecimal getDistcountPrice() { // BigDecimal推荐使用String构造器，防止精度转换问题 return super.getPrice().multiply(new BigDecimal(\u0026#34;0.8\u0026#34;)); } /** * 根据里氏替换原则：父类实现的方法不应该覆盖 * 获取原价 * * @return */ @Override public BigDecimal getPrice() { return super.getPrice(); } @Override public String toString() { return super.toString(); } } 测试类\npackage top.fjy8018.designpattern.principle.openclose; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import java.io.PrintStream; import java.math.BigDecimal; /** * 开闭原则单元测试 * * 由于底层通过加锁实现，故生产环境严禁使用{@link PrintStream#println(String)} */ @Slf4j class JavaCourseTest { /** * 普通情况 */ @Test public void normal() { Course javaCourse = new JavaCourse(1, \u0026#34;Java课程\u0026#34;, new BigDecimal(\u0026#34;99\u0026#34;)); log.info(\u0026#34;课程Id：{},课程名称：{},价格：{}\u0026#34;, javaCourse.getId(), javaCourse.getName(), javaCourse.getPrice()); } /** * 促销打折情况 */ @Test public void discount() { Course course = new JavaDiscountCourse(1, \u0026#34;Java课程\u0026#34;, new BigDecimal(\u0026#34;99\u0026#34;)); JavaDiscountCourse javaCourse = (JavaDiscountCourse) course; log.info(\u0026#34;课程Id：{},课程名称：{},原价：{},折后价格：{}\u0026#34;, javaCourse.getId(), javaCourse.getName(), javaCourse.getPrice(), javaCourse.getDistcountPrice().setScale(2, BigDecimal.ROUND_HALF_UP)); } } 依赖倒置原则 介绍 定义：高层模块不应该依赖低层模块，二者都应该依赖其抽象\n抽象不应该依赖细节；细节应该依赖抽象\n针对接口编程，不要针对实现编程\n优点：可以减少类间的耦合性、提高系统稳定性，提高代码可读性和可维护性，可降低修改程序所造成的风险\n代码样例 改造前 package top.fjy8018.designpattern.principle.dependenceinversion.before; import lombok.extern.slf4j.Slf4j; /** * 低层模块（模块被调用方） * \u0026lt;p\u0026gt; * 此时如果要加一个课程，必须修改该类 * 而对低层模块的修改应该尽量避免 * * @author F嘉阳 * @date 2018-09-18 17:14 */ @Slf4j public class Student { public void studyJavaCourse() { log.info(\u0026#34;学习Java课程\u0026#34;); } public void studyPythonCourse() { log.info(\u0026#34;学习Python课程\u0026#34;); } public void studyVueCourse() { log.info(\u0026#34;学习Vue课程\u0026#34;); } } 测试类\npackage top.fjy8018.designpattern.principle.dependenceinversion.before; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; /** * 依赖倒置实现前单元测试 * 高层模块（模块调用方） */ class StudentTest { /** * 此时存在的问题： * 如果高层模块要学多一门课程 * 则低层模块{@link Student} 也要对应修改，即存在高层模块依赖低层模块，面向具体实现类编程 */ @Test public void study() { Student student = new Student(); student.studyJavaCourse(); student.studyPythonCourse(); student.studyVueCourse(); } } 改造后 类图\n抽象学习课程接口\npackage top.fjy8018.designpattern.principle.dependenceinversion.after; /** * 学习课程接口 * 接口作为抽象存在，成为一个契约 * 面向抽象编程比面向具体实现细节编程实现简单，灵活度高 * * @author F嘉阳 * @date 2018-09-18 17:19 */ public interface DCourse { void study(); } 具体学习过程接口，java课程\npackage top.fjy8018.designpattern.principle.dependenceinversion.after; import lombok.extern.slf4j.Slf4j; /** * 课程都面向接口，满足依赖倒置原则：高层次的模块不应该依赖于低层次的模块，他们都应该依赖于抽象 * * @author F嘉阳 * @date 2018-09-18 17:19 */ @Slf4j public class JavaDCourse implements DCourse { @Override public void study() { log.info(\u0026#34;学习Java课程\u0026#34;); } } Python课程\npackage top.fjy8018.designpattern.principle.dependenceinversion.after; import lombok.extern.slf4j.Slf4j; /** * 所有课程实现类后面都不应该修改，添加新课程则增加新实现类，满足开闭原则 * * @author F嘉阳 * @date 2018-09-18 17:19 */ @Slf4j public class PythonDCourse implements DCourse { @Override public void study() { log.info(\u0026#34;学习Python课程\u0026#34;); } } Vue课程\npackage top.fjy8018.designpattern.principle.dependenceinversion.after; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2018-09-18 17:19 */ @Slf4j public class VueDCourse implements DCourse { @Override public void study() { log.info(\u0026#34;学习Vue课程\u0026#34;); } } 三种注入方式——参数注入 package top.fjy8018.designpattern.principle.dependenceinversion.after; /** * @author F嘉阳 * @date 2018-09-18 17:20 */ public class Student { /** * 具体学习哪个课程，交给高层模块通过参数注入 * * @param course 课程 */ public void studyCourse(DCourse course) { course.study(); } } 测试类\npackage top.fjy8018.designpattern.principle.dependenceinversion.after; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; /** * 依赖倒置接口方法实现后单元测试 * 高层模块 */ class StudentTest { /** * 修改后高层不依赖低层模块{@link Student}的实现 * 此时如果高层模块要学多一门课程 * 则不需要修改Student，直接增加一个实现{@link DCourse} 的接口课程即可 * 实现了高层模块与低层模块的解耦 */ @Test void study() { Student student = new Student(); student.studyCourse(new JavaDCourse()); student.studyCourse(new PythonDCourse()); student.studyCourse(new VueDCourse()); } } 三种注入方式——参数注入 package top.fjy8018.designpattern.principle.dependenceinversion.after; /** * 构造器注入课程 * * @author F嘉阳 * @date 2018-09-18 17:40 */ public class Student2 { private DCourse dCourse; public Student2(DCourse dCourse) { this.dCourse = dCourse; } /** * 具体学习哪个课程，交给高层模块通过构造器注入 */ public void studyCourse() { dCourse.study(); } } 测试类\npackage top.fjy8018.designpattern.principle.dependenceinversion.after; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; /** * 依赖倒置构造器实现实现后单元测试 * 高层模块 */ class Student2Test { @Test void study() { Student2 student = new Student2(new JavaDCourse()); student.studyCourse(); } } 三种注入方式——setter注入 package top.fjy8018.designpattern.principle.dependenceinversion.after; /** * Setter注入课程 * * @author F嘉阳 * @date 2018-09-18 17:40 */ public class Student3 { private DCourse dCourse; public Student3() { } /** * 具体学习哪个课程，交给高层模块通过构造器注入 */ public void studyCourse() { dCourse.study(); } public void setdCourse(DCourse dCourse) { this.dCourse = dCourse; } } 测试类\npackage top.fjy8018.designpattern.principle.dependenceinversion.after; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; /** * 依赖倒置Setter实现实现后单元测试 * 高层模块 */ class Student3Test { @Test void study() { Student3 student = new Student3(); student.setdCourse(new JavaDCourse()); student.studyCourse(); student.setdCourse(new PythonDCourse()); student.studyCourse(); student.setdCourse(new VueDCourse()); student.studyCourse(); } } 单一职责原则 介绍 定义:不要存在多于一个导致类变更的原因\n一个类/接口/方法只负责一项职责\n优点:降低类的复杂度、提高类的可读性提高系统的可维护性、降低变更引起的风险\n代码样例 改造前 类单一职责\npackage top.fjy8018.designpattern.principle.singleresponsibility.before; import lombok.extern.slf4j.Slf4j; /** * 单一职责：一个类应该只有一个发生变化的原因 * * @author F嘉阳 * @date 2018-09-22 10:35 */ @Slf4j public class Bird { /** * 当前类有两个职责：走和飞 * 没有使用单一职责当业务变更必须修改代码，带来风险 * * @param name */ public void mainMoveMode(String name) { if (\u0026#34;鸵鸟\u0026#34;.equals(name)) { log.info(\u0026#34;{}用脚走\u0026#34;, name); } else { log.info(\u0026#34;{}用翅膀飞\u0026#34;, name); } // 当边界值越来越多（承担更多职责）的时候变更变得更加困难 } } 方法单一职责\npackage top.fjy8018.designpattern.principle.singleresponsibility.before; /** * 方法单一职责 * * @author F嘉阳 * @date 2018-09-22 10:57 */ public class Method { private String name; private Integer age; /** * 此时该方法承担两个职责：更新名字和年龄 * * @param name * @param age */ public void updateUserInfo(String name, Integer age) { this.name = name; this.age = age; } /** * 承担多个职责，同时包含可变参数，即不一定更新哪些属性 * * @param name * @param properties */ public void updateProperties(String name, String... properties) { // TODO 更新用户名和属性 } } 接口单一职责\npackage top.fjy8018.designpattern.principle.singleresponsibility.before; /** * 接口单一职责 * 当前接口承担多个职责 * * @author F嘉阳 * @date 2018-09-22 10:43 */ public interface SCourse { /** * 课程信息相关职责 */ String getCourseName(); byte[] getCourseVideo(); /** * 课程管理相关职责 */ void paid(); void studyCourse(); } 改造后 类单一职责\npackage top.fjy8018.designpattern.principle.singleresponsibility.after; import lombok.extern.slf4j.Slf4j; /** * 当前类只承担一个职责：飞 * * @author F嘉阳 * @date 2018-09-22 10:40 */ @Slf4j public class FlyBird { public void mainMoveMode(String name) { log.info(\u0026#34;{}用翅膀飞\u0026#34;, name); } } package top.fjy8018.designpattern.principle.singleresponsibility.after; import lombok.extern.slf4j.Slf4j; /** * 当前类只承担一个职责：走 * * @author F嘉阳 * @date 2018-09-22 10:40 */ @Slf4j public class WalkBird { public void mainMoveMode(String name) { log.info(\u0026#34;{}用脚走\u0026#34;, name); } } 方法单一职责\npackage top.fjy8018.designpattern.principle.singleresponsibility.after; /** * 方法单一职责 * * @author F嘉阳 * @date 2018-09-22 10:57 */ public class Method { private String name; private Integer age; /** * 此时该方法承担一个职责：更新名字 * * @param name */ public void updateUserName(String name) { this.name = name; } /** * 只更新年龄 * * @param age */ public void updateUserAge(Integer age) { this.age = age; } /** * 对于有布尔值的方法也建议拆分 * * @param name * @param bool */ public void updateProperties(String name, boolean bool) { if (bool) { // TODO } else { // TODO } this.name = name; } } 接口单一职责\n类图\n课程管理相关\npackage top.fjy8018.designpattern.principle.singleresponsibility.after; /** * 只承担课程管理相关职责 * * @author F嘉阳 * @date 2018-09-22 10:46 */ public interface CourseManagement { void paid(); void studyCourse(); } 课程信息相关\npackage top.fjy8018.designpattern.principle.singleresponsibility.after; /** * 一个接口变更，只对一个实现类有影响，对其他接口无关 * 只承担课程信息相关职责 * * @author F嘉阳 * @date 2018-09-22 10:45 */ public interface CourseInfo { String getCourseName(); byte[] getCourseVideo(); } 课程实现类\npackage top.fjy8018.designpattern.principle.singleresponsibility.after; import lombok.extern.slf4j.Slf4j; /** * 课程实现类 * 接口实现单一职责后可以通过实现类多重实现的方式组合 * * @author F嘉阳 * @date 2018-09-22 10:46 */ @Slf4j public class CourseImpl implements CourseInfo, CourseManagement { @Override public String getCourseName() { return \u0026#34;设计模式课程\u0026#34;; } @Override public byte[] getCourseVideo() { return new byte[0]; } @Override public void paid() { log.info(\u0026#34;支付成功\u0026#34;); } @Override public void studyCourse() { log.info(\u0026#34;学习课程\u0026#34;); } } ​\n测试类\npackage top.fjy8018.designpattern.principle.singleresponsibility.before; import org.junit.jupiter.api.Test; import top.fjy8018.designpattern.principle.singleresponsibility.after.FlyBird; import top.fjy8018.designpattern.principle.singleresponsibility.after.WalkBird; import static org.junit.jupiter.api.Assertions.*; class BirdTest { /** * 没有使用单一职责情况 */ @Test void mainMoveModeBefore() { Bird bird = new Bird(); bird.mainMoveMode(\u0026#34;大雁\u0026#34;); // 业务变更 bird.mainMoveMode(\u0026#34;鸵鸟\u0026#34;); } /** * 使用单一职责后 * 不同职责调用交由应用层负责 */ @Test void mainMoveModeAfter() { FlyBird flyBird = new FlyBird(); flyBird.mainMoveMode(\u0026#34;大雁\u0026#34;); WalkBird walkBird = new WalkBird(); walkBird.mainMoveMode(\u0026#34;鸵鸟\u0026#34;); } } 介绍 定义:用多个专门的接口,而不使用单一的总接口, 客户端不应该依赖它不需要的接口\n一个类对一个类的依赖应该建立在最小的接口上 建立单一接口,不要建立庞大臃肿的接口\n尽量细化接口,接口中的方法尽量少\n注意适度原则,一定要适度\n优点:符合我们常说的高内聚低耦合的设计思想，从而使得类具有很好的可读性、可扩展性和可维护性。\n代码样例 改造前 类图\n接口\npackage top.fjy8018.designpattern.principle.interfacesegregation.before; /** * 接口隔离原则：客户端不应该依赖它不需要的接口；一个类对另一个类的依赖应该建立在最小的接口上 * 旧版：一个接口承担过多功能，使得实现类对有些不需要的方法也必须实现 * * @author F嘉阳 * @date 2018-09-22 11:40 */ public interface IAnimalAction { void eat(); void fly(); void swim(); } 实现类\npackage top.fjy8018.designpattern.principle.interfacesegregation.before; /** * @author F嘉阳 * @date 2018-09-22 11:41 */ public class Bird implements IAnimalAction { @Override public void eat() { } @Override public void fly() { } @Override public void swim() { // 不需要的接口，但被迫实现 } } package top.fjy8018.designpattern.principle.interfacesegregation.before; /** * @author F嘉阳 * @date 2018-09-22 11:44 */ public class Dog implements IAnimalAction { @Override public void eat() { } @Override public void fly() { // 不需要的接口，但被迫实现 } @Override public void swim() { } } 改造后 类图\n接口\npackage top.fjy8018.designpattern.principle.interfacesegregation.after; /** * 隔离接口方法，但注意适度，不是越少越好 * * @author F嘉阳 * @date 2018-09-22 11:42 */ public interface IEatAction { void eat(); } package top.fjy8018.designpattern.principle.interfacesegregation.after; /** * @author F嘉阳 * @date 2018-09-22 11:43 */ public interface IFlyAction { void fly(); } package top.fjy8018.designpattern.principle.interfacesegregation.after; /** * @author F嘉阳 * @date 2018-09-22 11:43 */ public interface ISwimAction { void swim(); } 实现类\npackage top.fjy8018.designpattern.principle.interfacesegregation.after; /** * 实现类可以实现需要的接口 * * @author F嘉阳 * @date 2018-09-22 11:43 */ public class Bird implements IFlyAction, IEatAction { @Override public void eat() { } @Override public void fly() { } } package top.fjy8018.designpattern.principle.interfacesegregation.after; /** * 接口隔离注重接口方法之间的关系 * 单一职责关心类、方法、接口承担的职责，即一个接口只要职责单一就不关系方法之间的关系 * * @author F嘉阳 * @date 2018-09-22 11:44 */ public class Dog implements IEatAction, ISwimAction { @Override public void eat() { } @Override public void swim() { } } 迪米特原则 介绍 定义:一个对象应该对其他对象保持最少的了解。又叫最少知道原则\n尽量降低类与类之间的耦合\n优点:降低类之间的耦合\n特点：\n强调只和朋友交流,不和陌生人说话\n朋友:\n出现在成员变量、方法的输入、输出参数中的类称为成员朋友类，而出现在方法体内部的类不属于朋友类。 代码样例 类图\npackage top.fjy8018.designpattern.principle.demeter; import lombok.extern.slf4j.Slf4j; import java.util.List; /** * 迪米特法则（Law of Demeter）又叫作最少知识原则（Least Knowledge Principle 简写LKP）， * 就是说一个对象应当对其他对象有尽可能少的了解 * 和朋友交流，不和陌生人说话 * 朋友指的是出现在类成员变量、方法参数、返回值的对象 * * @author F嘉阳 * @date 2018-09-22 17:02 */ @Slf4j public class Boss { /** * 此处朋友为{@link TeamLeader},{@link DCourse}不是类成员变量、方法参数、返回值的对象，所以不应该在类内引用 * 即当前类知道的应该尽可能少 * * @param teamLeader */ public void commandCheckNumber(TeamLeader teamLeader) { List courses = teamLeader.checkNumberOfCourses(); log.info(\u0026#34;课程的数量是：{}\u0026#34;, courses.size()); } } package top.fjy8018.designpattern.principle.demeter; /** * @author F嘉阳 * @date 2018-09-22 17:04 */ public class DCourse { } package top.fjy8018.designpattern.principle.demeter; import java.util.ArrayList; import java.util.List; /** * @author F嘉阳 * @date 2018-09-22 17:04 */ public class TeamLeader { /** * {@link DCourse}作为方法返回值，属于“朋友” * * @return */ public List\u0026lt;DCourse\u0026gt; checkNumberOfCourses() { List\u0026lt;DCourse\u0026gt; courseList = new ArrayList\u0026lt;DCourse\u0026gt;(); for (int i = 0; i \u0026lt; 20; i++) { courseList.add(new DCourse()); } return courseList; } } 测试类\npackage top.fjy8018.designpattern.principle.demeter; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; class BossTest { @Test void commandCheckNumber() { Boss boss = new Boss(); boss.commandCheckNumber(new TeamLeader()); } } 里氏替换原则 介绍 定义: 如果对每一个类型为T1的对象o1,都有类型为T2的对象o2,使得以T1定义的所有程序P在所有的对象o1都替换成2时, 程序P的行为没有发生变化,那么类型T2是类型T1的子类型。\n定义扩展: 一个软件实体如果适用一个父类的话那一定适用于其子类,所有引用父类的地方必须能透明地使用其子类的对象,子类对象能够 替换父类对象,而程序逻辑不变。\n引申意义:子类可以扩展父类的功能,但不能改变父类原有的功能。\n含义1:子类可以实现父类的抽象方法但不能覆盖父类的非抽象方法。\n含义2:子类中可以增加自己特有的方法。\n代码样例——类级别 改造前\n类图\npackage top.fjy8018.designpattern.principle.liskovsubstitution.classdemo.before; /** * 里氏替换原则：如果对每一个类型为S的对象o1，都有类型为T的对象o2，使得以T定义的所有程序P在所有的对象o1都代换为o2，程序P的行为没有发生变化，那么类型S是类型T的子类型 * 即：所有使用父类对象的地方替换成子类对象依然正常运行 * * @author F嘉阳 * @date 2018-09-22 20:01 */ public class Rectangle { private long length; private long width; public long getLength() { return length; } public void setLength(long length) { this.length = length; } public long getWidth() { return width; } public void setWidth(long width) { this.width = width; } } package top.fjy8018.designpattern.principle.liskovsubstitution.classdemo.before; /** * @author F嘉阳 * @date 2018-09-22 20:03 */ public class Square extends Rectangle { private long sideLength; @Override public long getLength() { return sideLength; } /** * 覆盖了父类的实现，不满足里氏替换原则 * * @param length */ @Override public void setLength(long length) { this.sideLength = length; } @Override public long getWidth() { return sideLength; } @Override public void setWidth(long width) { this.sideLength = width; } } 改造后\n类图\npackage top.fjy8018.designpattern.principle.liskovsubstitution.classdemo.after; /** * 通过四边形接口连接，防止继承泛滥 * * @author F嘉阳 * @date 2018-09-22 20:11 */ public interface Quadrangle { long getWidth(); long getLength(); } package top.fjy8018.designpattern.principle.liskovsubstitution.classdemo.after; /** * @author F嘉阳 * @date 2018-09-22 20:03 */ public class ASquare implements Quadrangle { private long sideLength; @Override public long getWidth() { return sideLength; } @Override public long getLength() { return sideLength; } public void setLength(long sideLength) { this.sideLength = sideLength; } } package top.fjy8018.designpattern.principle.liskovsubstitution.classdemo.after; /** * 里氏替换原则：如果对每一个类型为S的对象o1，都有类型为T的对象o2，使得以T定义的所有程序P在所有的对象o1都代换为o2，程序P的行为没有发生变化，那么类型S是类型T的子类型 * 即：所有使用父类对象的地方替换成子类对象依然正常运行 * * @author F嘉阳 * @date 2018-09-22 20:01 */ public class ARectangle implements Quadrangle { private long length; private long width; @Override public long getWidth() { return width; } public void setWidth(long width) { this.width = width; } @Override public long getLength() { return length; } public void setLength(long length) { this.length = length; } } 测试类\npackage top.fjy8018.designpattern.principle.liskovsubstitution.classdemo.before; import lombok.extern.slf4j.Slf4j; import org.junit.jupiter.api.Test; import top.fjy8018.designpattern.principle.liskovsubstitution.classdemo.after.ARectangle; import top.fjy8018.designpattern.principle.liskovsubstitution.classdemo.after.Quadrangle; @Slf4j class SquareTest { /** * resize要求传入父类对象，对长宽进行设置 * * @param rectangle */ public static void resize(Rectangle rectangle) { while (rectangle.getWidth() \u0026lt;= rectangle.getLength()) { rectangle.setWidth(rectangle.getWidth() + 1); log.info(\u0026#34;width:{} length:{}\u0026#34;, rectangle.getWidth(), rectangle.getLength()); if (rectangle.getWidth() \u0026gt; 50 || rectangle.getLength() \u0026gt; 50) { log.error(\u0026#34;方法溢出\u0026#34;); break; } } log.info(\u0026#34;resize方法结束 width:{} length:{}\u0026#34;, rectangle.getWidth(), rectangle.getLength()); } public static void resize(Quadrangle quadrangle) { while (quadrangle.getWidth() \u0026lt;= quadrangle.getLength()) { // quadrangle.setWidth(quadrangle.getWidth()+1);报错，原有方法已经不能正确运行，说明原有方法设计存在问题 log.info(\u0026#34;width:{} length:{}\u0026#34;, quadrangle.getWidth(), quadrangle.getLength()); if (quadrangle.getWidth() \u0026gt; 50 || quadrangle.getLength() \u0026gt; 50) { log.error(\u0026#34;方法溢出\u0026#34;); break; } } log.info(\u0026#34;resize方法结束 width:{} length:{}\u0026#34;, quadrangle.getWidth(), quadrangle.getLength()); } @Test void before() { // 是长方形时业务正常运行 Rectangle rectangle = new Rectangle(); rectangle.setWidth(10); rectangle.setLength(20); resize(rectangle); // 使用之类替换父类对象，业务无法运行，方法溢出 Square square = new Square(); square.setLength(40); resize(square); } @Test void after() { // 通过里氏替换原则改造后 ARectangle aRectangle = new ARectangle(); aRectangle.setLength(20); aRectangle.setWidth(10); } } 代码样例——方法级别 改造前\npackage top.fjy8018.designpattern.principle.liskovsubstitution.method.input.before; import lombok.extern.slf4j.Slf4j; import java.util.Map; /** * @author F嘉阳 * @date 2018-09-23 11:29 */ @Slf4j class Base { public void method(Map map) { log.info(\u0026#34;父类被执行\u0026#34;); } } package top.fjy8018.designpattern.principle.liskovsubstitution.method.input.before; import lombok.extern.slf4j.Slf4j; import java.util.HashMap; /** * @author F嘉阳 * @date 2018-09-23 11:30 */ @Slf4j class Child extends Base { public void method(HashMap map) { log.info(\u0026#34;子类HashMap入参方法被执行\u0026#34;); } } 改造后\npackage top.fjy8018.designpattern.principle.liskovsubstitution.method.input.after; import lombok.extern.slf4j.Slf4j; import java.util.HashMap; /** * @author F嘉阳 * @date 2018-09-23 11:29 */ @Slf4j class Base { public void method(HashMap map) { log.info(\u0026#34;父类被执行\u0026#34;); } } package top.fjy8018.designpattern.principle.liskovsubstitution.method.input.after; import lombok.extern.slf4j.Slf4j; import java.util.HashMap; import java.util.Map; /** * 里氏替换原则要求方法入参必须比父类范围更宽，返回值必须比父类更窄 * * @author F嘉阳 * @date 2018-09-23 11:30 */ @Slf4j class Child extends Base { /** * 方法重写，入参必须等于父类或者比父类范围更宽 * 而返回值比父类更窄已经交由IDE判断，若不符合，则编译不通过 * * @param map */ @Override public void method(HashMap map) { log.info(\u0026#34;子类HashMap入参方法被执行\u0026#34;); } /** * 方法重写，是子类额外的方法 * * @param map */ public void method(Map map) { log.info(\u0026#34;子类Map入参方法被执行\u0026#34;); } } 测试类\npackage top.fjy8018.designpattern.principle.liskovsubstitution.method.input.after; import org.junit.jupiter.api.Test; import java.util.HashMap; class ChildTest { @Test void before() { } @Test void after() { Child child = new Child(); // 此时希望调用子类重写的方法，若子类没重写，则应执行父类的方法 child.method(new HashMap()); } } 合成(组合)聚合复用原则 介绍 定义:尽量使用对象组合/聚合,而不是继承关系达到软件复用的目的\n聚合hasA和组合 contains-A\n优点:可以使系统更加灵活,降低类与类之间的耦合度,一个类的变化对其他类造成的影响相对较少\n何时使用合成/聚合、继承\n聚合has-A、组合 contains-A继承is-A\n代码样例 改造前\n类图\npackage top.fjy8018.designpattern.principle.compositionaggregation.before; /** * 合成复用原则：要尽量使用组合，尽量不要使用继承 * * @author F嘉阳 * @date 2018-09-23 15:50 */ public class DBConnection { public String getConnection() { return \u0026#34;MySQL数据库连接\u0026#34;; } } package top.fjy8018.designpattern.principle.compositionaggregation.before; import lombok.extern.slf4j.Slf4j; /** * 直接使用继承，属于聚合关系，业务变更困难 * * @author F嘉阳 * @date 2018-09-23 15:51 */ @Slf4j public class ProductDao extends DBConnection { public void addProduct() { log.info(\u0026#34;使用{}\u0026#34;, super.getConnection()); } } 改造后\n类图\npackage top.fjy8018.designpattern.principle.compositionaggregation.after; /** * 合成复用原则：要尽量使用组合，尽量不要使用继承 * * @author F嘉阳 * @date 2018-09-23 15:50 */ public abstract class ADBConnection { public abstract String getConnection(); } package top.fjy8018.designpattern.principle.compositionaggregation.after; import lombok.extern.slf4j.Slf4j; /** * @author F嘉阳 * @date 2018-09-23 15:51 */ @Slf4j public class AProductDao { private final ADBConnection connection; /** * 通过构造器注入，解除继承关系，同时符合里氏替换原则和开闭原则 * * @param connection */ public AProductDao(ADBConnection connection) { this.connection = connection; } public void addProduct() { log.info(\u0026#34;使用{}\u0026#34;, connection.getConnection()); } } package top.fjy8018.designpattern.principle.compositionaggregation.after; /** * @author F嘉阳 * @date 2018-09-23 15:56 */ public class MysqlConnection extends ADBConnection { @Override public String getConnection() { return \u0026#34;MySQL数据库连接\u0026#34;; } } package top.fjy8018.designpattern.principle.compositionaggregation.after; /** * @author F嘉阳 * @date 2018-09-23 15:57 */ public class OracleConnection extends ADBConnection { @Override public String getConnection() { return \u0026#34;Oracle数据库连接\u0026#34;; } } 测试类\npackage top.fjy8018.designpattern.principle.compositionaggregation.before; import org.junit.jupiter.api.Test; import top.fjy8018.designpattern.principle.compositionaggregation.after.AProductDao; import top.fjy8018.designpattern.principle.compositionaggregation.after.MysqlConnection; import top.fjy8018.designpattern.principle.compositionaggregation.after.OracleConnection; import static org.junit.jupiter.api.Assertions.*; class ProductDaoTest { @Test void before() { ProductDao productDao = new ProductDao(); productDao.addProduct(); } @Test void after() { // 变更交由业务层决定 AProductDao productDao1 = new AProductDao(new MysqlConnection()); productDao1.addProduct(); AProductDao productDao2 = new AProductDao(new OracleConnection()); productDao2.addProduct(); } } ","date":"2020-01-22T09:25:00+08:00","permalink":"/archives/237/","title":"设计模式学习——原则"},{"content":"一、介绍 1. InnoDB写入慢的原因 采用BTREE索引，IO占用高\n2. TokuDB优点 高压缩比（1:12，是InnoDB的14倍）,高写入性能（是InnoDB的10-20倍） 在线创建索引和字段 支持事物（写入性能是InnoDB的9-20倍） 支持主从同步 3. 归档规划 使用TokuDB引擎保存归档数据，拥有高速写入特性 使用双机热备方案搭建归档库，具备高可用性 使用pt-archiver执行数据归档，简便易行 二、 安装TokuDB 安装jemlloc\nyum install -y jemalloc 编辑配置文件\nvi /etc/my.cnf …… [mysqld_safe] malloc-lib=/usr/lib64/libjemalloc.so.1 …… 重启MySQL\nsystemctl restart mysqld 开启Linux大页内存，同时开启内存碎片整理\nlinux系统默认在系统启动时进行内存分配，后续将不再管理内存\necho never \u0026gt; /sys/kernel/mm/transparent_hugepage/enabled echo never \u0026gt; /sys/kernel/mm/transparent_hugepage/defrag 安装TokuDB\nyum install -y Percona-Server-tokudb-57.x86_64 ps-admin --enable -uroot -p 重启数据库再次执行安装\nservice mysqld restart ps-admin --enable -uroot -p 提示引擎已经成功安装\n查看安装结果\nshow engines ; 三、配置Replication集群 集群设计——双向同步Replication集群\n原因：默认情况下Replication模式单向同步，若主库宕机，则向从库写入数据。若主库重新上线，则不会自动同步数据，导致数据不一致\n架构图\nreplication集群同步原理\n配置互为主从的replication集群\n在两个TokuDB数据库上创建用户\nCREATE USER \u0026#39;tokubackup\u0026#39;@\u0026#39;%\u0026#39; IDENTIFIED BY \u0026#39;Abc_123456\u0026#39; ; GRANT super, reload, replication slave ON *.* TO \u0026#39;tokubackup\u0026#39;@\u0026#39;%\u0026#39; ; FLUSH PRIVILEGES ; 修改两个TokuDB的配置文件/etc/my.cnf，如下：\n[mysqld] server_id = 101 log_bin = mysql_bin relay_log = relay_bin …… [mysqld] server_id = 102 log_bin = mysql_bin relay_log = relay_bin 重新启动两个TokuDB节点\nservice mysqld restart 分别在两个TokuDB上执行下面4句SQL\n当前数据库架构，首先让B节点向A节点同步数据\ngraph LR A节点--\u003eB节点; subgraph Master A节点 end subgraph Slave B节点 end 在B节点执行语句\n#关闭同步服务 stop slave; #设置同步的Master节点 change master to master_host=\u0026#34;pxc1\u0026#34;,master_port=3306,master_user=\u0026#34;backup\u0026#34;, master_password=\u0026#34;Abc_123456\u0026#34;; #启动同步服务 start slave; #查看同步状态 show slave status; 执行结果\n当如下两个字段为yes，则说明主从同步已经开启\n在A节点启动主从同步\n#关闭同步服务 stop slave; #设置同步的Master节点 change master to master_host=\u0026#34;pxc8\u0026#34;,master_port=3306,master_user=\u0026#34;backup\u0026#34;, master_password=\u0026#34;Abc_123456\u0026#34;; #启动同步服务 start slave; #查看同步状态 show slave status; 执行结果 ![image-20191230111537571](https://gitea.fjy8018.top/fjy8018/images/raw/branch/blog/img/image-20191230111537571.png) ## 四、创建归档表 通常按年份定义归档表分表名称，选择任意一个节点执行即可，此处选择在PXC1节点上执行 ```mysql CREATE DATABASE test; USE test; CREATE TABLE t_purchase_20191230 ( id INT UNSIGNED PRIMARY KEY, purchase_price DECIMAL(10,2) NOT NULL, purchase_num INT UNSIGNED NOT NULL, purchase_sum DECIMAL (10,2) NOT NULL, purchase_buyer INT UNSIGNED NOT NULL, purchase_date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, company_id INT UNSIGNED NOT NULL, goods_id INT UNSIGNED NOT NULL, KEY idx_company_id(company_id), KEY idx_goods_id(goods_id) )engine=TokuDB; 在PXC8节点上也看到了新增的数据表\n五、配置Haproxy+Keepalived双机热备 在两个节点上安装Haproxy，此处使用之前搭建的两个节点Haproxy\nyum install -y haproxy 修改配置文件\nvi /etc/haproxy/haproxy.cfg global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 listen admin_stats bind 0.0.0.0:4001 mode http stats uri /dbs stats realm Global\\ statistics stats auth admin:Abc%6870 listen proxy_mysql bind 0.0.0.0:13306 mode tcp balance roundrobin option tcplog server mycat_1 192.168.1.33:8066 check port 8066 weight 1 maxconn 2000 server mycat_2 192.168.1.36:8066 check port 8066 weight 1 maxconn 2000 option tcpka # 新增配置 listen proxy_mysql_backup bind 0.0.0.0:13307 mode tcp balance roundrobin option tcplog #日志格式 server backup_1 pxc1:3306 check port 3306 maxconn 2000 server backup_2 pxc8:3306 check port 3306 maxconn 2000 option tcpka #使用keepalive检测死链 重启Haproxy\n开启防火墙的VRRP协议\nfirewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --protocol vrrp -j ACCEPT firewall-cmd --reload 查看配置结果\n在两个节点上安装Keepalived，此处复用之前的Keepalived\n六、准备归档数据 在两个PXC分片上创建进货表，默认是InnoDB引擎\nCREATE TABLE t_purchase ( id INT UNSIGNED PRIMARY KEY, purchase_price DECIMAL(10,2) NOT NULL, purchase_num INT UNSIGNED NOT NULL, purchase_sum DECIMAL (10,2) NOT NULL, purchase_buyer INT UNSIGNED NOT NULL, purchase_date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, company_id INT UNSIGNED NOT NULL, goods_id INT UNSIGNED NOT NULL, KEY idx_company_id(company_id), KEY idx_goods_id(goods_id) ) 配置MyCat的schema.xml文件，并重启MyCat\n\u0026lt;table name=\u0026#34;t_purchase\u0026#34; dataNode=\u0026#34;dn1,dn2\u0026#34; rule=\u0026#34;mod-long\u0026#34; /\u0026gt; 使用Java程序导入10万条测试数据，由mycat路由到两个分片中\n测试数据的时间为2019年11月21日\n七、执行数据归档 pt-archiver用途：\n导出线上数据,到线下数据作处理\n清理过期数据,并把数据归档到本地归档表中,或者远端归档服务器\n安装pt-archiver\nyum install -y percona-toolkit 查看版本\n$ pt-archiver --version pt-archiver 3.1.0 执行数据归档\n--bulk-delete --bulk-insert表示对源库批量删除，对归档库批量导入，该归档过程包含事务，故不会导致数据丢失\npt-archiver --source h=192.168.1.9,P=13306,u=admin,p=\u0026#39;password\u0026#39;,D=test,t=t_purchase --dest h=192.168.1.9,P=13307,u=admin,p=\u0026#39;password\u0026#39;,D=test,t=t_purchase_20191230 --no-check-charset --where \u0026#39;purchase_date\u0026lt;\u0026#34;2019-11-30 00:00:00\u0026#34;\u0026#39; --progress 5000 --bulk-delete --bulk-insert --limit=10000 --statistics 迁移结果\n查询源库，记录已全部迁移\n查询归档库\n","date":"2019-12-30T08:50:00+08:00","image":"/images/recovered/de47efcb191c722de4a9.png","permalink":"/archives/236/","title":"PXC系列学习笔记（七）——大数据归档——冷热数据分离"},{"content":"一、介绍 通用导入方法：数据量大应当使用load data方式导入，而不是使用source sql文件的方式\n此处为了验证PXC集群性能，需要创造一千万条数据进行SQL语句性能测试\n二、数据生成 若对数据内容无要求，则可使用通用工具生成随机数据，此处面向具体业务场景，故使用Java应用程序生成数据\n1. 创建SpringBoot工程 使用SpringBoot快速创建数据导入工程，引入jdbc和jpa依赖\n... \u0026lt;parent\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-parent\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;2.2.2.RELEASE\u0026lt;/version\u0026gt; \u0026lt;relativePath/\u0026gt; \u0026lt;!-- lookup parent from repository --\u0026gt; \u0026lt;/parent\u0026gt; ... \u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-data-jpa\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;mysql\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;mysql-connector-java\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;runtime\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.projectlombok\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;lombok\u0026lt;/artifactId\u0026gt; \u0026lt;optional\u0026gt;true\u0026lt;/optional\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-test\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;test\u0026lt;/scope\u0026gt; \u0026lt;exclusions\u0026gt; \u0026lt;exclusion\u0026gt; \u0026lt;groupId\u0026gt;org.junit.vintage\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;junit-vintage-engine\u0026lt;/artifactId\u0026gt; \u0026lt;/exclusion\u0026gt; \u0026lt;/exclusions\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; 2. 编写实体类 package top.fjy8018.jdbcbench; /** * @author F嘉阳 * @date 2019-12-28 13:08 */ @Entity(name = \u0026#34;tb_test\u0026#34;) @Data public class DBTest { @Id private Integer id; private String name; } 实现JPA接口\npackage top.fjy8018.jdbcbench; /** * @author F嘉阳 * @date 2019-12-28 13:09 */ public interface DBTestRepository extends JpaRepository\u0026lt;DBTest,Integer\u0026gt; {} 3. 配置JDBC连接 此处有个重要配置，就是rewriteBatchedStatements=true，加上该配置，批量写入数据的性能能提升数百倍。\n参考资料：jdbcTemplate.batchUpdate在批量执行的时候，性能差没有效果，看看怎么解决的。\nspring: datasource: driver-class-name: com.mysql.cj.jdbc.Driver username: admin password: password url: jdbc:mysql://192.168.1.9:13306/test?useSSL=false\u0026amp;useUnicode=true\u0026amp;characterEncoding=UTF-8\u0026amp;rewriteBatchedStatements=true\u0026amp;serverTimezone=Asia/Shanghai jpa: properties: hibernate: jdbc: batch_size: 10000 order_inserts: true generate_statistics: true 4.准备数据库 每个PXC分片只开启一个节点，导入一个节点中则不会产生限流，导入完成后拷贝数据文件到其他节点再启动即可\n修改PXC节点文件，然后重启PXC服务\n## 不等待事务提交先写入硬盘 innodb_flush_log_at_trx_commit = 0 ## 日志数据直接写入磁盘，不进入日志缓冲区 innodb_flush_method = O_DIRECT ## 缓存越大越好 innodb_buffer_pool_size = 200M 创建t_test数据表\nCREATE TABLE tb_test( id INT UNSIGNED PRIMARY KEY, name VARCHAR(200) NOT NULL ); 配置MyCat\n\u0026lt;table name=\u0026#34;t_test\u0026#34; dataNode=\u0026#34;dn1,dn2\u0026#34; rule=\u0026#34;mod-long\u0026#34; /\u0026gt; \u0026lt;?xml version=\u0026#34;1.0\u0026#34;?\u0026gt; \u0026lt;!DOCTYPE mycat:schema SYSTEM \u0026#34;schema.dtd\u0026#34;\u0026gt; \u0026lt;mycat:schema xmlns:mycat=\u0026#34;http://io.mycat/\u0026#34;\u0026gt; \u0026lt;!--配置虚拟逻辑数据表--\u0026gt; \u0026lt;schema name=\u0026#34;test\u0026#34; checkSQLschema=\u0026#34;false\u0026#34; sqlMaxLimit=\u0026#34;100\u0026#34;\u0026gt; \u0026lt;table name=\u0026#34;t_user\u0026#34; dataNode=\u0026#34;dn1,dn2\u0026#34; rule=\u0026#34;mod-long\u0026#34; /\u0026gt; \u0026lt;table name=\u0026#34;tb_test\u0026#34; dataNode=\u0026#34;dn1,dn2\u0026#34; rule=\u0026#34;mod-long\u0026#34; /\u0026gt; \u0026lt;table name=\u0026#34;t_customer\u0026#34; dataNode=\u0026#34;dn1,dn2\u0026#34; rule=\u0026#34;sharding-customer\u0026#34;\u0026gt; \u0026lt;childTable name=\u0026#34;t_orders\u0026#34; primaryKey=\u0026#34;ID\u0026#34; joinKey=\u0026#34;customer_id\u0026#34; parentKey=\u0026#34;id\u0026#34;/\u0026gt; \u0026lt;/table\u0026gt; \u0026lt;/schema\u0026gt; \u0026lt;!--配置分片关系--\u0026gt; \u0026lt;dataNode name=\u0026#34;dn1\u0026#34; dataHost=\u0026#34;cluster1\u0026#34; database=\u0026#34;test\u0026#34; /\u0026gt; \u0026lt;dataNode name=\u0026#34;dn2\u0026#34; dataHost=\u0026#34;cluster2\u0026#34; database=\u0026#34;test\u0026#34; /\u0026gt; \u0026lt;!--配置连接信息--\u0026gt; \u0026lt;dataHost name=\u0026#34;cluster1\u0026#34; maxCon=\u0026#34;1000\u0026#34; minCon=\u0026#34;10\u0026#34; balance=\u0026#34;0\u0026#34; writeType=\u0026#34;0\u0026#34; dbType=\u0026#34;mysql\u0026#34; dbDriver=\u0026#34;native\u0026#34; switchType=\u0026#34;1\u0026#34; slaveThreshold=\u0026#34;100\u0026#34;\u0026gt; \u0026lt;heartbeat\u0026gt;select user()\u0026lt;/heartbeat\u0026gt; \u0026lt;!--多数情况 读多写少--\u0026gt; \u0026lt;writeHost host=\u0026#34;W1\u0026#34; url=\u0026#34;pxc4:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;admin\u0026#34;\u0026gt; \u0026lt;/writeHost\u0026gt; \u0026lt;/dataHost\u0026gt; \u0026lt;dataHost name=\u0026#34;cluster2\u0026#34; maxCon=\u0026#34;1000\u0026#34; minCon=\u0026#34;10\u0026#34; balance=\u0026#34;0\u0026#34; writeType=\u0026#34;0\u0026#34; dbType=\u0026#34;mysql\u0026#34; dbDriver=\u0026#34;native\u0026#34; switchType=\u0026#34;1\u0026#34; slaveThreshold=\u0026#34;100\u0026#34;\u0026gt; \u0026lt;heartbeat\u0026gt;select user()\u0026lt;/heartbeat\u0026gt; \u0026lt;!--多数情况 读多写少--\u0026gt; \u0026lt;writeHost host=\u0026#34;W1\u0026#34; url=\u0026#34;pxc7:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;admin\u0026#34;\u0026gt; \u0026lt;/writeHost\u0026gt; \u0026lt;/dataHost\u0026gt; \u0026lt;/mycat:schema\u0026gt; 5.编写单元测试 package top.fjy8018.jdbcbench; @Slf4j @Component class JdbcBenchTest extends JdbcBenchApplicationTests{ @Autowired private DBTestRepository repository; @Autowired private JdbcTemplate jdbcTemplate; @Test public void insertTest(){ DBTest t = new DBTest(); t.setId(1); t.setName(1+\u0026#34;，测试实体\u0026#34;); DBTest save = repository.save(t); Assert.notNull(save,\u0026#34;保存测试失败\u0026#34;); } @Test public void insertData() throws Exception { for (int i = 1; i \u0026lt;= 10000000;) { List\u0026lt;DBTest\u0026gt; tests = new LinkedList\u0026lt;\u0026gt;(); for (int j = 1; j \u0026lt;= 50000; j++) { DBTest t = new DBTest(); t.setId(j+i); t.setName(j+i+\u0026#34;，测试实体\u0026#34;); tests.add(t); } i = i+50000; log.info(\u0026#34;插入{}条数据\u0026#34;,i); batchInsert(tests); tests.clear(); } } @Transactional public void batchInsert(List\u0026lt;DBTest\u0026gt; dbTestList) { String sql = \u0026#34;insert into tb_test(id,name) values(?,?)\u0026#34;; log.info(\u0026#34;批次大小：{}\u0026#34;,dbTestList.size()); jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() { public void setValues(PreparedStatement ps, int i) throws SQLException { String name = dbTestList.get(i).getName(); int id = dbTestList.get(i).getId(); ps.setInt(1, id); ps.setString(2, name); } public int getBatchSize() { return dbTestList.size(); } }); } } 执行单元测试，大约1小时写入完成，不同服务器配置写入性能差距明显，应当以实际为准\n三、数据导入后的操作 关闭PXC节点(还原配置文件) 拷贝数据文件到其他PXC节点 关闭 MyCat(还原配置文件) 启动PC和 MyCat 四、大数据性能测试——limit关键字 SELECT id, name FROM tb_test LIMIT 100, 100; SELECT id, name FROM tb_test LIMIT 10000, 100; SELECT id, name FROM tb_test LIMIT 1000000, 100; SELECT id, name From tb_test LIMIT 5000000, 100; 现象：执行最后一条sql语句时速度慢，且cpu和内存占用率极大，主要对mycat节点内存占用一直居高不下\n分析索引使用情况，rows字段显示其操作了4746302条数据，走全表扫描\nid select_type table partitions type possible_keys key key_len ref rows filtered Extra 1 SIMPLE tb_test ALL 4746302 100 原因：\n全表扫描,速度极慢 limit语句的查询时间与起始记录的位置成正比 mysql limit的语句是很方便,但是对记录很多的表并不适合直接使用 ","date":"2019-12-30T08:49:04+08:00","image":"/images/recovered/de47efcb191c722de4a9.png","permalink":"/archives/235/","title":"PXC系列学习笔记（六）——Java千万数据量导入和查询优化 "},{"content":"一、介绍 为了检测当前搭建的PXC集群性能，需要执行基准测试，看是否符合性能指标，并对sql进行针对性优化\n二、安装Sysbench 选择集群以外的节点安装测试工具，保证测试的准确性，并且能与集群在网络上互通。\n本次选择的机器配置如下，且位于同一内网网段，网络影响因素几乎忽略不计\n名称 IP CPU 内存 磁盘 Sysbench 192.168.1.24 2 12G 20G 在线安装\ncurl -s https://packagecloud.io/install/repositories/akopytov/sysbench/script.rpm.sh | sudo bash yum -y install sysbench 本地安装\n下载压缩文件\nhttps://codeload.github.com/akopytov/sysbench/zip/1.0\n安装依赖包\nyum install -y automake libtool yum install -y mysql-devel 执行安装\n#cd sysbench ./autogen.sh ./configure make make install sysbench --version 执行测试 参数介绍\n参数名称 功能意义 -oltp-test-mode 执行模式(simple只测试查询、nontrx无事务、complex有事务) -oltp-tables-count 测试表的数量 -oltp-table-size 测试表的记录数 -threads 并发连接数 -time 测试执行时间（秒） -report-interval 生成报告单的间隔时间（秒） COMMAND 命令\n参数名称 功能意义 prepare 准备测试数据 run 执行测试 cleanUD 清除测试数据 准备测试库，手动连接到pxc集群，创建名为sbtest的测试库\nsysbench /usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --mysql-host=192.168.1.9 --mysql-port=3306 --mysql-user=admin --mysql-password=Abc_123456 --oltp-tables-count=10 --oltp-table-size=100000 prepare 执行测试\nsysbench /usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --mysql-host=192.168.1.9 --mysql-port=3306 --mysql-user=admin --mysql-password=Abc_123456 --oltp-test-mode=complex --threads=10 --time=300 --report-interval=10 run \u0026gt;\u0026gt; /home/mysysbench.log 5分钟压测结果\nSQL statistics: queries performed: read: 1122604 write: 307573 other: 160792 total: 1590969 transactions: 72545 (241.79 per sec.) queries: 1590969 (5302.59 per sec.) ignored errors: 7641 (25.47 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 300.0339s total number of events: 72545 Latency (ms): min: 10.26 avg: 41.35 max: 2246.45 95th percentile: 78.60 sum: 2999916.73 Threads fairness: events (avg/stddev): 7254.5000/602.52 execution time (avg/stddev): 299.9917/0.01 清理数据\nsysbench /usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --mysql-host=192.168.1.9 --mysql-port=3306 --mysql-user=admin --mysql-password=Abc_123456 --oltp-tables-count=10 cleanup 修改haproxy，跳过mycat，直接对pxc集群进行测试\nlisten proxy_mysql bind 0.0.0.0:13306 mode tcp balance roundrobin option tcplog server pxc_1 192.168.1.32:3306 check port 3306 weight 1 maxconn 2000 server pxc_2 192.168.1.33:3306 check port 3306 weight 1 maxconn 2000 server pxc_3 192.168.1.35:3306 check port 3306 weight 1 maxconn 2000 option tcpka 结果\n推荐测试配置：\n数据表建议不低于10个,单表数据量不低于500万行。如果是配备了SSD或者 PCIE SSD的话,则建议单表数据量最少不低于1亿行，测试时间应当在24小时以上 ","date":"2019-12-30T08:48:05+08:00","image":"/images/recovered/de47efcb191c722de4a9.png","permalink":"/archives/234/","title":"PXC系列学习笔记（五）——Sysbench基准测试 "},{"content":"一、介绍 目前已经搭建好了2套PXC集群，使用了单台mycat进行数据路由，在数据库层实现了高可用，但若mycat宕机，后端数据库依然无法使用，故还需要搭建多一台mycat，结合HAProxy和Keepalived实现负载均衡、双机热备。\n二、集群架构 三、搭建 服务器资源规划如下\n名称 IP CPU 内存 磁盘 服务 pxc3 192.168.1.33 1 2G 40G mycat pxc4 192.168.1.35 1 2G 40G haproxy、Keepalived pxc5 192.168.1.36 1 2G 40G mycat pxc6 192.168.1.37 1 2G 40G haproxy、Keepalived 1. 安装mycat集群 用两个虚拟机实例，各自部署MyCat，采用相同配置\n2. 安装Haproxy集群 用一个虚拟机实例部署Haproxy\n安装Haproxy，需要开放两个端口，3306（TCP转发），4001（监控界面）\n$ yum install -y haproxy 编辑配置文件\n$ vi /etc/haproxy/haproxy.cfg 由于该节点同时也运行了PXC数据库，3306端口已经占用，故使用13306端口\nglobal log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 listen admin_stats bind 0.0.0.0:4001 mode http stats uri /dbs stats realm Global\\ statistics stats auth admin:Abc%6870 listen proxy_mysql bind 0.0.0.0:13306 mode tcp balance roundrobin option tcplog server mycat_1 192.168.1.33:8066 check port 8066 weight 1 maxconn 2000 server mycat_2 192.168.1.36:8066 check port 8066 weight 1 maxconn 2000 option tcpka 启动Haproxy\n$ service haproxy start 访问Haproxy监控画面\nhttp://192.168.1.35:4001/dbs\n用另外一个虚拟机同样按照上述操作安装Haproxy\n3. 安装Keepalived集群 在某个Haproxy虚拟机实例上部署Keepalived\n开启防火墙的VRRP协议\n#开启VRRP firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --protocol vrrp -j ACCEPT # iptable配置 iptables -I INPUT -p vrrp -j ACCEPT #应用设置 firewall-cmd --reload 安装Keepalived\nyum install -y keepalived 编辑配置文件\nvi /etc/keepalived/keepalived.conf vrrp_instance VI_1 { state MASTER # 争抢虚拟IP，若不争抢，则设置为slave interface ens192 virtual_router_id 51 # 唯一ID，范围0-255 priority 100 # 抢占权重 advert_int 1 # 心跳检测间隔时间 authentication { # 心跳检测账户和密码 auth_type PASS auth_pass 123456 } virtual_ipaddress { 192.168.1.9 # 新虚拟IP } } 启动Keepalived\n$ systemctl start keepalived 在两台Keepalived节点上ping虚拟IP，看是否已经创建并可以访问\n$ ping 192.168.1.9 在另外一个Haproxy虚拟机上，按照上述方法部署Keepalived\n使用MySQL客户端连接192.168.1.9，执行增删改查数据\n","date":"2019-12-30T08:46:37+08:00","image":"/images/recovered/de47efcb191c722de4a9.png","permalink":"/archives/233/","title":"PXC系列学习笔记（四）——搭建双机热备的MyCat集群"},{"content":"一. 介绍 由于每个PXC集群有3个节点，故需要使用mycat进行读写分离，分摊单个节点的读写压力。\n同时由于有两套PXC集群，故需要针对不同表采用不同分片算法进行数据路由。\n因为mycat不支持跨分片表连接，还需要配置父子表等，让需要表连接操作的数据路由到同一个分片中。\n二. JDK安装与配置 安装JDK\n#搜索JDK版本 yum search jdk #安装JDK1.8开发版 yum install java-1.8.0-openjdk-devel.x86_64 配置环境变量\n#查看JDK安装路径 ls -lrt /etc/alternatives/java vi /etc/profile #在文件结尾加上JDK路径 JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/ PATH=$JAVA_HOME/bin:$PATH export PATH JAVA_HOME source /etc/profile 三. 创建数据表 创建两套PXC集群，每套3台机器\n在两组PXC集群中分别创建t_user数据表\nCREATE TABLE t_user ( id INT(10) UNSIGNED NOT NULL, username VARCHAR(200) NOT NULL, password VARCHAR(200) NOT NULL, tel CHAR(11) NOT NULL, locked TINYINT(1) UNSIGNED NULL, PRIMARY KEY (ID) USING BTREE, INDEX `idx_username` (`username`) USING BTREE, UNIQUE INDEX `unq_username` (`username`) USING BTREE ) COMMENT \u0026#39;pxc测试表\u0026#39;; 四、MyCat安装与配置 下载MyCat\nhttp://dl.mycat.io/1.6.5/Mycat-server-1.6.5-release-20180122220033-linux.tar.gz\n上传MyCat压缩包到虚拟机\n安装unzip程序包，解压缩MyCat\nyum install unzip unzip MyCAT压缩包名称 开放防火墙8066和9066端口，关闭SELINUX\n修改MyCat的bin目录中所有.sh文件的权限\nchmod -R 777 ./*.sh MyCat启动与关闭\n#cd MyCat的bin目录 ./startup_nowrap.sh #启动MyCat ps -aux #查看系统进程 kill -9 MyCat进程编号 mycat配置文件解析 文件 作用 修改内容 rule.xml 切分算法 修改mod-long分片数量为2 server.xml 虚拟MySQL 修改用户名、密码和逻辑库 scnema．Xml 数据库连接、读写分离、负载均衡、数据表映射 定义连接、读写分离、负载均衡、数据表映射 修改server.xml文件，设置MyCat帐户和虚拟逻辑库\n\u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;UTF-8\u0026#34;?\u0026gt; \u0026lt;!DOCTYPE mycat:server SYSTEM \u0026#34;server.dtd\u0026#34;\u0026gt; \u0026lt;mycat:server xmlns:mycat=\u0026#34;http://io.mycat/\u0026#34;\u0026gt; \u0026lt;system\u0026gt; \u0026lt;property name=\u0026#34;nonePasswordLogin\u0026#34;\u0026gt;0\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;useHandshakeV10\u0026#34;\u0026gt;1\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;useSqlStat\u0026#34;\u0026gt;0\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;useGlobleTableCheck\u0026#34;\u0026gt;0\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;sequnceHandlerType\u0026#34;\u0026gt;2\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;subqueryRelationshipCheck\u0026#34;\u0026gt;false\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;processorBufferPoolType\u0026#34;\u0026gt;0\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;handleDistributedTransactions\u0026#34;\u0026gt;0\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;useOffHeapForMerge\u0026#34;\u0026gt;1\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;memoryPageSize\u0026#34;\u0026gt;64k\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;spillsFileBufferSize\u0026#34;\u0026gt;1k\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;useStreamOutput\u0026#34;\u0026gt;0\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;systemReserveMemorySize\u0026#34;\u0026gt;384m\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;useZKSwitch\u0026#34;\u0026gt;false\u0026lt;/property\u0026gt; \u0026lt;/system\u0026gt; \u0026lt;!--这里是设置的admin用户和虚拟逻辑库--\u0026gt; \u0026lt;user name=\u0026#34;admin\u0026#34; defaultAccount=\u0026#34;true\u0026#34;\u0026gt; \u0026lt;property name=\u0026#34;password\u0026#34;\u0026gt;Abc_123456\u0026lt;/property\u0026gt; \u0026lt;property name=\u0026#34;schemas\u0026#34;\u0026gt;test\u0026lt;/property\u0026gt; \u0026lt;/user\u0026gt; \u0026lt;/mycat:server\u0026gt; 修改schema.xml文件，设置数据库连接和虚拟数据表\n此处有两个分片，则配置两个dataHost\n重点配置介绍 1. balance 属性 负载均衡类型，目前的取值有 3 种：\nbalance=\u0026ldquo;0\u0026rdquo;, 不开启读写分离机制，所有读操作都发送到当前可用的 writeHost 上。 balance=\u0026ldquo;1\u0026rdquo;，全部的 readHost 与 stand by writeHost 参与 select 语句的负载均衡，简单的说，当双 主双从模式(M1-\u0026gt;S1，M2-\u0026gt;S2，并且 M1 与 M2 互为主备)，正常情况下，M2,S1,S2 都参与 select 语句的负载 均衡。 balance=\u0026ldquo;2\u0026rdquo;，所有读操作都随机的在 writeHost、readhost 上分发。防止写节点浪费 balance=\u0026ldquo;3\u0026rdquo;，所有读请求随机的分发到 wiriterHost 对应的 readhost 执行，writerHost 不负担读压力， 注意 balance=3 只在 1.4 及其以后版本有，1.3 没有。 2. writeType 属性 负载均衡类型，目前的取值有 3 种：\nwriteType=\u0026ldquo;0\u0026rdquo;, 所有写操作发送到配置的第一个 writeHost，第一个挂了切到还生存的第二个 writeHost， 重新启动后已切换后的为准，切换记录在配置文件中:dnindex.properties . writeType=\u0026ldquo;1\u0026rdquo;，所有写操作都随机的发送到配置的 writeHost，1.5 以后废弃不推荐。 3. switchType 属性 -1 表示不自动切换。 1 默认值，自动切换。 2 基于 MySQL 主从同步的状态决定是否切换。 \u0026lt;?xml version=\u0026#34;1.0\u0026#34;?\u0026gt; \u0026lt;!DOCTYPE mycat:schema SYSTEM \u0026#34;schema.dtd\u0026#34;\u0026gt; \u0026lt;mycat:schema xmlns:mycat=\u0026#34;http://io.mycat/\u0026#34;\u0026gt; \u0026lt;!--配置虚拟逻辑数据表--\u0026gt; \u0026lt;schema name=\u0026#34;test\u0026#34; checkSQLschema=\u0026#34;false\u0026#34; sqlMaxLimit=\u0026#34;100\u0026#34;\u0026gt; \u0026lt;table name=\u0026#34;t_user\u0026#34; dataNode=\u0026#34;dn1,dn2\u0026#34; rule=\u0026#34;mod-long\u0026#34; /\u0026gt; \u0026lt;/schema\u0026gt; \u0026lt;!--配置分片关系--\u0026gt; \u0026lt;dataNode name=\u0026#34;dn1\u0026#34; dataHost=\u0026#34;cluster1\u0026#34; database=\u0026#34;test\u0026#34; /\u0026gt; \u0026lt;dataNode name=\u0026#34;dn2\u0026#34; dataHost=\u0026#34;cluster2\u0026#34; database=\u0026#34;test\u0026#34; /\u0026gt; \u0026lt;!--配置连接信息--\u0026gt; \u0026lt;dataHost name=\u0026#34;cluster1\u0026#34; maxCon=\u0026#34;1000\u0026#34; minCon=\u0026#34;10\u0026#34; balance=\u0026#34;2\u0026#34; writeType=\u0026#34;0\u0026#34; dbType=\u0026#34;mysql\u0026#34; dbDriver=\u0026#34;native\u0026#34; switchType=\u0026#34;1\u0026#34; slaveThreshold=\u0026#34;100\u0026#34;\u0026gt; \u0026lt;heartbeat\u0026gt;select user()\u0026lt;/heartbeat\u0026gt; \u0026lt;!--多数情况 读多写少--\u0026gt; \u0026lt;writeHost host=\u0026#34;W1\u0026#34; url=\u0026#34;pxc2:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;password\u0026#34;\u0026gt; \u0026lt;readHost host=\u0026#34;W1R1\u0026#34; url=\u0026#34;pxc3:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;password\u0026#34; /\u0026gt; \u0026lt;readHost host=\u0026#34;W1R2\u0026#34; url=\u0026#34;pxc4:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;password\u0026#34; /\u0026gt; \u0026lt;/writeHost\u0026gt; \u0026lt;!--写节点冗余--\u0026gt; \u0026lt;writeHost host=\u0026#34;W2\u0026#34; url=\u0026#34;pxc3:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;password\u0026#34;\u0026gt; \u0026lt;readHost host=\u0026#34;W2R1\u0026#34; url=\u0026#34;pxc2:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;password\u0026#34; /\u0026gt; \u0026lt;readHost host=\u0026#34;W2R2\u0026#34; url=\u0026#34;pxc4:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;password\u0026#34; /\u0026gt; \u0026lt;/writeHost\u0026gt; \u0026lt;/dataHost\u0026gt; \u0026lt;dataHost name=\u0026#34;cluster2\u0026#34; maxCon=\u0026#34;1000\u0026#34; minCon=\u0026#34;10\u0026#34; balance=\u0026#34;2\u0026#34; writeType=\u0026#34;0\u0026#34; dbType=\u0026#34;mysql\u0026#34; dbDriver=\u0026#34;native\u0026#34; switchType=\u0026#34;1\u0026#34; slaveThreshold=\u0026#34;100\u0026#34;\u0026gt; \u0026lt;heartbeat\u0026gt;select user()\u0026lt;/heartbeat\u0026gt; \u0026lt;!--多数情况 读多写少--\u0026gt; \u0026lt;writeHost host=\u0026#34;W1\u0026#34; url=\u0026#34;pxc5:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;password\u0026#34;\u0026gt; \u0026lt;readHost host=\u0026#34;W1R1\u0026#34; url=\u0026#34;pxc6:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;password\u0026#34; /\u0026gt; \u0026lt;readHost host=\u0026#34;W1R2\u0026#34; url=\u0026#34;pxc7:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;password\u0026#34; /\u0026gt; \u0026lt;/writeHost\u0026gt; \u0026lt;!--写节点冗余--\u0026gt; \u0026lt;writeHost host=\u0026#34;W2\u0026#34; url=\u0026#34;pxc6:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;password\u0026#34;\u0026gt; \u0026lt;readHost host=\u0026#34;W2R1\u0026#34; url=\u0026#34;pxc5:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;password\u0026#34; /\u0026gt; \u0026lt;readHost host=\u0026#34;W2R2\u0026#34; url=\u0026#34;pxc7:3306\u0026#34; user=\u0026#34;admin\u0026#34; password=\u0026#34;password\u0026#34; /\u0026gt; \u0026lt;/writeHost\u0026gt; \u0026lt;/dataHost\u0026gt; \u0026lt;/mycat:schema\u0026gt; 修改rule.xml文件，把mod-long的count值修改成2\n\u0026lt;function name=\u0026#34;mod-long\u0026#34; class=\u0026#34;io.mycat.route.function.PartitionByMod\u0026#34;\u0026gt; \u0026lt;property name=\u0026#34;count\u0026#34;\u0026gt;2\u0026lt;/property\u0026gt; \u0026lt;/function\u0026gt; 放行端口\n8066：数据服务\n9066：管理端口\n重启MyCat\n$ ./startup_nowrap.sh 关闭命令：\n$ ps -aux | grep mycat $ kill $PID 向t_user表写入数据，感受数据的切分\nUSE test; select * from t_user; #第一条记录被切分到第二个分片 INSERT INTO t_user(id,username,password,tel,locked) VALUES(1,\u0026#34;A\u0026#34;,HEX(AES_ENCRYPT(\u0026#39;123456\u0026#39;,\u0026#39;HelloWorld\u0026#39;)),\u0026#34;12345678901\u0026#34;,false); #第二条记录被切分到第一个分片 INSERT INTO t_user(id,username,password,tel,locked) VALUES(2,\u0026#34;B\u0026#34;,HEX(AES_ENCRYPT(\u0026#39;123456\u0026#39;,\u0026#39;HelloWorld\u0026#39;)),\u0026#34;12345678912\u0026#34;,false); 五、配置数据分片和父子表 1. 数据切分算法 切分算法 适用场合 备注 主键求模切分 数据增长速度慢,难于增加分片 有明确主键值 枚举值切分 归类存储数据,适合大多数业务 主键范围切分 数据快速增长,容易增加分片 有明确主键值 日期切分 数据快速增长,容易增加分片 （1）主键求模切分 求模切分适合用在初始数据很大,但是数据增长不快的场景\n求模切分的弊病在于扩展新分片难度大,迁移的数据太多\n建议扩张后的分片数是原有分片的2n倍\n（2）枚举值切分 按照某个字段的值（数字）来切分数据 2. 数据分片 在conf目录下创建customer-hash-int文件，内容如下：\n101=0 102=0 103=0 104=1 105=1 106=1 在rule.xml文件中加入自定义和\n\u0026lt;function name=\u0026#34;customer-hash-int\u0026#34; class=\u0026#34;io.mycat.route.function.PartitionByFileMap\u0026#34;\u0026gt; \u0026lt;property name=\u0026#34;mapFile\u0026#34;\u0026gt;customer-hash-int.txt\u0026lt;/property\u0026gt; \u0026lt;/function\u0026gt; \u0026lt;tableRule name=\u0026#34;sharding-customer\u0026#34;\u0026gt; \u0026lt;rule\u0026gt; \u0026lt;columns\u0026gt;sharding_id\u0026lt;/columns\u0026gt; \u0026lt;algorithm\u0026gt;customer-hash-int\u0026lt;/algorithm\u0026gt; \u0026lt;/rule\u0026gt; \u0026lt;/tableRule\u0026gt; 修改schema.xml文件，配置数据分片算法\n\u0026lt;!--配置虚拟逻辑数据表--\u0026gt; \u0026lt;schema name=\u0026#34;test\u0026#34; checkSQLschema=\u0026#34;false\u0026#34; sqlMaxLimit=\u0026#34;100\u0026#34;\u0026gt; \u0026lt;table name=\u0026#34;t_user\u0026#34; dataNode=\u0026#34;dn1,dn2\u0026#34; rule=\u0026#34;mod-long\u0026#34; /\u0026gt; \u0026lt;table name=\u0026#34;t_customer\u0026#34; dataNode=\u0026#34;dn1,dn2\u0026#34; rule=\u0026#34;sharding-customer\u0026#34;/\u0026gt; \u0026lt;/schema\u0026gt; 重载mycat配置\n使用navicat打开命令行界面，执行语句\nreload @@config_all; USE test; CREATE TABLE t_customer( id INT UNSIGNED PRIMARY KEY, username VARCHAR(200) NOT NULL, sharding_id INT NOT NULL ); select * from t_customer; insert into t_customer(id, username, sharding_id) VALUES (1,\u0026#39;tom\u0026#39;,101); insert into t_customer(id, username, sharding_id) VALUES (2,\u0026#39;jetty\u0026#39;,102); insert into t_customer(id, username, sharding_id) VALUES (3,\u0026#39;tomcat\u0026#39;,105); 第一个分片\n第二个分片\n查询mycat数据\n3. 父子表 出现原因：在 MyCat中是不允许跨分片做表连接查询的\n解决原理：部分业务总会可以抽象出父子关系的表。这类表适用于 ER 分片表，子表的记录与 所关联的父表记录存放在同一个数据分片上，避免数据 Join 跨库操作\n修改schema.xml文件，添加父子表定义。此处定义成父表t_customer允许分在不同分片，子表t_orders不允许跨分片。一个父表可以包含多个子表，子表也可嵌套多个子表。 \u0026lt;table name=\u0026#34;t_customer\u0026#34; dataNode=\u0026#34;dn1,dn2\u0026#34; rule=\u0026#34;sharding-customer\u0026#34;\u0026gt; \u0026lt;childTable name=\u0026#34;t_orders\u0026#34; primaryKey=\u0026#34;ID\u0026#34; joinKey=\u0026#34;customer_id\u0026#34; parentKey=\u0026#34;id\u0026#34;/\u0026gt; \u0026lt;/table\u0026gt; 重载mycat配置\n使用navicat打开命令行界面，执行语句\nreload @@config_all; 在MyCat上执行如下SQL：\nUSE test; CREATE TABLE t_orders( id INT PRIMARY KEY, customer_id INT NOT NULL, datetime TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); 向t_customer表和t_orders表写入数据，查看字表数据跟随父表切分到同一个分片\n-- 预期写入第一个分片 insert into t_orders(id, customer_id) VALUES (1,1); insert into t_orders(id, customer_id) VALUES (2,1); insert into t_orders(id, customer_id) VALUES (3,1); -- 预期写入第二个分片 insert into t_orders(id, customer_id) VALUES (4,2); insert into t_orders(id, customer_id) VALUES (5,3); 分片一t_customer查询结果\n分片一t_orders查询结果\n分片二t_customer查询结果\n分片二t_orders查询结果\n同一个客户订单和客户信息都存在相同分片中，结果符合预期\n连接查询测试\nselect c.username,t.id,t.datetime from t_customer c join t_orders t on c.id = t.customer_id ","date":"2019-12-30T08:45:00+08:00","image":"/images/recovered/de47efcb191c722de4a9.png","permalink":"/archives/232/","title":"PXC系列学习笔记（三）——MyCat安装和读写分离配置"},{"content":"1. 介绍 PXC集群具有强一致性特点，用于需要多节点一致性的业务，例如金融系统、交易、支付系统、库存系统等\n本次计划搭建PXC集群使用两个分片，每个分片3个节点组成一个PXC集群\n名称 IP CPU 内存 磁盘 pxc2 192.168.1.32 1 2G 40G pxc3 192.168.1.33 1 2G 40G pxc4 192.168.1.35 1 2G 40G pxc5 192.168.1.36 1 2G 40G pxc6 192.168.1.37 1 2G 40G pxc7 192.168.1.38 1 2G 40G 2. 删除MariaDB程序包 $ yum -y remove mari* 3. 开放防火墙端口 $ firewall-cmd --zone=public --add-port=3306/tcp --permanent $ firewall-cmd --zone=public --add-port=4444/tcp --permanent $ firewall-cmd --zone=public --add-port=4567/tcp --permanent $ firewall-cmd --zone=public --add-port=4568/tcp --permanent 4. 关闭SELINUX vi /etc/selinux/config 把SELINUX属性值设置成disabled\nreboot 5. 离线安装PXC（推荐） 进入官网下载5.7版本的Percona，注意，此处下载的是集群版本，下载后分发到所有集群节点执行相同的安装和配置操作\n进入RPM文件目录，执行安装命令\n$ yum localinstall *.rpm 参考前一篇内容，修改MySQL配置文件、创建账户等操作\n6. 创建PXC集群 停止MySQL服务\n修改每个PXC节点的/etc/my.cnf文件（在不同节点上，注意调整文件内容）\nserver-id=1 #PXC集群中MySQL实例的唯一ID，不能重复，必须是数字 wsrep_provider=/usr/lib64/galera3/libgalera_smm.so wsrep_cluster_name=pxc-cluster #PXC集群的名称 wsrep_cluster_address=gcomm://pxc2,pxc3,pxc4 wsrep_node_name=pxc2 #当前节点的名称 wsrep_node_address=pxc2 #当前节点的IP wsrep_sst_method=xtrabackup-v2 #同步方法（mysqldump、rsync、xtrabackup） wsrep_sst_auth= admin:password #同步使用的帐户 pxc_strict_mode=ENFORCING #同步严厉模式 binlog_format=ROW #基于ROW复制（安全可靠） default_storage_engine=InnoDB #默认引擎 innodb_autoinc_lock_mode=2 #主键自增长不锁表 主节点的管理命令（第一个启动的PXC节点）\n$ systemctl start mysql@bootstrap.service $ systemctl stop mysql@bootstrap.service $ systemctl restart mysql@bootstrap.service 非主节点的管理命令（非第一个启动的PXC节点）\n$ systemctl start mysql $ systemctl stop mysql $ systemctl restart mysql 查看PXC集群状态信息\nshow status like \u0026#39;wsrep_cluster%\u0026#39; ; 按照上述配置方法，创建两组PXC集群\n7. PXC节点启动与关闭 如果最后关闭的PXC节点是安全退出的，那么下次启动要最先启动这个节点，而且要以主节点启动 如果最后关闭的PXC节点不是安全退出的，那么要先修改/var/lib/mysql/grastate.dat 文件，把其中的safe_to_bootstrap属性值设置为1，再安装主节点启动 ","date":"2019-12-30T08:42:07+08:00","image":"/images/recovered/de47efcb191c722de4a9.png","permalink":"/archives/231/","title":"PXC系列学习笔记（二）——PXC集群安装配置"},{"content":"本系列笔记是对慕课网课程的学习与总结MySQL数据库集群-PXC方案\nPercona数据库介绍 官网 Percona官网\nPercona数据库提供MySQL、MongoDB等衍生版本，在大数据、同步性能上做了大量优化\n安装 Percona官方也有提供docker镜像，但为了方便学习，全使用云服务器安装Percona数据库，本次使用的机器配置为\nCPU：1C RAM：1G DISK：40G 1. 离线安装Percona 安装前置环境\n$ yum install https://repo.percona.com/yum/percona-release-latest.noarch.rpm -y $ yum install qpress 进入官网下载5.7版本的Percona，注意，此处下载的是单机版本\n进入RPM安装文件目录，执行下面的脚本\n$ yum localinstall *.rpm 管理MySQL服务\n$ systemctl enable mysqld $ systemctl start mysqld $ systemctl stop mysqld $ systemctl restart mysqld 2. 在线安装Percona（不推荐） 在线安装不便于统一版本和多节点分发\n使用yum命令安装\n$ yum install https://repo.percona.com/yum/percona-release-latest.noarch.rpm $ yum install Percona-Server-server-57 管理MySQL服务\n$ service mysql start $ service mysql stop $ service mysql restart 3. 开放防火墙端口 $ firewall-cmd --zone=public --add-port=3306/tcp --permanent $ firewall-cmd --reload 4. 修改MySQL配置文件 vi /etc/my.cnf [mysqld] character_set_server = utf8 bind-address = 0.0.0.0 #跳过DNS解析 skip-name-resolve $ service mysql restart 5. 禁止开机启动MySQL 若某一节点宕机，其自动启动后会开始自动同步，若宕机时间过长，PXC集群会限制其他节点的写入操作，直到所有数据同步成功\n宕机后科学做法：手动从其他节点中拷贝数据库文件，再启动，减少同步数据\n$ chkconfig mysqld off 注意，关闭了开机启动后无法使用service mysql restart 命令启动数据库，需要使用下面的命令启动\n$ service mysqld restart 6. 初始化MySQL数据库 查看MySQL初始密码\ncat /var/log/mysqld.log | grep \u0026#34;A temporary password\u0026#34; 修改MySQL密码\nmysql_secure_installation 修改过程全输入y，并录入密码\n创建远程管理员账户\nmysql -u root -p CREATE USER \u0026#39;admin\u0026#39;@\u0026#39;%\u0026#39; IDENTIFIED BY \u0026#39;password\u0026#39;; GRANT all privileges ON *.* TO \u0026#39;admin\u0026#39;@\u0026#39;%\u0026#39;; FLUSH PRIVILEGES; ","date":"2019-12-29T20:03:00+08:00","image":"/images/recovered/de47efcb191c722de4a9.png","permalink":"/archives/230/","title":"PXC系列学习笔记（一）——PXC单节点安装配置"},{"content":"背景 目前使用5台服务器搭建了Kubernetes集群环境，监控、日志采集均已落地，业务也手工迁移到集群中顺利运行，故需要将原本基于原生docker环境的CICD流程迁移到Kubernetes集群中\n优势 Kubernetes集群实现CICD有几个显著优势\nDeployment 天然支持滚动部署、结合其他Kubernetes特性还能实现蓝绿部署、金丝雀部署等 新版本的GitLab与GitLab Runner天然支持Kubernetes集群，支持runner自动伸缩，减小资源占用 环境 Kubernetes版本：1.14\nGitLab版本：12.2.5\nGitLab-Runner版本：12.1.0\nDocker环境版本：17.03.1\nGitLab-Runner部署 配置介绍 原始环境的gitlab runner通过手动执行官网提供的注册命令和启动命令，分成两部部署，需要较多的手工操作，而在Kubernetes中，其支持使用Helm一键部署，官方文档如下\nGitLab Runner Helm Chart\n其实官方文档的指引并不清晰，许多配置在文档中没有介绍用法，推荐去其源码仓库查看详细的参数使用文档\nThe Kubernetes executor\n其中介绍了几个关键配置，在后面修改工程的ci配置文件时会用到\n使用DinD方式构建已经不再推荐 官方文档介绍\nUse docker-in-docker workflow with Docker executor The second approach is to use the special docker-in-docker (dind) Docker image with all tools installed (docker) and run the job script in context of that image in privileged mode.\nNote: docker-compose is not part of docker-in-docker (dind). To use docker-compose in your CI builds, follow the docker-compose installation instructions.\nDanger: By enabling --docker-privileged, you are effectively disabling all of the security mechanisms of containers and exposing your host to privilege escalation which can lead to container breakout. For more information, check out the official Docker documentation on Runtime privilege and Linux capabilities.\nDocker-in-Docker works well, and is the recommended configuration, but it is not without its own challenges:\nWhen using docker-in-docker, each job is in a clean environment without the past history. Concurrent jobs work fine because every build gets it’s own instance of Docker engine so they won’t conflict with each other. But this also means jobs can be slower because there’s no caching of layers. By default, Docker 17.09 and higher uses --storage-driver overlay2 which is the recommended storage driver. See Using the overlayfs driver for details. Since the docker:19.03.1-dind container and the Runner container don’t share their root filesystem, the job’s working directory can be used as a mount point for child containers. For example, if you have files you want to share with a child container, you may create a subdirectory under /builds/$CI_PROJECT_PATH and use it as your mount point (for a more thorough explanation, check issue #41227): 总之使用DinD进行容器构建并非不可行，但面临许多问题，例如使用overlay2网络需要Docker版本高于 17.09\nUsing docker:dind Running the docker:dind also known as the docker-in-docker image is also possible but sadly needs the containers to be run in privileged mode. If you\u0026rsquo;re willing to take that risk other problems will arise that might not seem as straight forward at first glance. Because the docker daemon is started as a service usually in your .gitlab-ci.yaml it will be run as a separate container in your Pod. Basically containers in Pods only share volumes assigned to them and an IP address by which they can reach each other using localhost. /var/run/docker.sock is not shared by the docker:dind container and the docker binary tries to use it by default.\nTo overwrite this and make the client use TCP to contact the Docker daemon, in the other container, be sure to include the environment variables of the build container:\nDOCKER_HOST=tcp://localhost:2375 for no TLS connection. DOCKER_HOST=tcp://localhost:2376 for TLS connection. Make sure to configure those properly. As of Docker 19.03, TLS is enabled by default but it requires mapping certificates to your client. You can enable non-TLS connection for DIND or mount certificates as described in Use Docker In Docker Workflow wiht Docker executor\n在Docker 19.03.1版本之后默认开启了TLS配置，在构建的环境变量中需要声明，否则报连接不上docker的错误，并且使用DinD构建需要runner开启特权模式，以访问主机的资源，并且由于使用了特权模式，在Pod中对runner需要使用的资源限制将失效\n使用Kaniko构建Docker镜像 目前官方提供另一种方式在docker容器中构建并推送镜像，实现更加优雅，可以实现无缝迁移，那就是kaniko\nBuilding a Docker image with kaniko\n其优势官网描述如下\n在Kubernetes集群中构建Docker映像的另一种方法是使用kaniko。iko子\n允许您构建没有特权访问权限的映像。 无需Docker守护程序即可工作。 在后面的实践中会使用两种方式构建Docker镜像，可根据实际情况选择\n使用Helm部署 拉取Helm Gitlab-Runner仓库到本地，修改配置\nGitLab Runner\n将原有的gitlab-runner配置迁移到Helm中，迁移后如下\nimage: alpine-v12.1.0 imagePullPolicy: IfNotPresent gitlabUrl: https://gitlab.fjy8018.top/ runnerRegistrationToken: \u0026#34;***\u0026#34; unregisterRunners: true terminationGracePeriodSeconds: 3600 concurrent: 10 checkInterval: 30 rbac: create: true clusterWideAccess: false metrics: enabled: true listenPort: 9090 runners: image: ubuntu:16.04 imagePullSecrets: - name: registry-secret locked: false tags: \u0026#34;k8s\u0026#34; runUntagged: true privileged: true pollTimeout: 180 outputLimit: 4096 cache: {} builds: {} services: {} helpers: {} resources: limits: memory: 2048Mi cpu: 1500m requests: memory: 128Mi cpu: 200m affinity: {} nodeSelector: {} tolerations: [] hostAliases: - ip: \u0026#34;192.168.1.13\u0026#34; hostnames: - \u0026#34;gitlab.fjy8018.top\u0026#34; - ip: \u0026#34;192.168.1.30\u0026#34; hostnames: - \u0026#34;harbor.fjy8018.top\u0026#34; podAnnotations: {} 其中配置了私钥、内网harbor地址、harbor拉取资源私钥，资源限制策略\nGitLab-Runner选择可能导致的坑 选择runner镜像为alpine-v12.1.0，这一点单独说一下，目前最新的runner版本为12.5.0，但其有许多问题，alpine新版镜像在Kubernetes中间断发生无法解析DNS的问题，反映到GitLab-Runner中就是Could not resolve host和server misbehaving\n查阅解决方法\n通过查询发现，其官方仓库还有多个相关issue没有关闭\n官方gitlab：Kubernetes runner: Could not resolve host\nstackoverflow：Gitlab Runner is not able to resolve DNS of Gitlab Server\n给出的解决方案无一例外都是降级到alpine-v12.1.0\nWe had same issue for couple of days. We tried change CoreDNS config, move runners to different k8s cluster and so on. Finally today i checked my personal runner and found that i\u0026rsquo;m using different version. Runners in cluster had gitlab/gitlab-runner:alpine-v12.3.0, when mine had gitlab/gitlab-runner:alpine-v12.0.1. We added line\nimage: gitlab/gitlab-runner:alpine-v12.1.0 in values.yaml and this solved problem for us\n其问题的根源应该在于alpine基础镜像对Kubernetes 集群支持有问题，\nndots breaks DNS resolving #64924\ndocker-alpine仓库对应也有未关闭的issue，其中就提到了关于DNS解析超时和异常的问题\nDNS Issue #255\n安装 一行命令安装即可\n$ helm install /root/gitlab-runner/ --name k8s-gitlab-runner --namespace gitlab-runner 输出如下\nNAME: k8s-gitlab-runner LAST DEPLOYED: Tue Nov 26 21:51:57 2019 NAMESPACE: gitlab-runner STATUS: DEPLOYED RESOURCES: ==\u0026gt; v1/ConfigMap NAME DATA AGE k8s-gitlab-runner-gitlab-runner 5 0s ==\u0026gt; v1/Deployment NAME READY UP-TO-DATE AVAILABLE AGE k8s-gitlab-runner-gitlab-runner 0/1 1 0 0s ==\u0026gt; v1/Pod(related) NAME READY STATUS RESTARTS AGE k8s-gitlab-runner-gitlab-runner-744d598997-xwh92 0/1 Pending 0 0s ==\u0026gt; v1/Role NAME AGE k8s-gitlab-runner-gitlab-runner 0s ==\u0026gt; v1/RoleBinding NAME AGE k8s-gitlab-runner-gitlab-runner 0s ==\u0026gt; v1/Secret NAME TYPE DATA AGE k8s-gitlab-runner-gitlab-runner Opaque 2 0s ==\u0026gt; v1/ServiceAccount NAME SECRETS AGE k8s-gitlab-runner-gitlab-runner 1 0s NOTES: Your GitLab Runner should now be registered against the GitLab instance reachable at: \u0026#34;https://gitlab.fjy8018.top/\u0026#34; 查看gitlab admin页面，发现已经有一个runner成功注册\n工程配置 DinD方式构建所需配置 如果原本的ci文件是基于19.03 DinD镜像构建的则需要加上TLS相关配置\nimage: docker:19.03 variables: DOCKER_DRIVER: overlay DOCKER_HOST: tcp://localhost:2375 DOCKER_TLS_CERTDIR: \u0026#34;\u0026#34; ... 其余配置保持不变，使用DinD构建\nKubectl和Kubernetes权限配置 由于使用k8s集群，而通过集群部署需要使用kubectl客户端，故手动创建了一个kubectl docker镜像，使用gitlab触发dockerhub构建，构建内容公开透明，可放心使用，如有其它版本的构建需求也可提pull request，会在后面补充，目前用到的只有1.14.0\nfjy8018/kubectl\n有kubectl客户端，还需要配置连接TLS和连接账户\n为了保障安全，新建一个专门访问该工程命名空间的ServiceAccount\napiVersion: v1 kind: ServiceAccount metadata: name: hmdt-gitlab-ci namespace: hmdt 利用集群提供的RBAC机制，为该账户授予该命名空间的admin权限\napiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: hmdt-gitlab-role namespace: hmdt subjects: - kind: ServiceAccount name: hmdt-gitlab-ci namespace: hmdt roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: admin 创建后在查询其在k8s集群中生成的唯一名称，此处为hmdt-gitlab-ci-token-86n89\n$ kubectl describe sa hmdt-gitlab-ci -n hmdt Name: hmdt-gitlab-ci Namespace: hmdt Labels: \u0026lt;none\u0026gt; Annotations: kubectl.`Kubernetes`.io/last-applied-configuration: {\u0026#34;apiVersion\u0026#34;:\u0026#34;v1\u0026#34;,\u0026#34;kind\u0026#34;:\u0026#34;ServiceAccount\u0026#34;,\u0026#34;metadata\u0026#34;:{\u0026#34;annotations\u0026#34;:{},\u0026#34;name\u0026#34;:\u0026#34;hmdt-gitlab-ci\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;hmdt\u0026#34;}} Image pull secrets: \u0026lt;none\u0026gt; Mountable secrets: hmdt-gitlab-ci-token-86n89 Tokens: hmdt-gitlab-ci-token-86n89 Events: \u0026lt;none\u0026gt; 然后根据上面的Secret找到CA证书\n$ kubectl get secret hmdt-gitlab-ci-token-86n89 -n hmdt -o json | jq -r \u0026#39;.data[\u0026#34;ca.crt\u0026#34;]\u0026#39; | base64 -d 再找到对应的 Token\n$ kubectl get secret hmdt-gitlab-ci-token-86n89 -n hmdt -o json | jq -r \u0026#39;.data.token\u0026#39; | base64 -d Kubernetes关联GitLab配置 进入gitlab Kubernetes集群配置页面，填写相关信息，让gitlab自动连接上集群环境\n注意，需要将此处取消勾选，否则gitlab会自动创建新的用户账户，而不使用已经创建好的用户账户，在运行过程中会报无权限错误\n不取消导致的报错如下，gitlab创建了新的用户账户hmdt-prod-service-account，但没有操作指定命名空间的权限\nGitLab环境配置 创建环境\n名称和url可以按需自定义\nCI脚本配置 最终配置CI文件如下，该文件使用DinD方式构建Dockerfile\nimage: docker:19.03 variables: MAVEN_CLI_OPTS: \u0026#34;-s .m2/settings.xml --batch-mode -Dmaven.test.skip=true\u0026#34; MAVEN_OPTS: \u0026#34;-Dmaven.repo.local=.m2/repository\u0026#34; DOCKER_DRIVER: overlay DOCKER_HOST: tcp://localhost:2375 DOCKER_TLS_CERTDIR: \u0026#34;\u0026#34; SPRING_PROFILES_ACTIVE: docker IMAGE_VERSION: \u0026#34;1.8.6\u0026#34; DOCKER_REGISTRY_MIRROR: \u0026#34;https://XXX.mirror.aliyuncs.com\u0026#34; stages: - test - package - review - deploy maven-build: image: maven:3-jdk-8 stage: test retry: 2 script: - mvn $MAVEN_CLI_OPTS clean package -U -B -T 2C artifacts: expire_in: 1 week paths: - target/*.jar maven-scan: stage: test retry: 2 image: maven:3-jdk-8 script: - mvn $MAVEN_CLI_OPTS verify sonar:sonar maven-deploy: stage: deploy retry: 2 image: maven:3-jdk-8 script: - mvn $MAVEN_CLI_OPTS deploy docker-harbor-build: image: docker:19.03 stage: package retry: 2 services: - name: docker:19.03-dind alias: docker before_script: - docker login -u \u0026#34;$CI_REGISTRY_USER\u0026#34; -p \u0026#34;$CI_REGISTRY_PASSWORD\u0026#34; $CI_REGISTRY script: - docker build --pull -t \u0026#34;$CI_REGISTRY_IMAGE:$IMAGE_VERSION\u0026#34; . - docker push \u0026#34;$CI_REGISTRY_IMAGE:$IMAGE_VERSION\u0026#34; - docker logout $CI_REGISTRY deploy_live: image: fjy8018/kubectl:v1.14.0 stage: deploy retry: 2 environment: name: prod url: https://XXXX script: - kubectl version - kubectl get pods -n hmdt - cd manifests/ - sed -i \u0026#34;s/__IMAGE_VERSION_SLUG__/${IMAGE_VERSION}/\u0026#34; deployment.yaml - kubectl apply -f deployment.yaml - kubectl rollout status -f deployment.yaml - kubectl get pods -n hmdt 若需要使用Kaniko构建Dockerfile，则配置如下\n注意，其中依赖的镜像gcr.io/kaniko-project/executor:debug属于谷歌镜像仓库，可能存在无法拉取的情况\nimage: docker:19.03 variables: MAVEN_CLI_OPTS: \u0026#34;-s .m2/settings.xml --batch-mode -Dmaven.test.skip=true\u0026#34; MAVEN_OPTS: \u0026#34;-Dmaven.repo.local=.m2/repository\u0026#34; DOCKER_DRIVER: overlay DOCKER_HOST: tcp://localhost:2375 DOCKER_TLS_CERTDIR: \u0026#34;\u0026#34; SPRING_PROFILES_ACTIVE: docker IMAGE_VERSION: \u0026#34;1.8.6\u0026#34; DOCKER_REGISTRY_MIRROR: \u0026#34;https://XXX.mirror.aliyuncs.com\u0026#34; cache: paths: - target/ stages: - test - package - review - deploy maven-build: image: maven:3-jdk-8 stage: test retry: 2 script: - mvn $MAVEN_CLI_OPTS clean package -U -B -T 2C artifacts: expire_in: 1 week paths: - target/*.jar maven-scan: stage: test retry: 2 image: maven:3-jdk-8 script: - mvn $MAVEN_CLI_OPTS verify sonar:sonar maven-deploy: stage: deploy retry: 2 image: maven:3-jdk-8 script: - mvn $MAVEN_CLI_OPTS deploy docker-harbor-build: stage: package retry: 2 image: name: gcr.io/kaniko-project/executor:debug entrypoint: [\u0026#34;\u0026#34;] script: - echo \u0026#34;{\\\u0026#34;auths\\\u0026#34;:{\\\u0026#34;$CI_REGISTRY\\\u0026#34;:{\\\u0026#34;username\\\u0026#34;:\\\u0026#34;$CI_REGISTRY_USER\\\u0026#34;,\\\u0026#34;password\\\u0026#34;:\\\u0026#34;$CI_REGISTRY_PASSWORD\\\u0026#34;}}}\u0026#34; \u0026gt; /kaniko/.docker/config.json - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$IMAGE_VERSION deploy_live: image: fjy8018/kubectl:v1.14.0 stage: deploy retry: 2 environment: name: prod url: https://XXXX script: - kubectl version - kubectl get pods -n hmdt - cd manifests/ - sed -i \u0026#34;s/__IMAGE_VERSION_SLUG__/${IMAGE_VERSION}/\u0026#34; deployment.yaml - kubectl apply -f deployment.yaml - kubectl rollout status -f deployment.yaml - kubectl get pods -n hmdt 执行流水线 runner自动扩缩容 Kubernetes中的runner会根据任务多少自动扩缩容，目前配置的上限为10个\nGrafana也能监控到集群在构建过程中的资源使用情况\n使用DinD构建Dockerfile结果 使用Kaniko构建Dockerfile的结果 部署结果 执行部署时gitlab会自动注入配置好的kubectl config\n构建结果 部署完成后可在环境配置页中查看部署结果，只有成功的部署才会被记录\n","date":"2019-11-27T10:24:00+08:00","image":"https://s2.ax1x.com/2019/11/27/Qpybmn.png","permalink":"/archives/227/","title":"k8s环境下GitLab+Helm+GitLab Runner Java项目CICD落地实践"},{"content":"介绍 StorageClass 为管理员提供了描述存储 \u0026quot;类\u0026quot; 的方法。 不同的类型可能会映射到不同的服务质量等级或备份策略，通常由运维制定的任意策略\n用户通过在 PersistentVolumeClaim 中包含存储类来请求动态供应的存储 ，动态卷供应的实现基于 storage.k8s.io API 组中的 StorageClass API 对象。 集群管理员可以根据需要定义多个 StorageClass 对象，每个对象指定一个卷插件（又名 provisioner）， 卷插件向卷供应商提供在创建卷时需要的数据卷信息及相关参数。\n环境 Kubernetes：v1.14.0\nDocker：17.03.1-ce\n机器数：$3 \\times master + 2 \\times worker$\n其中，master3、worker1、worker2都挂载了2块磁盘，非系统盘为纯净未分区磁盘，用于提供给GlusterFs管理\n安装GlusterFs 在所有可能被调度到pod的节点安装glusterfs\n$ yum install -y glusterfs glusterfs-fuse net-tools 启动GlusterFs与heketi 使用yaml文件启动glusterfs及其操作工具heketi\nglusterfs-daemonset.yaml\n--- kind: DaemonSet apiVersion: extensions/v1beta1 metadata: name: glusterfs labels: glusterfs: daemonset annotations: description: GlusterFS DaemonSet tags: glusterfs spec: template: metadata: name: glusterfs labels: glusterfs: pod glusterfs-node: pod spec: nodeSelector: storagenode: glusterfs hostNetwork: true containers: - image: gluster/gluster-centos:latest imagePullPolicy: IfNotPresent name: glusterfs env: # alternative for /dev volumeMount to enable access to *all* devices - name: HOST_DEV_DIR value: \u0026#34;/mnt/host-dev\u0026#34; # set GLUSTER_BLOCKD_STATUS_PROBE_ENABLE to \u0026#34;1\u0026#34; so the # readiness/liveness probe validate gluster-blockd as well - name: GLUSTER_BLOCKD_STATUS_PROBE_ENABLE value: \u0026#34;1\u0026#34; - name: GB_GLFS_LRU_COUNT value: \u0026#34;15\u0026#34; - name: TCMU_LOGDIR value: \u0026#34;/var/log/glusterfs/gluster-block\u0026#34; resources: requests: memory: 100Mi cpu: 100m volumeMounts: - name: glusterfs-heketi mountPath: \u0026#34;/var/lib/heketi\u0026#34; - name: glusterfs-run mountPath: \u0026#34;/run\u0026#34; - name: glusterfs-lvm mountPath: \u0026#34;/run/lvm\u0026#34; - name: glusterfs-etc mountPath: \u0026#34;/etc/glusterfs\u0026#34; - name: glusterfs-logs mountPath: \u0026#34;/var/log/glusterfs\u0026#34; - name: glusterfs-config mountPath: \u0026#34;/var/lib/glusterd\u0026#34; - name: glusterfs-host-dev mountPath: \u0026#34;/mnt/host-dev\u0026#34; - name: glusterfs-misc mountPath: \u0026#34;/var/lib/misc/glusterfsd\u0026#34; - name: glusterfs-block-sys-class mountPath: \u0026#34;/sys/class\u0026#34; - name: glusterfs-block-sys-module mountPath: \u0026#34;/sys/module\u0026#34; - name: glusterfs-cgroup mountPath: \u0026#34;/sys/fs/cgroup\u0026#34; readOnly: true - name: glusterfs-ssl mountPath: \u0026#34;/etc/ssl\u0026#34; readOnly: true - name: kernel-modules mountPath: \u0026#34;/usr/lib/modules\u0026#34; readOnly: true securityContext: capabilities: {} privileged: true readinessProbe: timeoutSeconds: 3 initialDelaySeconds: 40 exec: command: - \u0026#34;/bin/bash\u0026#34; - \u0026#34;-c\u0026#34; - \u0026#34;if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh readiness; else systemctl status glusterd.service; fi\u0026#34; periodSeconds: 25 successThreshold: 1 failureThreshold: 50 livenessProbe: timeoutSeconds: 3 initialDelaySeconds: 40 exec: command: - \u0026#34;/bin/bash\u0026#34; - \u0026#34;-c\u0026#34; - \u0026#34;if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh liveness; else systemctl status glusterd.service; fi\u0026#34; periodSeconds: 25 successThreshold: 1 failureThreshold: 50 volumes: - name: glusterfs-heketi hostPath: path: \u0026#34;/var/lib/heketi\u0026#34; - name: glusterfs-run - name: glusterfs-lvm hostPath: path: \u0026#34;/run/lvm\u0026#34; - name: glusterfs-etc hostPath: path: \u0026#34;/etc/glusterfs\u0026#34; - name: glusterfs-logs hostPath: path: \u0026#34;/var/log/glusterfs\u0026#34; - name: glusterfs-config hostPath: path: \u0026#34;/var/lib/glusterd\u0026#34; - name: glusterfs-host-dev hostPath: path: \u0026#34;/dev\u0026#34; - name: glusterfs-misc hostPath: path: \u0026#34;/var/lib/misc/glusterfsd\u0026#34; - name: glusterfs-block-sys-class hostPath: path: \u0026#34;/sys/class\u0026#34; - name: glusterfs-block-sys-module hostPath: path: \u0026#34;/sys/module\u0026#34; - name: glusterfs-cgroup hostPath: path: \u0026#34;/sys/fs/cgroup\u0026#34; - name: glusterfs-ssl hostPath: path: \u0026#34;/etc/ssl\u0026#34; - name: kernel-modules hostPath: path: \u0026#34;/usr/lib/modules\u0026#34; heketi-security.yaml 用户与权限相关\napiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: heketi-clusterrolebinding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: heketi-clusterrole subjects: - kind: ServiceAccount name: heketi-service-account namespace: default --- apiVersion: v1 kind: ServiceAccount metadata: name: heketi-service-account namespace: default --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: heketi-clusterrole rules: - apiGroups: - \u0026#34;\u0026#34; resources: - pods - pods/status - pods/exec verbs: - get - list - watch - create heketi-deployment.yaml 启动服务并使用ingress四层反向代理\nkind: Service apiVersion: v1 metadata: name: heketi labels: glusterfs: heketi-service deploy-heketi: support annotations: description: Exposes Heketi Service spec: selector: name: heketi ports: - name: heketi port: 80 targetPort: 8080 --- apiVersion: v1 kind: ConfigMap metadata: name: tcp-services namespace: ingress-nginx data: \u0026#34;30001\u0026#34;: default/heketi:80 --- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: heketi labels: glusterfs: heketi-deployment annotations: description: Defines how to deploy Heketi spec: replicas: 1 template: metadata: name: heketi labels: name: heketi glusterfs: heketi-pod spec: serviceAccountName: heketi-service-account containers: - image: heketi/heketi:6 imagePullPolicy: Always name: heketi env: - name: HEKETI_EXECUTOR value: \u0026#34;kubernetes\u0026#34; - name: HEKETI_DB_PATH value: \u0026#34;/var/lib/heketi/heketi.db\u0026#34; - name: HEKETI_FSTAB value: \u0026#34;/var/lib/heketi/fstab\u0026#34; - name: HEKETI_SNAPSHOT_LIMIT value: \u0026#34;14\u0026#34; - name: HEKETI_KUBE_GLUSTER_DAEMONSET value: \u0026#34;y\u0026#34; ports: - containerPort: 8080 volumeMounts: - name: db mountPath: /var/lib/heketi readinessProbe: timeoutSeconds: 3 initialDelaySeconds: 3 httpGet: path: /hello port: 8080 livenessProbe: timeoutSeconds: 3 initialDelaySeconds: 30 httpGet: path: /hello port: 8080 volumes: - name: db hostPath: path: \u0026#34;/heketi-data\u0026#34; GlusterFS初始化 创建 使用heketi-cli初始化GlusterFS节点，找到heketi容器并进入\n$ docker ps | grep hek 9c92cf25e4bb heketi/heketi@sha256:44058f20a91211234abc533c9b4b5c42764b8cd740e21d03377bee14ebca4497 \u0026#34;/usr/bin/heketi-s...\u0026#34; About a minute ago Up 58 seconds k8s_he eti_heketi-68f9dfdfbf-w4qrd_default_c0566bf2-09ff-11ea-b6ce-000c298872e2_0 ebafb92c4dc1 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 \u0026#34;/pause\u0026#34; About a minute ago Up About a minute k8s_POD_heketi-68f9dfdfbf-w4qrd_default_c0566bf2-09ff-11ea-b6ce-000c298872e2_0 $ docker exec -it 9c92 /bin/bash 配置容器本地环境变量，并配置节点json文件\n$ export HEKETI_CLI_SERVER=http://localhost:8080 $ vi topology.json 主机名需要唯一并可以访问\n{ \u0026#34;clusters\u0026#34;: [ { \u0026#34;nodes\u0026#34;: [ { \u0026#34;node\u0026#34;: { \u0026#34;hostnames\u0026#34;: { \u0026#34;manage\u0026#34;: [ \u0026#34;k8smaster3\u0026#34; ], \u0026#34;storage\u0026#34;: [ \u0026#34;192.168.1.17\u0026#34; ] }, \u0026#34;zone\u0026#34;: 1 }, \u0026#34;devices\u0026#34;: [\u0026#34;/dev/sdb\u0026#34;] }, { \u0026#34;node\u0026#34;: { \u0026#34;hostnames\u0026#34;: { \u0026#34;manage\u0026#34;: [ \u0026#34;k8sworker1\u0026#34; ], \u0026#34;storage\u0026#34;: [ \u0026#34;192.168.1.18\u0026#34; ] }, \u0026#34;zone\u0026#34;: 1 }, \u0026#34;devices\u0026#34;: [\u0026#34;/dev/sdb\u0026#34;] }, { \u0026#34;node\u0026#34;: { \u0026#34;hostnames\u0026#34;: { \u0026#34;manage\u0026#34;: [ \u0026#34;k8sworker2\u0026#34; ], \u0026#34;storage\u0026#34;: [ \u0026#34;192.168.1.19\u0026#34; ] }, \u0026#34;zone\u0026#34;: 1 }, \u0026#34;devices\u0026#34;: [\u0026#34;/dev/sdb\u0026#34;] } ] } ] } 装载json文件\n$ heketi-cli topology load --json topology.json Found node k8smaster3 on cluster 84d8a3a128c43a2a7d6c36aa6833823f Adding device /dev/sdb ... OK Found node k8sworker1 on cluster 84d8a3a128c43a2a7d6c36aa6833823f Adding device /dev/sdb ... OK Found node k8sworker2 on cluster 84d8a3a128c43a2a7d6c36aa6833823f Adding device /dev/sdb ... OK 可能出现的问题 1. 无法初始化 报错信息为\nCreating cluster ... ID: 84d8a3a128c43a2a7d6c36aa6833823f Allowing file volumes on cluster. Allowing block volumes on cluster. Creating node k8smaster3 ... ID: b97a53e296850d59bbdd4043b13ebf78 Adding device /dev/sdb ... Unable to add device: Unable to execute command on glusterfs-4vgh8: WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/centos/root not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sda1 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/centos/swap not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sda2 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds. Can\u0026#39;t initialize physical volume \u0026#34;/dev/sdb\u0026#34; of volume group \u0026#34;vg_9bc3bdaff53104647cd24abe9ca60745\u0026#34; without -ff /dev/sdb: physical volume not initialized. Creating node k8sworker1 ... ID: 3a57327dbd24d9aedcae01f72857ef5e Adding device /dev/sdb ... Unable to add device: Unable to execute command on glusterfs-wzcf8: WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/centos/root not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sda1 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/centos/swap not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sda2 not initialized in udev database even after waiting 10000000 microseconds. WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds. Can\u0026#39;t initialize physical volume \u0026#34;/dev/sdb\u0026#34; of volume group \u0026#34;vg_c6a688843cd06e333c83e56814c95c55\u0026#34; without -ff /dev/sdb: physical volume not initialized. Creating node k8sworker2 ... ID: 141f497c0797b26a22a31288ca97acf1 通过查阅资料，发现如下issue，官方提供磁盘擦除命令，执行后重新初始化节点\ndeploy fails using k8s 1.7.1 when heketi tries to add devices to cluster #2 #279\nDestroy the heketi and GlusterFS pods, then try the following on each of the nodes:\npvdisplay -C -o pv_name,vg_name # look for VGs on PVs on your target devices lvremove -ff \u0026lt;any VGs found above\u0026gt; vgremove -f \u0026lt;any VGs found above\u0026gt; pvremove -f \u0026lt;any PVs found above\u0026gt; wipefs -a \u0026lt;target device\u0026gt; Then try gk-deploy from the beginning again. Naturally, double-check that you\u0026rsquo;re definitely removing something only on the target devices. :)\n进入挂载磁盘节点，查看当前磁盘信息\n$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 40G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 39G 0 part ├─centos-root 253:0 0 35G 0 lvm / └─centos-swap 253:1 0 4G 0 lvm sdb 8:16 0 40G 0 disk sr0 11:0 1 1024M 0 rom 查看pvs名称\n$ pvs PV VG Fmt Attr PSize PFree /dev/sda2 centos lvm2 a-- \u0026lt;39.00g 0 /dev/sdb vg_c6a688843cd06e333c83e56814c95c55 lvm2 a-- 39.87g 39.87g 抹除磁盘\n$ pvdisplay -C -o /dev/sdb,vg_c6a688843cd06e333c83e56814c95c55 $ lvremove -ff vg_c6a688843cd06e333c83e56814c95c55 $ vgremove -f vg_c6a688843cd06e333c83e56814c95c55 $ pvremove -f /dev/sdb $ wipefs -a /dev/sdb 抹除后重新进入容器创建即可\n验证 进入3个存储节点，检查创建情况，此处以worker1为例\n$ netstat -ntlp | grep glusterd tcp 0 0 0.0.0.0:24007 0.0.0.0:* LISTEN 25997/glusterd $ docker ps | grep glus eae6ef1d4a37 22a008905c35 \u0026#34;/usr/local/bin/up...\u0026#34; 31 minutes ago Up 31 minutes k8s_glusterfs_glusterfs-wzcf8_default_940d0f57-09ff-11ea-b6ce-000c298872e2_0 858fd84edbc2 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 \u0026#34;/pause\u0026#34; 31 minutes ago Up 31 minutes k8s_POD_glusterfs-wzcf8_default_940d0f57-09ff-11ea-b6ce-000c298872e2_0 $ docker exec -it eae6 /bin/bash $ gluster pool list UUID Hostname State 6d4eda3d-a7c2-4e27-999c-972106c54c3d k8smaster3 Connected 56a6e1d4-46db-4ff2-83eb-39c313bc9197 192.168.1.19 Connected a8ac96bd-b596-433c-8e2a-e3a97873886b localhost Connected 创建storage-class 定义StorageClass，通过访问ingress-nginx四层代理暴露的端口管理glusterfs\nglusterfs-storage-class.yaml\napiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: glusterfs-storage-class provisioner: kubernetes.io/glusterfs parameters: resturl: \u0026#34;http://192.168.1.16:30001\u0026#34; restauthenabled: \u0026#34;false\u0026#34; 创建PVC 创建PVC， 可以被单个节点映射为ReadWriteOnce,或者多个节点映射为ReadOnlyMany,但不能同时使用这两种方式来映射。\nglusterfs-pvc.yaml\nkind: PersistentVolumeClaim apiVersion: v1 metadata: name: glusterfs-pvc spec: storageClassName: glusterfs-storage-class accessModes: - ReadWriteOnce resources: requests: storage: 1Gi 检查pvc，可见已经是已绑定(Bound)状态\n$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE glusterfs-pvc Bound pvc-7c4f94cd-0a04-11ea-b6ce-000c298872e2 1Gi RWO glusterfs-storage-class 128m 基于Helm安装Redis-HA 访问charts官方仓库：https://github.com/helm/charts\n在stable目录下找到redis-ha目录，修改values.yaml文件如下，主要配置存储类为storageClass: glusterfs-storage-class，然后修改用户名和密码等相关信息即可\nimage: repository: redis tag: 5.0.5-alpine pullPolicy: IfNotPresent replicas: 3 labels: {} configure-service-account/ serviceAccount: create: truetemplate Automatically proxies to Redis master. haproxy: enabled: false readOnly: enabled: false port: 6380 replicas: 1 image: repository: haproxy tag: 2.0.4 pullPolicy: IfNotPresent annotations: {} resources: {} service: type: ClusterIP loadBalancerIP: annotations: {} serviceAccount: create: true exporter: image: repository: quay.azk8s.cn/prometheus/haproxy-exporter tag: v0.9.0 enabled: false port: 9101 init: resources: {} timeout: connect: 4s server: 30s client: 30s securityContext: runAsUser: 1000 fsGroup: 1000 runAsNonRoot: true rbac: create: true sysctlImage: enabled: false command: [] registry: docker.io repository: busybox tag: 1.31.1 pullPolicy: Always mountHostSys: false configure-multiple-schedulers/ redis: port: 6379 masterGroupName: mymaster config: min-replicas-to-write: 1unlimited.instance. Default is volatile-lru.with a new slave. The only way to prevent this is to enable diskless replication. save: \u0026#34;900 1\u0026#34;the disk as intermediate storage. Default is false. repl-diskless-sync: \u0026#34;yes\u0026#34; rdbcompression: \u0026#34;yes\u0026#34; rdbchecksum: \u0026#34;yes\u0026#34; resources: {} sentinel: port: 26379 quorum: 2 config: down-after-milliseconds: 10000 failover-timeout: 180000 parallel-syncs: 5 resources: {} securityContext: runAsUser: 1000 fsGroup: 1000 runAsNonRoot: true assign-pod-node/#nodeselectorassign-pod-node/#taints-and-tolerations-beta-featureassign-pod-node/#affinity-and-anti-affinity nodeSelector: {} requiredDuringSchedulingIgnoredDuringExecution as opposed to preferred.assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature hardAntiAffinity: true additionalAffinities: {} affinity: | exporter: enabled: true image: oliver006/redis_exporter tag: v0.31.0 pullPolicy: IfNotPresent port: 9121 scrapePath: /metrics resources: {} extraArgs: {} serviceMonitor: enabled: falsePrometheus Operator podDisruptionBudget: {} auth: true redisPassword: password authKey: auth persistentVolume: enabled: true storageClass: glusterfs-storage-class accessModes: - ReadWriteOnce size: 100Mi annotations: {} init: resources: {} hostPath: chown: true 从修改后的本地文件安装redis-ha\n$ helm install /root/redis-ha/ --name k8s-redis-ha --namespace env-prod NAME: k8s-redis-ha LAST DEPLOYED: Mon Nov 18 22:13:20 2019 NAMESPACE: env-prod STATUS: DEPLOYED RESOURCES: ==\u0026gt; v1/ConfigMap NAME DATA AGE k8s-redis-ha-configmap 4 1s ==\u0026gt; v1/Pod(related) NAME READY STATUS RESTARTS AGE k8s-redis-ha-server-0 0/3 Pending 0 0s ==\u0026gt; v1/Role NAME AGE k8s-redis-ha 1s ==\u0026gt; v1/RoleBinding NAME AGE k8s-redis-ha 1s ==\u0026gt; v1/Secret NAME TYPE DATA AGE k8s-redis-ha Opaque 1 1s ==\u0026gt; v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE k8s-redis-ha ClusterIP None \u0026lt;none\u0026gt; 6379/TCP,26379/TCP,9121/TCP 1s k8s-redis-ha-announce-0 ClusterIP 10.101.55.132 \u0026lt;none\u0026gt; 6379/TCP,26379/TCP,9121/TCP 1s k8s-redis-ha-announce-1 ClusterIP 10.98.159.185 \u0026lt;none\u0026gt; 6379/TCP,26379/TCP,9121/TCP 1s k8s-redis-ha-announce-2 ClusterIP 10.102.215.171 \u0026lt;none\u0026gt; 6379/TCP,26379/TCP,9121/TCP 1s ==\u0026gt; v1/ServiceAccount NAME SECRETS AGE k8s-redis-ha 1 1s ==\u0026gt; v1/StatefulSet NAME READY AGE k8s-redis-ha-server 0/3 1s NOTES: Redis can be accessed via port 6379 and Sentinel can be accessed via port 26379 on the following DNS name from within your cluster: k8s-redis-ha.env-prod.svc.cluster.local To connect to your Redis server: 1. To retrieve the redis password: echo $(kubectl get secret k8s-redis-ha -o \u0026#34;jsonpath={.data[\u0026#39;auth\u0026#39;]}\u0026#34; | base64 --decode) 2. Connect to the Redis master pod that you can use as a client. By default the k8s-redis-ha-server-0 pod is configured as the master: kubectl exec -it k8s-redis-ha-server-0 sh -n env-prod 3. Connect using the Redis CLI (inside container): redis-cli -a \u0026lt;REDIS-PASS-FROM-SECRET\u0026gt; 检查pvc创建情况\n$ kubectl get pvc -n env-prod NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-k8s-redis-ha-server-0 Bound pvc-9348a01f-0a0d-11ea-b6ce-000c298872e2 1Gi RWO glusterfs-storage-class 62m data-k8s-redis-ha-server-1 Bound pvc-9a8defcf-0a0d-11ea-b6ce-000c298872e2 1Gi RWO glusterfs-storage-class 62m data-k8s-redis-ha-server-2 Bound pvc-a1975bf8-0a0d-11ea-b6ce-000c298872e2 1Gi RWO glusterfs-storage-class 62m 检查pv\n$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-7c4f94cd-0a04-11ea-b6ce-000c298872e2 1Gi RWO Delete Bound default/glusterfs-pvc glusterfs-storage-class 128m pvc-9348a01f-0a0d-11ea-b6ce-000c298872e2 1Gi RWO Delete Bound env-prod/data-k8s-redis-ha-server-0 glusterfs-storage-class 63m pvc-9a8defcf-0a0d-11ea-b6ce-000c298872e2 1Gi RWO Delete Bound env-prod/data-k8s-redis-ha-server-1 glusterfs-storage-class 63m pvc-a1975bf8-0a0d-11ea-b6ce-000c298872e2 1Gi RWO Delete Bound env-prod/data-k8s-redis-ha-server-2 glusterfs-storage-class 63m 配置ingress-nginx四层反向代理\napiVersion: v1 kind: ConfigMap metadata: name: tcp-services namespace: ingress-nginx data: \u0026#34;30001\u0026#34;: default/heketi:80 \u0026#34;30002\u0026#34;: env-prod/k8s-redis-ha:6379 \u0026#34;30003\u0026#34;: env-prod/k8s-redis-ha:26379 使用redis客户端连接即可\n","date":"2019-11-20T15:55:00+08:00","image":"https://s2.ax1x.com/2019/11/27/QC2YcT.png","permalink":"/archives/221/","title":"k8s共享存储以及Redis高可用集群快速搭建"},{"content":"架构 目前主流容器化监控平台采用Prometheus+Grafana架构，下图为Prometheus架构图，可以主动发现各种指标和采集数据，实现\n监控每个节点基础指标 监控每个容器基础指标 监控k8s集群组件基础指标 基础环境 Kubernetes：v1.14.0\nDocker：17.03.1-ce\n机器数：$3 \\times master + 2 \\times worker$\n部署 Helm Helm作为k8s平台软件包管理工具，结合prometheus-operator可以类似插件的方式快速搭建监控体系，Helm旗下包含charts项目，为软件包仓库\nHelm官方仓库：https://github.com/helm/helm\ncharts官方仓库：https://github.com/helm/charts\n安装Helm 通过官方仓库查看目前最新版为v2.16.0，但由于仍有不稳定的情况，故不推荐该版本，原因可通过其issue查看\nNo kind Job is registered for version batch/v1 #6894\nUnable to Deploy on Polyaxon with helm v2.16.0 (No kind Job is registered for version batch/v1) #601\n此处选择安装v2.15.2的版本，下载页面\n解压并放到可执行目录下\n$ tar -zxvf helm-v2.15.2-linux-amd64.tar.gz $ mv linux-amd64/helm /usr/local/bin/ Tiller安装 Tiller 是以 Deployment 方式部署在 Kubernetes 集群中的，由于国内网络原因，需要配置为阿里云镜像\n$ helm init --client-only --stable-repo-url https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts/ $ helm repo add incubator https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts-incubator/ $ helm repo update 因为官方的镜像无法拉取，使用-i指定自己的镜像\n$ helm init --service-account tiller --upgrade -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.15.2 --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts 创建TLS认证服务端\n$ helm init --service-account tiller --upgrade -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.15.2 --tiller-tls-cert /etc/kubernetes/ssl/tiller001.pem --tiller-tls-key /etc/kubernetes/ssl/tiller001-key.pem --tls-ca-cert /etc/kubernetes/ssl/ca.pem --tiller-namespace kube-system --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts Tiller授权 Tiller 需要访问ApiServer去对集群进行操作。目前的 Tiller 部署时默认没有定义授权的 ServiceAccount，这会导致访问 API Server 时被拒绝。所以我们需要明确为 Tiller 部署添加授权\n创建ServiceAccount和创建角色绑定\n$ kubectl create serviceaccount --namespace kube-system tiller $ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller 验证 查看Tiller的ServiceAccount，需要跟创建的名字一致：tiller\n$ kubectl get deploy --namespace kube-system tiller-deploy -o yaml|grep serviceAccount serviceAccount: tiller serviceAccountName: tiller 验证pods\n$ kubectl -n kube-system get pods|grep tiller tiller-deploy-b6c45d5bc-jhntj 1/1 Running 0 2d8h 验证版本\n$ helm version Client: \u0026amp;version.Version{SemVer:\u0026#34;v2.15.2\u0026#34;, GitCommit:\u0026#34;8dce272473e5f2a7bf58ce79bb5c3691db54c96b\u0026#34;, GitTreeState:\u0026#34;clean\u0026#34;} Server: \u0026amp;version.Version{SemVer:\u0026#34;v2.15.2\u0026#34;, GitCommit:\u0026#34;8dce272473e5f2a7bf58ce79bb5c3691db54c96b\u0026#34;, GitTreeState:\u0026#34;clean\u0026#34;} Operator 可以自定义资源类型(CDR)结合自定义控制器实现\n样例\napiVersion: app.example.com/v1 kind: AppService metadata: name: nginx-app spec: size: 2 image: nginx:1.7.9 ports: - port: 80 targetPort: 80 nodePort: 30002 Charts charts项目下stable目录里存放所有软件包内容，由于其默认从多处镜像仓库中拉取镜像，可能存在网络问题，需结合实际情况修改镜像配置，手动安装软件包\n推荐配置地址：\nquay.io 替换为 quay.azk8s.cn gcr.io/google-containers 替换为 registry.cn-hangzhou.aliyuncs.com/google-containers 本次需要安装的软件包为prometheus-operator，但其还包含多个依赖软件如下\ngrafana kube-state-metrics prometheus-node-exporter 若需要手动安装则需要按照整理目录如下\nprometheus-operator/ ├── charts │ ├── grafana │ ├── kube-state-metrics │ └── prometheus-node-exporter ... 手动安装 Etcd监控配置 监控etcd需要额外配置，而且较为复杂，具体步骤如下\n查看etcd在集群中的配置，由于本k8s集群基于kubeadm搭建，故etcd通过容器方式运行，查看方法\n# 查看etcd所在pod名称 $ kubectl get pod --all-namespaces | grep etcd kube-system etcd-k8smaster1 1/1 Running 0 7d1h # 展开etcd详细配置 $ kubectl get pod -n kube-system etcd-k8smaster1 -o yaml 配置内容查看其健康检查端点，--listen-client-urls指明其监听在https端点，故需要配置安全证书才能访问\napiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/config.hash: 6119961323d801d05a7dd23e429cda3f kubernetes.io/config.mirror: 6119961323d801d05a7dd23e429cda3f kubernetes.io/config.seen: \u0026#34;2019-11-05T19:17:28.480984831+08:00\u0026#34; kubernetes.io/config.source: file creationTimestamp: \u0026#34;2019-11-05T11:18:53Z\u0026#34; labels: component: etcd tier: control-plane name: etcd-k8smaster1 namespace: kube-system resourceVersion: \u0026#34;876368\u0026#34; selfLink: /api/v1/namespaces/kube-system/pods/etcd-k8smaster1 uid: 0ccf5e91-ffbe-11e9-a5d2-000c298872e2 spec: containers: - command: - etcd - --advertise-client-urls=https://192.168.1.15:2379 - --cert-file=/etc/kubernetes/pki/etcd/server.crt - --client-cert-auth=true - --data-dir=/var/lib/etcd - --initial-advertise-peer-urls=https://192.168.1.15:2380 - --initial-cluster=k8smaster1=https://192.168.1.15:2380 - --key-file=/etc/kubernetes/pki/etcd/server.key - --listen-client-urls=https://127.0.0.1:2379,https://192.168.1.15:2379 - --listen-peer-urls=https://192.168.1.15:2380 - --name=k8smaster1 - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt - --peer-client-cert-auth=true - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt - --snapshot-count=10000 - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt image: registry.cn-hangzhou.aliyuncs.com/imooc/etcd:3.3.10 imagePullPolicy: IfNotPresent livenessProbe: ... hostIP: 192.168.1.15 ... 配置安全证书\nk8s创建etcd时会自动创建相关证书，当前集群版本为1.14，证书位于/etc/kubernetes/pki/etcd/中\n$ ls /etc/kubernetes/pki/etcd/ ca.crt ca.key healthcheck-client.crt healthcheck-client.key peer.crt peer.key server.crt server.key 可以通过curl命令加载证书测试能否访问健康检查端点\n$ curl -k --cert /etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key https://192.168.1.15:2379/metrics 若有如下输出则正常\n# HELP etcd_debugging_mvcc_db_compaction_keys_total Total number of db keys compacted. # TYPE etcd_debugging_mvcc_db_compaction_keys_total counter etcd_debugging_mvcc_db_compaction_keys_total 1.030132e+06 # HELP etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds Bucketed histogram of db compaction pause duration. # TYPE etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds histogram etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le=\u0026#34;1\u0026#34;} 0 etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le=\u0026#34;2\u0026#34;} 0 etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le=\u0026#34;4\u0026#34;} 0 etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le=\u0026#34;8\u0026#34;} 0 ... 创建证书秘钥\n关联三个证书文件，创建k8s访问秘钥，秘钥名称为etcd-certs\n$ kubectl create secret generic etcd-certs -n monitoring --from-file=/etc/kubernetes/pki/etcd/ca.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key 写入prometheus配置文件values.yaml\n注意，此处路径前缀固定写法为/etc/prometheus/secrets/+秘钥名称+文件名称，若写错证书路径则无法在指标中查看到对应的target\n... serviceMonitor: ## Scrape interval. If not set, the Prometheus default scrape interval is used. ## interval: \u0026#34;\u0026#34; scheme: https insecureSkipVerify: true serverName: \u0026#34;\u0026#34; caFile: /etc/prometheus/secrets/etcd-certs/ca.crt certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key ... 创建 指定命名空间和文件目录，若命名空间不存在会自动创建\n$ helm install /root/prometheus-operator/ --name k8s-prometheus --namespace monitoring 等待一段时间，输出如下则创建成功\nNAME: k8s-prometheus LAST DEPLOYED: Tue Nov 12 20:27:18 2019 NAMESPACE: monitoring STATUS: DEPLOYED RESOURCES: ==\u0026gt; v1/Alertmanager NAME AGE k8s-prometheus-prometheus-alertmanager 33s ==\u0026gt; v1/ClusterRole NAME AGE k8s-prometheus-grafana-clusterrole 34s k8s-prometheus-prometheus-alertmanager 34s k8s-prometheus-prometheus-operator 34s k8s-prometheus-prometheus-operator-psp 34s k8s-prometheus-prometheus-prometheus 34s k8s-prometheus-prometheus-prometheus-psp 34s psp-k8s-prometheus-kube-state-metrics 34s psp-k8s-prometheus-prometheus-node-exporter 34s ==\u0026gt; v1/ClusterRoleBinding NAME AGE k8s-prometheus-grafana-clusterrolebinding 34s k8s-prometheus-prometheus-alertmanager 34s k8s-prometheus-prometheus-operator 34s k8s-prometheus-prometheus-operator-psp 34s k8s-prometheus-prometheus-prometheus 34s k8s-prometheus-prometheus-prometheus-psp 34s psp-k8s-prometheus-kube-state-metrics 34s psp-k8s-prometheus-prometheus-node-exporter 34s ==\u0026gt; v1/ConfigMap NAME DATA AGE k8s-prometheus-grafana 1 34s k8s-prometheus-grafana-config-dashboards 1 34s k8s-prometheus-grafana-test 1 34s k8s-prometheus-prometheus-apiserver 1 34s k8s-prometheus-prometheus-cluster-total 1 34s k8s-prometheus-prometheus-controller-manager 1 34s k8s-prometheus-prometheus-etcd 1 34s k8s-prometheus-prometheus-grafana-datasource 1 34s k8s-prometheus-prometheus-k8s-coredns 1 34s k8s-prometheus-prometheus-k8s-resources-cluster 1 34s k8s-prometheus-prometheus-k8s-resources-namespace 1 34s k8s-prometheus-prometheus-k8s-resources-node 1 34s k8s-prometheus-prometheus-k8s-resources-pod 1 34s k8s-prometheus-prometheus-k8s-resources-workload 1 34s k8s-prometheus-prometheus-k8s-resources-workloads-namespace 1 34s k8s-prometheus-prometheus-kubelet 1 34s k8s-prometheus-prometheus-namespace-by-pod 1 34s k8s-prometheus-prometheus-namespace-by-workload 1 34s k8s-prometheus-prometheus-node-cluster-rsrc-use 1 34s k8s-prometheus-prometheus-node-rsrc-use 1 34s k8s-prometheus-prometheus-nodes 1 34s k8s-prometheus-prometheus-persistentvolumesusage 1 34s k8s-prometheus-prometheus-pod-total 1 34s k8s-prometheus-prometheus-pods 1 34s k8s-prometheus-prometheus-prometheus 1 34s k8s-prometheus-prometheus-proxy 1 34s k8s-prometheus-prometheus-scheduler 1 34s k8s-prometheus-prometheus-statefulset 1 34s k8s-prometheus-prometheus-workload-total 1 34s ==\u0026gt; v1/DaemonSet NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE k8s-prometheus-prometheus-node-exporter 5 5 5 5 5 \u0026lt;none\u0026gt; 33s ==\u0026gt; v1/Deployment NAME READY UP-TO-DATE AVAILABLE AGE k8s-prometheus-grafana 1/1 1 1 33s k8s-prometheus-kube-state-metrics 1/1 1 1 33s k8s-prometheus-prometheus-operator 1/1 1 1 33s ==\u0026gt; v1/Pod(related) NAME READY STATUS RESTARTS AGE k8s-prometheus-grafana-769d6dfb69-mtbdg 2/2 Running 0 33s k8s-prometheus-kube-state-metrics-76b8ff78cd-kjxvl 1/1 Running 0 33s k8s-prometheus-prometheus-node-exporter-jqkvf 1/1 Running 0 33s k8s-prometheus-prometheus-node-exporter-jxpzm 1/1 Running 0 33s k8s-prometheus-prometheus-node-exporter-k8n6x 1/1 Running 0 33s k8s-prometheus-prometheus-node-exporter-kpjzk 1/1 Running 0 33s k8s-prometheus-prometheus-node-exporter-qc6kv 1/1 Running 0 33s k8s-prometheus-prometheus-operator-5f894bc75d-kvkvs 2/2 Running 0 33s ==\u0026gt; v1/Prometheus NAME AGE k8s-prometheus-prometheus-prometheus 33s ==\u0026gt; v1/PrometheusRule NAME AGE k8s-prometheus-prometheus-alertmanager.rules 32s k8s-prometheus-prometheus-etcd 32s k8s-prometheus-prometheus-general.rules 32s k8s-prometheus-prometheus-k8s.rules 32s k8s-prometheus-prometheus-kube-apiserver.rules 32s k8s-prometheus-prometheus-kube-prometheus-node-recording.rules 32s k8s-prometheus-prometheus-kube-scheduler.rules 32s k8s-prometheus-prometheus-kubernetes-absent 32s k8s-prometheus-prometheus-kubernetes-apps 32s k8s-prometheus-prometheus-kubernetes-resources 32s k8s-prometheus-prometheus-kubernetes-storage 32s k8s-prometheus-prometheus-kubernetes-system 32s k8s-prometheus-prometheus-kubernetes-system-apiserver 32s k8s-prometheus-prometheus-kubernetes-system-controller-manager 32s k8s-prometheus-prometheus-kubernetes-system-kubelet 32s k8s-prometheus-prometheus-kubernetes-system-scheduler 32s k8s-prometheus-prometheus-node-exporter 32s k8s-prometheus-prometheus-node-exporter.rules 32s k8s-prometheus-prometheus-node-network 32s k8s-prometheus-prometheus-node-time 32s k8s-prometheus-prometheus-node.rules 32s k8s-prometheus-prometheus-prometheus 32s k8s-prometheus-prometheus-prometheus-operator 32s ==\u0026gt; v1/Role NAME AGE k8s-prometheus-grafana-test 34s ==\u0026gt; v1/RoleBinding NAME AGE k8s-prometheus-grafana-test 34s ==\u0026gt; v1/Secret NAME TYPE DATA AGE alertmanager-k8s-prometheus-prometheus-alertmanager Opaque 1 34s k8s-prometheus-grafana Opaque 3 34s ==\u0026gt; v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE k8s-prometheus-grafana ClusterIP 10.100.203.67 \u0026lt;none\u0026gt; 80/TCP 33s k8s-prometheus-kube-state-metrics ClusterIP 10.98.35.173 \u0026lt;none\u0026gt; 8080/TCP 33s k8s-prometheus-prometheus-alertmanager ClusterIP 10.97.189.252 \u0026lt;none\u0026gt; 9093/TCP 33s k8s-prometheus-prometheus-coredns ClusterIP None \u0026lt;none\u0026gt; 9153/TCP 34s k8s-prometheus-prometheus-kube-controller-manager ClusterIP None \u0026lt;none\u0026gt; 10252/TCP 34s k8s-prometheus-prometheus-kube-etcd ClusterIP None \u0026lt;none\u0026gt; 2379/TCP 34s k8s-prometheus-prometheus-kube-proxy ClusterIP None \u0026lt;none\u0026gt; 10249/TCP 34s k8s-prometheus-prometheus-kube-scheduler ClusterIP None \u0026lt;none\u0026gt; 10251/TCP 34s k8s-prometheus-prometheus-node-exporter ClusterIP 10.100.146.95 \u0026lt;none\u0026gt; 9100/TCP 33s k8s-prometheus-prometheus-operator ClusterIP 10.105.23.58 \u0026lt;none\u0026gt; 8080/TCP,443/TCP 33s k8s-prometheus-prometheus-prometheus ClusterIP 10.99.7.136 \u0026lt;none\u0026gt; 9090/TCP 34s ==\u0026gt; v1/ServiceAccount NAME SECRETS AGE k8s-prometheus-grafana 1 34s k8s-prometheus-grafana-test 1 34s k8s-prometheus-kube-state-metrics 1 34s k8s-prometheus-prometheus-alertmanager 1 34s k8s-prometheus-prometheus-node-exporter 1 34s k8s-prometheus-prometheus-operator 1 34s k8s-prometheus-prometheus-prometheus 1 34s ==\u0026gt; v1/ServiceMonitor NAME AGE k8s-prometheus-prometheus-alertmanager 32s k8s-prometheus-prometheus-apiserver 32s k8s-prometheus-prometheus-coredns 32s k8s-prometheus-prometheus-grafana 32s k8s-prometheus-prometheus-kube-controller-manager 32s k8s-prometheus-prometheus-kube-etcd 32s k8s-prometheus-prometheus-kube-proxy 32s k8s-prometheus-prometheus-kube-scheduler 32s k8s-prometheus-prometheus-kube-state-metrics 32s k8s-prometheus-prometheus-kubelet 32s k8s-prometheus-prometheus-node-exporter 32s k8s-prometheus-prometheus-operator 32s k8s-prometheus-prometheus-prometheus 32s ==\u0026gt; v1beta1/ClusterRole NAME AGE k8s-prometheus-kube-state-metrics 34s ==\u0026gt; v1beta1/ClusterRoleBinding NAME AGE k8s-prometheus-kube-state-metrics 34s ==\u0026gt; v1beta1/MutatingWebhookConfiguration NAME AGE k8s-prometheus-prometheus-admission 33s ==\u0026gt; v1beta1/PodSecurityPolicy NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES k8s-prometheus-grafana false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim k8s-prometheus-grafana-test false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,downwardAPI,emptyDir,projected,secret k8s-prometheus-kube-state-metrics false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false secret k8s-prometheus-prometheus-alertmanager false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim k8s-prometheus-prometheus-node-exporter false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath k8s-prometheus-prometheus-operator false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim k8s-prometheus-prometheus-prometheus false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim ==\u0026gt; v1beta1/Role NAME AGE k8s-prometheus-grafana 34s ==\u0026gt; v1beta1/RoleBinding NAME AGE k8s-prometheus-grafana 34s ==\u0026gt; v1beta1/ValidatingWebhookConfiguration NAME AGE k8s-prometheus-prometheus-admission 32s NOTES: The Prometheus Operator has been installed. Check its status by running: kubectl --namespace monitoring get pods -l \u0026#34;release=k8s-prometheus\u0026#34; Visit https://github.com/coreos/prometheus-operator for instructions on how to create \u0026amp; configure Alertmanager and Prometheus instances using the Operator. 升级软件包 一般用于修改yaml配置文件重载应用\n$ helm upgrade k8s-prometheus /root/prometheus-operator/ -f /root/prometheus-operator/charts/grafana/values.yaml 删除 使用Helm的优势在于安装软件包和删除都非常方便，通过命令即可快速删除\n普通删除，会放入回收站\n$ helm delete k8s-prometheus 强制删除，不会放入回收站\n$ helm delete k8s-prometheus --purge 对于当前监控组件来说，删除后同时要删除CRD\n$ kubectl get crd | grep coreos 删除所有的CRD\n$ kubectl delete crd $(kubectl get crd | grep coreos | awk \u0026#39;{print $1}\u0026#39;) 访问指标面板 由于当前集群配置了ingress-nginx负载均衡，故新增两个配置文件即可访问Prometheus和Grafana\n访问Grafana\napiVersion: extensions/v1beta1 kind: Ingress metadata: name: prom-grafana namespace: monitoring spec: rules: - host: your.domain http: paths: - backend: serviceName: k8s-prometheus-grafana servicePort: 80 path: / 访问Prometheus\napiVersion: extensions/v1beta1 kind: Ingress metadata: name: prometheus namespace: monitoring spec: rules: - host: your.domain http: paths: - backend: serviceName: k8s-prometheus-prometheus-prometheus servicePort: web path: / Prometheus 访问Prometheus web站点，首页支持执行PromeQL查询采集指标信息\n访问target可见所有指标采集器都处于正常状态\n访问监控与告警可查看所有告警规则\nGrafana 首次访问会要求登陆\n查看用户名和密码 在grafana软件包的values.yaml配置中有如下描述，声明密码被存储在k8s的secret中\n# Administrator credentials when not using an existing secret (see below) adminUser: admin # adminPassword: strongpassword # Use an existing secret for the admin user. admin: existingSecret: \u0026#34;\u0026#34; userKey: admin-user passwordKey: admin-password 查看监控命名空间的秘钥\n$ kubectl get secrets -n monitoring NAME TYPE DATA AGE alertmanager-k8s-prometheus-prometheus-alertmanager Opaque 1 34m default-token-hrhgk kubernetes.io/service-account-token 3 2d10h etcd-certs Opaque 3 2d8h k8s-prometheus-grafana Opaque 3 34m k8s-prometheus-grafana-test-token-nnnvk kubernetes.io/service-account-token 3 34m k8s-prometheus-grafana-token-cdw2n kubernetes.io/service-account-token 3 34m k8s-prometheus-kube-state-metrics-token-6rxp4 kubernetes.io/service-account-token 3 34m k8s-prometheus-prometheus-admission Opaque 3 2d10h k8s-prometheus-prometheus-alertmanager-token-hxqm9 kubernetes.io/service-account-token 3 34m k8s-prometheus-prometheus-node-exporter-token-vjrcs kubernetes.io/service-account-token 3 34m k8s-prometheus-prometheus-operator-token-bkhrb kubernetes.io/service-account-token 3 34m k8s-prometheus-prometheus-prometheus-token-wf2v6 kubernetes.io/service-account-token 3 34m prometheus-k8s-prometheus-prometheus-prometheus Opaque 1 37m prometheus-k8s-prometheus-prometheus-prometheus-tls-assets Opaque 0 34m k8s-prometheus-grafana 即目标秘钥，展示其详情\n$ kubectl get secrets -n monitoring k8s-prometheus-grafana -o yaml apiVersion: v1 data: admin-password: b3BlcmF0b3ItYWRtaW4= admin-user: cm9vdA== ldap-toml: \u0026#34;\u0026#34; kind: Secret metadata: creationTimestamp: \u0026#34;2019-11-12T12:27:55Z\u0026#34; labels: app: grafana chart: grafana-4.0.2 heritage: Tiller release: k8s-prometheus name: k8s-prometheus-grafana namespace: monitoring resourceVersion: \u0026#34;1033230\u0026#34; selfLink: /api/v1/namespaces/monitoring/secrets/k8s-prometheus-grafana uid: da7e86d0-0547-11ea-a5d2-000c298872e2 type: Opaque 秘钥默认通过Base64 加密，通过在线网站对该Base64解密即可得到用户名和密码\n登录后可查看Grafana已经被配置好的多种可视化面板提供指标展示\n点击左上角即可查看已经配置好的指标展示面板\nEtcd监控面板\n节点监控\n计算资源监控\n","date":"2019-11-12T21:33:00+08:00","permalink":"/archives/220/","title":"k8s监控平台搭建"},{"content":"使用场景 为了进行规范化配置管理工作，基于gitlab规范分支管理流程，经常需要新建分支、设置保护分支和操作分支封板等，若只有几个工程则直接在gitlab web界面上操作即可，一旦工程数量增多，则会消耗大量时间，故考虑可通过gitlab 提供的web api编写脚本进行自动化操作\nApi指南 此处摘要部分官网API文档指南，所有api操作都需要在gitlab上申请访问令牌，该令牌应当至少包含api访问权限\n获取工程列表 支持多种参数过滤，非必传\nGet a list of all visible projects across GitLab for the authenticated user. When accessed without authentication, only public projects with “simple” fields are returned.\nGET /projects Attribute Type Required Description archived boolean no Limit by archived status visibility string no Limit by visibility public, internal, or private order_by string no Return projects ordered by id, name, path, created_at, updated_at, or last_activity_at fields. Default is created_at sort string no Return projects sorted in asc or desc order. Default is desc search string no Return list of projects matching the search criteria simple boolean no Return only limited fields for each project. This is a no-op without authentication as then only simple fields are returned. owned boolean no Limit by projects explicitly owned by the current user membership boolean no Limit by projects that the current user is a member of starred boolean no Limit by projects starred by the current user statistics boolean no Include project statistics with_custom_attributes boolean no Include custom attributes in response (admins only) with_issues_enabled boolean no Limit by enabled issues feature with_merge_requests_enabled boolean no Limit by enabled merge requests feature with_programming_language string no Limit by projects which use the given programming language wiki_checksum_failed boolean no Limit projects where the wiki checksum calculation has failed (Introduced in GitLab Premium 11.2) repository_checksum_failed boolean no Limit projects where the repository checksum calculation has failed (Introduced in GitLab Premium 11.2) min_access_level integer no Limit by current user minimal access level 分支操作 文档地址\n创建分支 需要传递仓库ID、源分支名、新分支名称\nCreate a new branch in the repository.\nPOST /projects/:id/repository/branches Parameters:\nAttribute Type Required Description id integer yes ID or URL-encoded path of the project owned by the authenticated user. branch string yes Name of the branch. ref string yes Branch name or commit SHA to create branch from. Example request:\ncurl --request POST --header \u0026#34;PRIVATE-TOKEN: \u0026lt;your_access_token\u0026gt;\u0026#34; https://gitlab.example.com/api/v4/projects/5/repository/branches?branch=newbranch\u0026amp;ref=master 保护分支 分支保护等级说明\nValid access levels\nThe access levels are defined in the ProtectedRefAccess.allowed_access_levels method. Currently, these levels are recognized:\n0 =\u0026gt; No access 30 =\u0026gt; Developer access 40 =\u0026gt; Maintainer access 60 =\u0026gt; Admin access 需要传递分支名称、访问级别等信息\nProtects a single repository branch or several project repository branches using a wildcard protected branch.\nPOST /projects/:id/protected_branches curl --request POST --header \u0026#34;PRIVATE-TOKEN: \u0026lt;your_access_token\u0026gt;\u0026#34; \u0026#39;https://gitlab.example.com/api/v4/projects/5/protected_branches?name=*-stable\u0026amp;push_access_level=30\u0026amp;merge_access_level=30\u0026amp;unprotect_access_level=40\u0026#39; Attribute Type Required Description id integer/string yes The ID or URL-encoded path of the project owned by the authenticated user name string yes The name of the branch or wildcard push_access_level string no Access levels allowed to push (defaults: 40, maintainer access level) merge_access_level string no Access levels allowed to merge (defaults: 40, maintainer access level) unprotect_access_level string no Access levels allowed to unprotect (defaults: 40, maintainer access level) allowed_to_push array no Array of access levels allowed to push, with each described by a hash allowed_to_merge array no Array of access levels allowed to merge, with each described by a hash allowed_to_unprotect array no Array of access levels allowed to unprotect, with each described by a hash code_owner_approval_required boolean no Prevent pushes to this branch if it matches an item in the CODEOWNERS file. (defaults: false) 取消保护分支 需要传递仓库ID、分支名称\nUnprotects the given protected branch or wildcard protected branch.\nDELETE /projects/:id/protected_branches/:name curl --request DELETE --header \u0026#34;PRIVATE-TOKEN: \u0026lt;your_access_token\u0026gt;\u0026#34; \u0026#39;https://gitlab.example.com/api/v4/projects/5/protected_branches/*-stable\u0026#39; Attribute Type Required Description id integer/string yes The ID or URL-encoded path of the project owned by the authenticated user name string yes The name of the branch 删除分支 需要传递仓库ID和分支名称\nDelete a branch from the repository.\nNote: In the case of an error, an explanation message is provided.\nDELETE /projects/:id/repository/branches/:branch Parameters:\nAttribute Type Required Description id integer/string yes ID or URL-encoded path of the project owned by the authenticated user. branch string yes Name of the branch. Example request:\ncurl --request DELETE --header \u0026#34;PRIVATE-TOKEN: \u0026lt;your_access_token\u0026gt;\u0026#34; https://gitlab.example.com/api/v4/projects/5/repository/branches/newbranch 接口测试 使用postman对接口进行测试\n获取工程列表 通过该接口可以查询所有工程的ID，注意，该接口默认使用了分页\n脚本编写 批量创建分支 定义工程ID数组，对数组进行遍历，并设置基础分支变量，方便调整，此处默认写死基础分支，有需要也可以通过变量传递\n#!/usr/bin/env bash # 访问令牌 PRIVATE_TOKEN=\u0026#34;XXXX\u0026#34; # 分支名称 branch_name=$1 base_branch=\u0026#34;BasicBranch\u0026#34; # 项目ID projectids=(47 48 50 51 52 54 55 56 58 59 60 62 63 64 65 67 68 69 70 71 74 75 89 90 91 92 93 95 103 104 113 114 115 143) for id in \u0026#34;${projectids[@]}\u0026#34; ; do curl --request POST --header \u0026#34;PRIVATE-TOKEN: $PRIVATE_TOKEN\u0026#34; \u0026#34;http://10.139.6.26:7077/api/v4/projects/$id/repository/branches?branch=$branch_name\u0026amp;ref=$base_branch\u0026#34; done 批量保护分支 此处写死了分支访问级别，可参照上文的参数进行配置\n#!/usr/bin/env bash # 访问令牌 PRIVATE_TOKEN=\u0026#34;XXXX\u0026#34; # 分支名称 branch_name=$1 # 项目ID projectids=(47 48 50 51 52 54 55 56 58 59 60 62 63 64 65 67 68 69 70 71 74 75 89 90 91 92 93 95 103 104 113 114 115 143) for id in \u0026#34;${projectids[@]}\u0026#34; ; do # 访问权限均为 Maintainer access curl --request POST --header \u0026#34;PRIVATE-TOKEN: $PRIVATE_TOKEN\u0026#34; \u0026#34;http://10.139.6.26:7077/api/v4/projects/$id/protected_branches?name=$branch_name\u0026amp;push_access_level=40\u0026amp;merge_access_level=40\u0026amp;unprotect_access_level=40\u0026#34; done 批量取消保护分支 调用对应的删除接口即可\n#!/usr/bin/env bash # 访问令牌 PRIVATE_TOKEN=\u0026#34;XXXX\u0026#34; # 分支名称 branch_name=$1 # 项目ID projectids=(47 48 50 51 52 54 55 56 58 59 60 62 63 64 65 67 68 69 70 71 74 75 89 90 91 92 93 95 103 104 113 114 115 143) for id in \u0026#34;${projectids[@]}\u0026#34; ; do # 访问权限均为 Maintainer access curl --request DELETE --header \u0026#34;PRIVATE-TOKEN: $PRIVATE_TOKEN\u0026#34; \u0026#34;http://10.139.6.26:7077/api/v4/projects/$id/protected_branches/$branch_name\u0026#34; done 批量分支封板 分支封板即设置访问级别为no one\n#!/usr/bin/env bash # 访问令牌 PRIVATE_TOKEN=\u0026#34;XXXX\u0026#34; # 分支名称 branch_name=$1 # 项目ID projectids=(47 48 50 51 52 54 55 56 58 59 60 62 63 64 65 67 68 69 70 71 74 75 89 90 91 92 93 95 103 104 113 114 115 143) for id in \u0026#34;${projectids[@]}\u0026#34; ; do # 分支封板 curl --request POST --header \u0026#34;PRIVATE-TOKEN: $PRIVATE_TOKEN\u0026#34; \u0026#34;http://10.139.6.26:7077/api/v4/projects/$id/protected_branches?name=$branch_name\u0026amp;push_access_level=0\u0026amp;merge_access_level=0\u0026amp;unprotect_access_level=40\u0026#34; done 批量删除分支 删除分支属于危险操作，操作需谨慎，因为不管是否为保护分支，都可以直接删除，而且通过api操作不会弹出确认操作\n#!/usr/bin/env bash # 访问令牌 PRIVATE_TOKEN=\u0026#34;XXXX\u0026#34; # 分支名称 branch_name=$1 # 项目ID projectids=(47 48 50 51 52 54 55 56 58 59 60 62 63 64 65 67 68 69 70 71 74 75 89 90 91 92 93 95 103 104 113 114 115 143) for id in \u0026#34;${projectids[@]}\u0026#34; ; do echo \u0026#34;删除项目ID：$id 的 $branch_name 分支\u0026#34; curl --request DELETE --header \u0026#34;PRIVATE-TOKEN: $PRIVATE_TOKEN\u0026#34; \u0026#34;http://10.139.6.26:7077/api/v4/projects/$id/repository/branches/$branch_name\u0026#34; done ","date":"2019-11-06T15:01:04+08:00","permalink":"/archives/219/","title":"GitLab Api使用"},{"content":"配置命令语法提示 执行命令查看指南\n$ kubectl completion -h Output shell completion code for the specified shell (bash or zsh). The shell code must be evaluated to provide interactive completion of kubectl commands. This can be done by sourcing it from the .bash_profile. Detailed instructions on how to do this are available here: https://kubernetes.io/docs/tasks/tools/install-kubectl/#enabling-shell-autocompletion Note for zsh users: [1] zsh completions are only supported in versions of zsh \u0026gt;= 5.2 Examples: # Installing bash completion on macOS using homebrew ## If running Bash 3.2 included with macOS brew install bash-completion ## or, if running Bash 4.1+ brew install bash-completion@2 ## If kubectl is installed via homebrew, this should start working immediately. ## If you\u0026#39;ve installed via other means, you may need add the completion to your completion directory kubectl completion bash \u0026gt; $(brew --prefix)/etc/bash_completion.d/kubectl # Installing bash completion on Linux ## If bash-completion is not installed on Linux, please install the \u0026#39;bash-completion\u0026#39; package ## via your distribution\u0026#39;s package manager. ## Load the kubectl completion code for bash into the current shell source \u0026lt;(kubectl completion bash) ## Write bash completion code to a file and source if from .bash_profile kubectl completion bash \u0026gt; ~/.kube/completion.bash.inc printf \u0026#34; # Kubectl shell completion source \u0026#39;$HOME/.kube/completion.bash.inc\u0026#39; \u0026#34; \u0026gt;\u0026gt; $HOME/.bash_profile source $HOME/.bash_profile # Load the kubectl completion code for zsh[1] into the current shell source \u0026lt;(kubectl completion zsh) # Set the kubectl completion code for zsh[1] to autoload on startup kubectl completion zsh \u0026gt; \u0026#34;${fpath[1]}/_kubectl\u0026#34; Usage: kubectl completion SHELL [options] Use \u0026#34;kubectl options\u0026#34; for a list of global command-line options (applies to all commands). 按照指南操作即可\n配置多个上下文环境 查看当前上下文配置\n$ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * minikube minikube minikube 编辑~/.kube/config文件新增需要的上下文配置即可，配置完成后查看上下文信息\n$ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE kubeadm kubeadm kubernetes-admin * minikube minikube minikube 切换上下文\n$ kubectl config use-context kubeadm Switched to context \u0026#34;kubeadm\u0026#34;. 再次查看，可见切换成功\n$ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * kubeadm kubeadm kubernetes-admin minikube minikube minikube Pod相关操作 默认查看default命名空间的Pod $ kubectl get pod NAME READY STATUS RESTARTS AGE nginx-deployment-54f57cf6bf-pznmh 1/1 Running 0 52s nginx-deployment-54f57cf6bf-r9q89 1/1 Running 0 52s 查看所有命名空间的Pod $ kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-58cc8c89f4-c297b 1/1 Running 0 10m kube-system coredns-58cc8c89f4-spzhj 1/1 Running 0 10m kube-system etcd-k8s-master 1/1 Running 0 9m57s kube-system kube-apiserver-k8s-master 1/1 Running 0 10m kube-system kube-controller-manager-k8s-master 1/1 Running 0 9m58s kube-system kube-flannel-ds-amd64-6fwg5 1/1 Running 0 5m kube-system kube-flannel-ds-amd64-6rchg 1/1 Running 0 9m12s kube-system kube-flannel-ds-amd64-z4plh 1/1 Running 0 4m54s kube-system kube-proxy-cdcs4 1/1 Running 0 5m kube-system kube-proxy-nlmbb 1/1 Running 0 4m54s kube-system kube-proxy-xlb62 1/1 Running 0 10m kube-system kube-scheduler-k8s-master 1/1 Running 0 10m 指定Pod对应的app名称 $ kubectl get pod -l app=nginx NAME READY STATUS RESTARTS AGE nginx-deployment-54f57cf6bf-pznmh 1/1 Running 0 58s nginx-deployment-54f57cf6bf-r9q89 1/1 Running 0 58s 查看更多信息 $ kubectl get deployment -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR nginx-deployment 2/2 2 2 64s nginx nginx:1.7.9 app=nginx 查看Pod详细信息 $ kubectl describe pod Name: nginx-deployment-54f57cf6bf-67llv Namespace: default Priority: 0 Node: k8s-node1/10.0.2.15 Start Time: Sun, 20 Oct 2019 10:19:42 +0800 Labels: app=nginx pod-template-hash=54f57cf6bf Annotations: \u0026lt;none\u0026gt; Status: Running IP: 172.100.1.3 IPs: IP: 172.100.1.3 Controlled By: ReplicaSet/nginx-deployment-54f57cf6bf Containers: nginx: Container ID: docker://ffd4b4fa3fd4678a61797cfdcba31ea2b0b36459b65a95100484eda4be286b9f Image: nginx:1.7.9 Image ID: docker-pullable://nginx@sha256:e3456c851a152494c3e4ff5fcc26f240206abac0c9d794affb40e0714846c451 Port: 80/TCP Host Port: 0/TCP State: Running Started: Sun, 20 Oct 2019 10:19:45 +0800 Ready: True Restart Count: 0 Environment: \u0026lt;none\u0026gt; Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-zjpq9 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-zjpq9: Type: Secret (a volume populated by a Secret) SecretName: default-token-zjpq9 Optional: false QoS Class: BestEffort Node-Selectors: \u0026lt;none\u0026gt; Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled \u0026lt;unknown\u0026gt; default-scheduler Successfully assigned default/nginx-deployment-54f57cf6bf-67llv to k8s-node1 Normal Pulled 46s kubelet, k8s-node1 Container image \u0026#34;nginx:1.7.9\u0026#34; already present on machine Normal Created 46s kubelet, k8s-node1 Created container nginx Normal Started 45s kubelet, k8s-node1 Started container nginx Name: nginx-deployment-54f57cf6bf-r9q89 Namespace: default Priority: 0 Node: k8s-node2/10.0.2.15 Start Time: Sun, 20 Oct 2019 10:17:24 +0800 Labels: app=nginx pod-template-hash=54f57cf6bf Annotations: \u0026lt;none\u0026gt; Status: Running IP: 172.100.2.3 IPs: IP: 172.100.2.3 Controlled By: ReplicaSet/nginx-deployment-54f57cf6bf Containers: nginx: Container ID: docker://61894b3762d723eabe2510284bb107098bdbcb8da4f841eea863a9c7a59c5806 Image: nginx:1.7.9 Image ID: docker-pullable://nginx@sha256:e3456c851a152494c3e4ff5fcc26f240206abac0c9d794affb40e0714846c451 Port: 80/TCP Host Port: 0/TCP State: Running Started: Sun, 20 Oct 2019 10:18:08 +0800 Ready: True Restart Count: 0 Environment: \u0026lt;none\u0026gt; Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-zjpq9 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-zjpq9: Type: Secret (a volume populated by a Secret) SecretName: default-token-zjpq9 Optional: false QoS Class: BestEffort Node-Selectors: \u0026lt;none\u0026gt; Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled \u0026lt;unknown\u0026gt; default-scheduler Successfully assigned default/nginx-deployment-54f57cf6bf-r9q89 to k8s-node2 Normal Pulling 3m4s kubelet, k8s-node2 Pulling image \u0026#34;nginx:1.7.9\u0026#34; Normal Pulled 2m24s kubelet, k8s-node2 Successfully pulled image \u0026#34;nginx:1.7.9\u0026#34; Normal Created 2m22s kubelet, k8s-node2 Created container nginx Normal Started 2m22s kubelet, k8s-node2 Started container nginx Deployment相关操作 部署Deployment $ kubectl create -f nginx_deployment.yml deployment.apps/nginx-deployment created 升级Deployment apply兼有create的语义，故也可直接使用apply创建\\部署deployment\n$ kubectl apply -f nginx_deployment_scale.yml deployment.apps/nginx-deployment configured 查看Deployment更多信息 $ kubectl get deployment -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR nginx-deployment 2/2 2 2 5m28s nginx nginx:1.8 app=nginx 查看详细信息 $ kubectl describe deployments.apps Name: nginx-deployment Namespace: default CreationTimestamp: Sun, 20 Oct 2019 10:17:24 +0800 Labels: \u0026lt;none\u0026gt; Annotations: deployment.kubernetes.io/revision: 3 kubectl.kubernetes.io/last-applied-configuration: {\u0026#34;apiVersion\u0026#34;:\u0026#34;apps/v1\u0026#34;,\u0026#34;kind\u0026#34;:\u0026#34;Deployment\u0026#34;,\u0026#34;metadata\u0026#34;:{\u0026#34;annotations\u0026#34;:{},\u0026#34;name\u0026#34;:\u0026#34;nginx-deployment\u0026#34;,\u0026#34;namespace\u0026#34;:\u0026#34;default\u0026#34;},\u0026#34;spec\u0026#34;:{\u0026#34;replica... Selector: app=nginx Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=nginx Containers: nginx: Image: nginx:1.9.1 Port: 80/TCP Host Port: 0/TCP Environment: \u0026lt;none\u0026gt; Mounts: \u0026lt;none\u0026gt; Volumes: \u0026lt;none\u0026gt; Conditions: Type Status Reason ---- ------ ------ Progressing True NewReplicaSetAvailable Available True MinimumReplicasAvailable OldReplicaSets: \u0026lt;none\u0026gt; NewReplicaSet: nginx-deployment-56f8998dbc (2/2 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 22m deployment-controller Scaled up replica set nginx-deployment-54f57cf6bf to 2 Normal ScalingReplicaSet 18m deployment-controller Scaled up replica set nginx-deployment-9f46bb5 to 1 Normal ScalingReplicaSet 17m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 1 Normal ScalingReplicaSet 17m deployment-controller Scaled up replica set nginx-deployment-9f46bb5 to 2 Normal ScalingReplicaSet 16m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 0 Normal ScalingReplicaSet 15m deployment-controller Scaled up replica set nginx-deployment-9f46bb5 to 4 Normal ScalingReplicaSet 12m deployment-controller Scaled down replica set nginx-deployment-9f46bb5 to 3 Normal ScalingReplicaSet 6m25s deployment-controller Scaled up replica set nginx-deployment-56f8998dbc to 1 Normal ScalingReplicaSet 5m43s deployment-controller Scaled down replica set nginx-deployment-9f46bb5 to 2 Normal ScalingReplicaSet 63s (x6 over 5m43s) deployment-controller (combined from similar events): Scaled down replica set nginx-deployment-56f8998dbc to 2 Node相关 查看Node $ kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master Ready master 24m v1.16.2 k8s-node1 Ready \u0026lt;none\u0026gt; 18m v1.16.2 k8s-node2 Ready \u0026lt;none\u0026gt; 18m v1.16.2 查看更多信息 $ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-master Ready master 24m v1.16.2 10.0.2.15 \u0026lt;none\u0026gt; CentOS Linux 7 (Core) 3.10.0-957.12.2.el7.x86_64 docker://18.9.8 k8s-node1 Ready \u0026lt;none\u0026gt; 18m v1.16.2 10.0.2.15 \u0026lt;none\u0026gt; CentOS Linux 7 (Core) 3.10.0-957.12.2.el7.x86_64 docker://18.9.8 k8s-node2 Ready \u0026lt;none\u0026gt; 18m v1.16.2 10.0.2.15 \u0026lt;none\u0026gt; CentOS Linux 7 (Core) 3.10.0-957.12.2.el7.x86_64 docker://18.9.8 查看标签 $ kubectl get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS k8s-master Ready master 24m v1.16.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master,kubernetes.io/os=linux,node-role.kubernetes.io/master= k8s-node1 Ready \u0026lt;none\u0026gt; 18m v1.16.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node1,kubernetes.io/os=linux k8s-node2 Ready \u0026lt;none\u0026gt; 18m v1.16.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node2,kubernetes.io/os=linux 设置角色标签 $ kubectl label nodes k8s-node1 node-role.kubernetes.io/worker= node/k8s-node1 labeled 查看详细信息 $ kubectl describe nodes Name: k8s-master Roles: master Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=k8s-master kubernetes.io/os=linux node-role.kubernetes.io/master= Annotations: flannel.alpha.coreos.com/backend-data: {\u0026#34;VtepMAC\u0026#34;:\u0026#34;32:38:c5:18:f0:ed\u0026#34;} flannel.alpha.coreos.com/backend-type: vxlan flannel.alpha.coreos.com/kube-subnet-manager: true flannel.alpha.coreos.com/public-ip: 10.0.2.15 kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Sun, 20 Oct 2019 10:05:02 +0800 Taints: node-role.kubernetes.io/master:NoSchedule Unschedulable: false Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Sun, 20 Oct 2019 10:27:46 +0800 Sun, 20 Oct 2019 10:05:02 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Sun, 20 Oct 2019 10:27:46 +0800 Sun, 20 Oct 2019 10:05:02 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Sun, 20 Oct 2019 10:27:46 +0800 Sun, 20 Oct 2019 10:05:02 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Sun, 20 Oct 2019 10:27:46 +0800 Sun, 20 Oct 2019 10:07:42 +0800 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.0.2.15 Hostname: k8s-master Capacity: cpu: 2 ephemeral-storage: 41921540Ki hugepages-2Mi: 0 memory: 1882144Ki pods: 110 Allocatable: cpu: 2 ephemeral-storage: 38634891201 hugepages-2Mi: 0 memory: 1779744Ki pods: 110 System Info: Machine ID: d74c50274729456aad2a6198de9863b3 System UUID: D74C5027-4729-456A-AD2A-6198DE9863B3 Boot ID: fb2ea7bd-afb2-4051-afb7-78d9f59ec6f6 Kernel Version: 3.10.0-957.12.2.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: docker://18.9.8 Kubelet Version: v1.16.2 Kube-Proxy Version: v1.16.2 PodCIDR: 172.100.0.0/24 PodCIDRs: 172.100.0.0/24 Non-terminated Pods: (8 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- kube-system coredns-58cc8c89f4-c297b 100m (5%) 0 (0%) 70Mi (4%) 170Mi (9%) 23m kube-system coredns-58cc8c89f4-spzhj 100m (5%) 0 (0%) 70Mi (4%) 170Mi (9%) 23m kube-system etcd-k8s-master 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22m kube-system kube-apiserver-k8s-master 250m (12%) 0 (0%) 0 (0%) 0 (0%) 22m kube-system kube-controller-manager-k8s-master 200m (10%) 0 (0%) 0 (0%) 0 (0%) 22m kube-system kube-flannel-ds-amd64-6rchg 100m (5%) 100m (5%) 50Mi (2%) 50Mi (2%) 21m kube-system kube-proxy-xlb62 0 (0%) 0 (0%) 0 (0%) 0 (0%) 23m kube-system kube-scheduler-k8s-master 100m (5%) 0 (0%) 0 (0%) 0 (0%) 22m Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 850m (42%) 100m (5%) memory 190Mi (10%) 390Mi (22%) ephemeral-storage 0 (0%) 0 (0%) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NodeHasSufficientMemory 23m (x8 over 23m) kubelet, k8s-master Node k8s-master status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 23m (x8 over 23m) kubelet, k8s-master Node k8s-master status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 23m (x7 over 23m) kubelet, k8s-master Node k8s-master status is now: NodeHasSufficientPID Normal Starting 23m kube-proxy, k8s-master Starting kube-proxy. Name: k8s-node1 Roles: \u0026lt;none\u0026gt; Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=k8s-node1 kubernetes.io/os=linux Annotations: flannel.alpha.coreos.com/backend-data: {\u0026#34;VtepMAC\u0026#34;:\u0026#34;16:5c:40:b1:14:b6\u0026#34;} flannel.alpha.coreos.com/backend-type: vxlan flannel.alpha.coreos.com/kube-subnet-manager: true flannel.alpha.coreos.com/public-ip: 10.0.2.15 kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Sun, 20 Oct 2019 10:11:03 +0800 Taints: \u0026lt;none\u0026gt; Unschedulable: false Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Sun, 20 Oct 2019 10:28:08 +0800 Sun, 20 Oct 2019 10:11:03 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Sun, 20 Oct 2019 10:28:08 +0800 Sun, 20 Oct 2019 10:11:03 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Sun, 20 Oct 2019 10:28:08 +0800 Sun, 20 Oct 2019 10:11:03 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Sun, 20 Oct 2019 10:28:08 +0800 Sun, 20 Oct 2019 10:11:34 +0800 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.0.2.15 Hostname: k8s-node1 Capacity: cpu: 1 ephemeral-storage: 41921540Ki hugepages-2Mi: 0 memory: 1882296Ki pods: 110 Allocatable: cpu: 1 ephemeral-storage: 38634891201 hugepages-2Mi: 0 memory: 1779896Ki pods: 110 System Info: Machine ID: 5d806dde9b0444bdb5c9800a228267d8 System UUID: 5D806DDE-9B04-44BD-B5C9-800A228267D8 Boot ID: e6b7222c-686c-49cb-8e56-9bb263852d36 Kernel Version: 3.10.0-957.12.2.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: docker://18.9.8 Kubelet Version: v1.16.2 Kube-Proxy Version: v1.16.2 PodCIDR: 172.100.1.0/24 PodCIDRs: 172.100.1.0/24 Non-terminated Pods: (4 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- default nginx-deployment-9f46bb5-7vlb7 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m53s default nginx-deployment-9f46bb5-c7czx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6m35s kube-system kube-flannel-ds-amd64-6fwg5 100m (10%) 100m (10%) 50Mi (2%) 50Mi (2%) 17m kube-system kube-proxy-cdcs4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 17m Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 100m (10%) 100m (10%) memory 50Mi (2%) 50Mi (2%) ephemeral-storage 0 (0%) 0 (0%) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Starting 17m kubelet, k8s-node1 Starting kubelet. Normal NodeHasSufficientMemory 17m kubelet, k8s-node1 Node k8s-node1 status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 17m kubelet, k8s-node1 Node k8s-node1 status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 17m kubelet, k8s-node1 Node k8s-node1 status is now: NodeHasSufficientPID Normal NodeAllocatableEnforced 17m kubelet, k8s-node1 Updated Node Allocatable limit across pods Normal Starting 17m kube-proxy, k8s-node1 Starting kube-proxy. Normal NodeReady 16m kubelet, k8s-node1 Node k8s-node1 status is now: NodeReady Name: k8s-node2 Roles: \u0026lt;none\u0026gt; Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=k8s-node2 kubernetes.io/os=linux Annotations: flannel.alpha.coreos.com/backend-data: {\u0026#34;VtepMAC\u0026#34;:\u0026#34;76:49:24:1f:e5:d6\u0026#34;} flannel.alpha.coreos.com/backend-type: vxlan flannel.alpha.coreos.com/kube-subnet-manager: true flannel.alpha.coreos.com/public-ip: 10.0.2.15 kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Sun, 20 Oct 2019 10:11:09 +0800 Taints: \u0026lt;none\u0026gt; Unschedulable: false Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Sun, 20 Oct 2019 10:27:42 +0800 Sun, 20 Oct 2019 10:11:09 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Sun, 20 Oct 2019 10:27:42 +0800 Sun, 20 Oct 2019 10:11:09 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Sun, 20 Oct 2019 10:27:42 +0800 Sun, 20 Oct 2019 10:11:09 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Sun, 20 Oct 2019 10:27:42 +0800 Sun, 20 Oct 2019 10:11:40 +0800 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.0.2.15 Hostname: k8s-node2 Capacity: cpu: 1 ephemeral-storage: 41921540Ki hugepages-2Mi: 0 memory: 1882296Ki pods: 110 Allocatable: cpu: 1 ephemeral-storage: 38634891201 hugepages-2Mi: 0 memory: 1779896Ki pods: 110 System Info: Machine ID: 7c59ba12638e4fdfa20cfeda0aa2fda1 System UUID: 7C59BA12-638E-4FDF-A20C-FEDA0AA2FDA1 Boot ID: e402f75c-42e7-454d-aaca-f96853091c15 Kernel Version: 3.10.0-957.12.2.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: docker://18.9.8 Kubelet Version: v1.16.2 Kube-Proxy Version: v1.16.2 PodCIDR: 172.100.2.0/24 PodCIDRs: 172.100.2.0/24 Non-terminated Pods: (3 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- default nginx-deployment-9f46bb5-mv4b7 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5m52s kube-system kube-flannel-ds-amd64-z4plh 100m (10%) 100m (10%) 50Mi (2%) 50Mi (2%) 17m kube-system kube-proxy-nlmbb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 17m Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 100m (10%) 100m (10%) memory 50Mi (2%) 50Mi (2%) ephemeral-storage 0 (0%) 0 (0%) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Starting 17m kubelet, k8s-node2 Starting kubelet. Normal NodeHasSufficientMemory 17m kubelet, k8s-node2 Node k8s-node2 status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 17m kubelet, k8s-node2 Node k8s-node2 status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 17m kubelet, k8s-node2 Node k8s-node2 status is now: NodeHasSufficientPID Normal NodeAllocatableEnforced 17m kubelet, k8s-node2 Updated Node Allocatable limit across pods Normal Starting 17m kube-proxy, k8s-node2 Starting kube-proxy. Normal NodeReady 16m kubelet, k8s-node2 Node k8s-node2 status is now: NodeReady 服务伸缩 scale命令方式 $ kubectl scale --replicas=6 deployment/nginx-deployment deployment.apps/nginx-deployment scaled 查看副本数 $ kubectl get replicaset NAME DESIRED CURRENT READY AGE nginx-deployment-54f57cf6bf 0 0 0 21m nginx-deployment-56f8998dbc 6 6 3 5m7s nginx-deployment-9f46bb5 0 0 0 16m ","date":"2019-10-20T13:00:00+08:00","image":"https://s2.ax1x.com/2019/11/27/QC2UuF.png","permalink":"/archives/218/","title":"Kubernetes基础命令学习"},{"content":"搭建方法分类 如果是搭建Kubernetes的学习环境，则可以直接使用minikube快速搭建单节点的Kubernetes环境，官方推荐使用kubeadm搭建Kubernetes集群生产环境\n但kubeadm需要连接谷歌容器仓库获取镜像，在网络受限的情况下无法搭建成功，故有两种搭建方法：\n服务器使用代理，代理所有的http和https连接，可以自由访问任意地址 使用阿里云镜像地址 本次采用第二种方式进行搭建\n环境准备 本次宿主机为CentOS 7，结合3台virtual box虚拟机模拟集群环境，虚拟机使用Vagrant进行管理，vagrant安装比较简单，此处不再赘述\n宿主机配置：\nCPU：4C RAM：8G 虚拟机配置 由于网络受限，vagrant默认拉取的操作系统文件地址不可访问，需要自行从官方下载镜像或者使用代理下载镜像，此处对此操作也不再赘述\n三台虚拟机配置文件\n# -*- mode: ruby -*- # vi: set ft=ruby : Vagrant.require_version \u0026#34;\u0026gt;= 1.6.0\u0026#34; boxes = [ { :name =\u0026gt; \u0026#34;k8s-master\u0026#34;, :eth1 =\u0026gt; \u0026#34;192.168.205.120\u0026#34;, :mem =\u0026gt; \u0026#34;2048\u0026#34;, :cpu =\u0026gt; \u0026#34;2\u0026#34; }, { :name =\u0026gt; \u0026#34;k8s-node1\u0026#34;, :eth1 =\u0026gt; \u0026#34;192.168.205.121\u0026#34;, :mem =\u0026gt; \u0026#34;2048\u0026#34;, :cpu =\u0026gt; \u0026#34;1\u0026#34; }, { :name =\u0026gt; \u0026#34;k8s-node2\u0026#34;, :eth1 =\u0026gt; \u0026#34;192.168.205.122\u0026#34;, :mem =\u0026gt; \u0026#34;2048\u0026#34;, :cpu =\u0026gt; \u0026#34;1\u0026#34; } ] Vagrant.configure(2) do |config| config.vm.box = \u0026#34;centos/7\u0026#34; boxes.each do |opts| config.vm.define opts[:name] do |config| config.vm.hostname = opts[:name] config.vm.provider \u0026#34;vmware_fusion\u0026#34; do |v| v.vmx[\u0026#34;memsize\u0026#34;] = opts[:mem] v.vmx[\u0026#34;numvcpus\u0026#34;] = opts[:cpu] end config.vm.provider \u0026#34;virtualbox\u0026#34; do |v| v.customize [\u0026#34;modifyvm\u0026#34;, :id, \u0026#34;--memory\u0026#34;, opts[:mem]] v.customize [\u0026#34;modifyvm\u0026#34;, :id, \u0026#34;--cpus\u0026#34;, opts[:cpu]] end config.vm.network :private_network, ip: opts[:eth1] end end config.vm.provision \u0026#34;shell\u0026#34;, privileged: true, path: \u0026#34;./setup.sh\u0026#34; end 所有虚拟机启动后都会执行./setup.sh脚本配置基础环境，脚本内容如下\n#/bin/sh sudo yum install -y vim telnet bind-utils wget yum-utils sudo yum-config-manager \\ --add-repo \\ https://download.docker.com/linux/centos/docker-ce.repo sudo yum -y install docker-ce-18.09.8 docker-ce-cli-18.09.8 containerd.io if [ ! $(getent group docker) ]; then sudo groupadd docker; else echo \u0026#34;docker user group already exists\u0026#34; fi sudo gpasswd -a $USER docker sudo systemctl restart docker # open password auth for backup if ssh key doesn\u0026#39;t work, bydefault, username=vagrant password=vagrant sudo sed -i \u0026#39;s/PasswordAuthentication no/PasswordAuthentication yes/g\u0026#39; /etc/ssh/sshd_config sudo systemctl restart sshd sudomkdir /etc/yum.repos.d/bak \u0026amp;\u0026amp; mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/bak sudowget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.cloud.tencent.com/repo/centos7_base.repo sudowget -O /etc/yum.repos.d/epel.repo http://mirrors.cloud.tencent.com/repo/epel-7.repo yum clean all \u0026amp;\u0026amp; yum makecache sudo bash -c \u0026#39;cat \u0026lt;\u0026lt;EOF \u0026gt; /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF\u0026#39; sudo wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo sudo mkdir -p /etc/docker sudo tee /etc/docker/daemon.json \u0026lt;\u0026lt;-\u0026#39;EOF\u0026#39; { \u0026#34;registry-mirrors\u0026#34;: [\u0026#34;https://clayphwh.mirror.aliyuncs.com\u0026#34;] } EOF sudo systemctl daemon-reload sudo systemctl restart docker sudo setenforce 0 sudo yum install -y kubelet kubeadm kubectl sudo bash -c \u0026#39;cat \u0026lt;\u0026lt;EOF \u0026gt; /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward=1 EOF\u0026#39; sudo sysctl --system sudo systemctl stop firewalld sudo systemctl disable firewalld sudo swapoff -a sudo systemctl enable docker.service sudo systemctl enable kubelet.service 脚本内已将K8S镜像均配置为阿里云镜像仓库，并配置了阿里云容器加速服务地址，更便于在国内拉取镜像\n启动虚拟机 使用命令启动虚拟机\n$ vagrant up 启动需要较长时间，若出现大量超时也请耐心等待\nLoaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.163.com * updates: mirrors.163.com http://vault.centos.org/7.1.1503/os/x86_64/repodata/0e6e90965f55146ba5025ea450f822d1bb0267d0299ef64dd4365825e6bad995-c7-x86_64-comps.xml.gz: [Errno 12] Timeout on http://vault.centos.org/7.1.1503/os/x86_64/repodata/0e6e90965f55146ba5025ea450f822d1bb0267d0299ef64dd4365825e6bad995-c7-x86_64-comps.xml.gz: (28, \u0026#39;Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds\u0026#39;) Trying other mirror. http://vault.centos.org/7.1.1503/updates/x86_64/repodata/93b71f445d2ec2138d28152612f4fb29c8e76ee31f2666b964d88249b4e0a955-primary.sqlite.bz2: [Errno 12] Timeout on http://vault.centos.org/7.1.1503/updates/x86_64/repodata/93b71f445d2ec2138d28152612f4fb29c8e76ee31f2666b964d88249b4e0a955-primary.sqlite.bz2: (28, \u0026#39;Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds\u0026#39;) Trying other mirror. http://vault.centos.org/7.2.1511/os/x86_64/repodata/c6411f1cc8a000ed2b651b49134631d279abba1ec1f78e5dcca79a52d8c1eada-primary.sqlite.bz2: [Errno 12] Timeout on http://vault.centos.org/7.2.1511/os/x86_64/repodata/c6411f1cc8a000ed2b651b49134631d279abba1ec1f78e5dcca79a52d8c1eada-primary.sqlite.bz2: (28, \u0026#39;Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds\u0026#39;) Trying other mirror. ... 启动完成后通过命令查看虚拟机状态\n$ vagrant status Current machine states: k8s-master running (virtualbox) k8s-node1 running (virtualbox) k8s-node2 running (virtualbox) This environment represents multiple VMs. The VMs are all listed above with their current state. For more information about a specific VM, run `vagrant status NAME`. K8S master节点初始化 进入主节点，查看环境是否已经配置完成\n$ vagrant ssh k8s-master 执行以下三条语句查看是否有预期输出\n[vagrant@k8s-master ~]$ sudo which kubeadm /bin/kubeadm [vagrant@k8s-master ~]$ sudo which kubelet /bin/kubelet [vagrant@k8s-master ~]$ sudo which kubectl /bin/kubectl [vagrant@k8s-master ~]$ sudo docker version Client: Version: 18.09.8 API version: 1.39 Go version: go1.10.8 Git commit: 0dd43dd87f Built: Wed Jul 17 17:40:31 2019 OS/Arch: linux/amd64 Experimental: false Server: Docker Engine - Community Engine: Version: 18.09.8 API version: 1.39 (minimum version 1.12) Go version: go1.10.8 Git commit: 0dd43dd Built: Wed Jul 17 17:10:42 2019 OS/Arch: linux/amd64 Experimental: false 使用初始化命令，并指定拉取镜像地址为阿里云地址，同时指定容器网络地址和注册中心广播地址\n$ sudo kubeadm init --pod-network-cidr 172.100.0.0/16 --image-repository registry.aliyuncs.com/google_containers --apiserver-advertise-address 192.168.205.120 输出如下即初始化完成\n$ sudo kubeadm init --pod-network-cidr 172.100.0.0/16 --image-repository registry.aliyuncs.com/google_containers --apiserver-advertise-address 192.168.205.120 W1020 01:31:27.560129 20473 version.go:101] could not fetch a `Kubernetes`version from the internet: unable to get URL \u0026#34;https://dl.k8s.io/release/stable-1.txt\u0026#34;: Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) W1020 01:31:27.560713 20473 version.go:102] falling back to the local client version: v1.16.2 [init] Using `Kubernetes`version: v1.16.2 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected \u0026#34;cgroupfs\u0026#34; as the Docker cgroup driver. The recommended driver is \u0026#34;systemd\u0026#34;. Please follow the guide at https://kubernetes.io/docs/setup/cri/ [preflight] Pulling images required for setting up a `Kubernetes`cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using \u0026#39;kubeadm config images pull\u0026#39; [kubelet-start] Writing kubelet environment file with flags to file \u0026#34;/var/lib/kubelet/kubeadm-flags.env\u0026#34; [kubelet-start] Writing kubelet configuration to file \u0026#34;/var/lib/kubelet/config.yaml\u0026#34; [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder \u0026#34;/etc/kubernetes/pki\u0026#34; [certs] Generating \u0026#34;ca\u0026#34; certificate and key [certs] Generating \u0026#34;apiserver\u0026#34; certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master `kubernetes`kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.205.120] [certs] Generating \u0026#34;apiserver-kubelet-client\u0026#34; certificate and key [certs] Generating \u0026#34;front-proxy-ca\u0026#34; certificate and key [certs] Generating \u0026#34;front-proxy-client\u0026#34; certificate and key [certs] Generating \u0026#34;etcd/ca\u0026#34; certificate and key [certs] Generating \u0026#34;etcd/server\u0026#34; certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.205.120 127.0.0.1 ::1] [certs] Generating \u0026#34;etcd/peer\u0026#34; certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.205.120 127.0.0.1 ::1] [certs] Generating \u0026#34;etcd/healthcheck-client\u0026#34; certificate and key [certs] Generating \u0026#34;apiserver-etcd-client\u0026#34; certificate and key [certs] Generating \u0026#34;sa\u0026#34; key and public key [kubeconfig] Using kubeconfig folder \u0026#34;/etc/kubernetes\u0026#34; [kubeconfig] Writing \u0026#34;admin.conf\u0026#34; kubeconfig file [kubeconfig] Writing \u0026#34;kubelet.conf\u0026#34; kubeconfig file [kubeconfig] Writing \u0026#34;controller-manager.conf\u0026#34; kubeconfig file [kubeconfig] Writing \u0026#34;scheduler.conf\u0026#34; kubeconfig file [control-plane] Using manifest folder \u0026#34;/etc/kubernetes/manifests\u0026#34; [control-plane] Creating static Pod manifest for \u0026#34;kube-apiserver\u0026#34; [control-plane] Creating static Pod manifest for \u0026#34;kube-controller-manager\u0026#34; [control-plane] Creating static Pod manifest for \u0026#34;kube-scheduler\u0026#34; [etcd] Creating static Pod manifest for local etcd in \u0026#34;/etc/kubernetes/manifests\u0026#34; [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory \u0026#34;/etc/kubernetes/manifests\u0026#34;. This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed. [apiclient] All control plane components are healthy after 59.511578 seconds [upload-config] Storing the configuration used in ConfigMap \u0026#34;kubeadm-config\u0026#34; in the \u0026#34;kube-system\u0026#34; Namespace [kubelet] Creating a ConfigMap \u0026#34;kubelet-config-1.16\u0026#34; in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node k8s-master as control-plane by adding the label \u0026#34;node-role.kubernetes.io/master=\u0026#39;\u0026#39;\u0026#34; [mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: rod570.p67pymzbil4m6u8d [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the \u0026#34;cluster-info\u0026#34; ConfigMap in the \u0026#34;kube-public\u0026#34; namespace [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your `Kubernetes`control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run \u0026#34;kubectl apply -f [podnetwork].yaml\u0026#34; with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.205.120:6443 --token rod570.p67pymzbil4m6u8d \\ --discovery-token-ca-cert-hash sha256:8f4cb9d555c78d58befeb3cfa3f7537989aa599e53e4f1bae929d8cc7afd1476 kubectl 配置 拷贝kubectl配置\n$ rm -rf ~/.kube $ mkdir -p $HOME/.kube $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config $ sudo chown $(id -u):$(id -g) $HOME/.kube/config 若不配置直接使用则报无法连接错误\n[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces The connection to the server localhost:8080 was refused - did you specify the right host or port? 此时使用kubectl可查看k8s容器运行情况，输出如下则为初始化成功，启动由于未配置网络插件，故coredns一直处于Pending状态\n[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-58cc8c89f4-c297b 0/1 Pending 0 81s kube-system coredns-58cc8c89f4-spzhj 0/1 Pending 0 81s kube-system etcd-k8s-master 1/1 Running 0 39s kube-system kube-apiserver-k8s-master 1/1 Running 0 49s kube-system kube-controller-manager-k8s-master 1/1 Running 0 40s kube-system kube-proxy-xlb62 1/1 Running 0 81s kube-system kube-scheduler-k8s-master 1/1 Running 0 51s 安装网络插件 网络插件有多个可选，可通过官网查看，此处选择安装flannel CNI网络插件\nFor flannel to work correctly, you must pass --pod-network-cidr=10.244.0.0/16 to kubeadm init.\nSet /proc/sys/net/bridge/bridge-nf-call-iptables to 1 by running sysctl net.bridge.bridge-nf-call-iptables=1 to pass bridged IPv4 traffic to iptables’ chains. This is a requirement for some CNI plugins to work, for more information please see here.\nMake sure that your firewall rules allow UDP ports 8285 and 8472 traffic for all hosts participating in the overlay network. see here .\nNote that flannel works on amd64, arm, arm64, ppc64le and s390x under Linux. Windows (amd64) is claimed as supported in v0.11.0 but the usage is undocumented.\nkubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml For more information about flannel, see the CoreOS flannel repository on GitHub .\n相关的网络配置已在脚本中写入，此时直接执行yml脚本即可\n$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml 再次查看容器运行情况，所有容器均已运行，则master节点配置完成\n[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-58cc8c89f4-c297b 1/1 Running 0 5m17s kube-system coredns-58cc8c89f4-spzhj 1/1 Running 0 5m17s kube-system etcd-k8s-master 1/1 Running 0 4m35s kube-system kube-apiserver-k8s-master 1/1 Running 0 4m45s kube-system kube-controller-manager-k8s-master 1/1 Running 0 4m36s kube-system kube-flannel-ds-amd64-6rchg 1/1 Running 0 3m50s kube-system kube-proxy-xlb62 1/1 Running 0 5m17s kube-system kube-scheduler-k8s-master 1/1 Running 0 4m47s 配置worker节点 以k8s-node1为例，执行master初始化时提供的脚本即可加入k8s集群，输出如下则加入成功\n[vagrant@k8s-node1 ~]$ sudo kubeadm join 192.168.205.120:6443 --token 8ry5oo.y48ksgurn103zq4h \\ \u0026gt; --discovery-token-ca-cert-hash sha256:ead07352591500c2cfe3321bf87d2e068790e16f2a7e0cc23541c864d24006d4 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected \u0026#34;cgroupfs\u0026#34; as the Docker cgroup driver. The recommended driver is \u0026#34;systemd\u0026#34;. Please follow the guide at https://kubernetes.io/docs/setup/cri/ [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with \u0026#39;kubectl -n kube-system get cm kubeadm-config -oyaml\u0026#39; [kubelet-start] Downloading configuration for the kubelet from the \u0026#34;kubelet-config-1.16\u0026#34; ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file \u0026#34;/var/lib/kubelet/config.yaml\u0026#34; [kubelet-start] Writing kubelet environment file with flags to file \u0026#34;/var/lib/kubelet/kubeadm-flags.env\u0026#34; [kubelet-start] Activating the kubelet service [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run \u0026#39;kubectl get nodes\u0026#39; on the control-plane to see this node join the cluster. 稍等片刻进入master节点查看容器运行情况，会发现多了几个网络容器，属于预期情况\n[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-58cc8c89f4-c297b 1/1 Running 0 6m28s kube-system coredns-58cc8c89f4-spzhj 1/1 Running 0 6m28s kube-system etcd-k8s-master 1/1 Running 0 5m46s kube-system kube-apiserver-k8s-master 1/1 Running 0 5m56s kube-system kube-controller-manager-k8s-master 1/1 Running 0 5m47s kube-system kube-flannel-ds-amd64-6fwg5 1/1 Running 0 49s kube-system kube-flannel-ds-amd64-6rchg 1/1 Running 0 5m1s kube-system kube-flannel-ds-amd64-z4plh 1/1 Running 0 43s kube-system kube-proxy-cdcs4 1/1 Running 0 49s kube-system kube-proxy-nlmbb 1/1 Running 0 43s kube-system kube-proxy-xlb62 1/1 Running 0 6m28s kube-system kube-scheduler-k8s-master 1/1 Running 0 5m58s 集群验证 在master节点中启动nginx容器测试集群情况\n$ kubectl create deployment nginx --image=nginx 稍等片刻，输出如下则集群正常运行，k8s集群配置完成\n[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default nginx-86c57db685-2gkxm 1/1 Running 0 58s kube-system coredns-58cc8c89f4-c297b 1/1 Running 0 7m46s kube-system coredns-58cc8c89f4-spzhj 1/1 Running 0 7m46s kube-system etcd-k8s-master 1/1 Running 0 7m4s kube-system kube-apiserver-k8s-master 1/1 Running 0 7m14s kube-system kube-controller-manager-k8s-master 1/1 Running 0 7m5s kube-system kube-flannel-ds-amd64-6fwg5 1/1 Running 0 2m7s kube-system kube-flannel-ds-amd64-6rchg 1/1 Running 0 6m19s kube-system kube-flannel-ds-amd64-z4plh 1/1 Running 0 2m1s kube-system kube-proxy-cdcs4 1/1 Running 0 2m7s kube-system kube-proxy-nlmbb 1/1 Running 0 2m1s kube-system kube-proxy-xlb62 1/1 Running 0 7m46s kube-system kube-scheduler-k8s-master 1/1 Running 0 7m16s ","date":"2019-10-20T12:31:00+08:00","image":"https://s2.ax1x.com/2019/11/27/QCRTo9.jpg","permalink":"/archives/217/","title":"Kubernetes 集群搭建"},{"content":"介绍 作为Elastic Stack 的核心，Elasticsearch 是一个分布式、RESTful 风格的搜索和数据分析引擎，本次主要用于存储业务数据和基于业务日志进行数据统计分析。\n目标 本次目标是搭建个人项目所使用的小型es高可用环境，目前需要使用es的服务有些部署在美国的服务器，有些部署在国内的服务器，带宽延迟各不相同，为了保证应用连接es延迟最小，计划同时在美国和中国都部署节点，不同区域的应用连接不同的节点\n基础环境 本次配置全局使用真实服务器，服务器配置如下\n名称 CPU 内存 系统 Docker版本 磁盘大小 区域 es01 1 2G CentOS 7 19.03.1 40 G 亚洲 es02 1 2G CentOS 7 19.03.1 20G 美洲 es03 4 4G CentOS 7 19.03.1 36G 美洲 es04 1 2G CentOS 7 19.03.1 50G 亚洲 所有服务器部署的es版本均为：6.6.2\n从表中可见内存和磁盘都不大，如果是公司项目配置肯定比这个好很多，但对于个人使用来说足够了，至于为什么是需要4台而不是3台在最后搭建完成后会进行分析\n部署前准备 就算使用了Docker容器，elasticsearch仍然不像普通镜像那么简单启动，es对虚拟内存敏感，因此服务器必须是内核虚拟化KVM架构，不支持OpenVZ虚拟，参考官方说明\nProduction modeedit The vm.max_map_count kernel setting needs to be set to at least 262144 for production use. Depending on your platform:\nLinux\nThe vm.max_map_count setting should be set permanently in /etc/sysctl.conf:\n$ grep vm.max_map_count /etc/sysctl.conf vm.max_map_count=262144 To apply the setting on a live system type: sysctl -w vm.max_map_count=262144\nmacOS with Docker for Mac\nThe vm.max_map_count setting must be set within the xhyve virtual machine:\n$ screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty Just press enter and configure the sysctl setting as you would for Linux:\nsysctl -w vm.max_map_count=262144 按照指引设置内存参数 $ sudo sysctl -w vm.max_map_count=262144 前置知识 裂脑事件 Elasticsearch牺牲了一致性，以确保可用性和分区容错。其背后的原因是短期的不良行为比短期的不可用性问题少。换句话说，当群集中的Elasticsearch节点无法复制对数据的更改时，它们将继续为应用程序提供服务。当节点能够复制其数据时，它们将尝试聚合副本并实现最终的一致性。\nElasticsearch通过选举主节点来解决之前的问题，主节点负责数据库操作，例如创建新索引，在群集节点周围移动分片等等。主节点与其他节点主动协调其操作，确保数据可以由非主节点汇聚。\n在某些情况下，先前的机制可能会失败，从而导致裂脑事件。当Elasticsearch集群分为两个区块时，若每个区块都有一个节点都认为它们是主节点，因为主节点将在数据上独立工作，数据一致性就会丢失。因此，节点将对相同的查询做出不同的响应。这将会是灾难性的事件，因为来自两个主节点的数据无法自动重新加入，并且需要相当多的手动工作来纠正这种情况。\n节点部署 3个主节点节点均使用docker-compose命令部署，需要自行安装好相关环境\nversion: \u0026#39;2\u0026#39; services: es01: image: docker.elastic.co/elasticsearch/elasticsearch:6.6.2 container_name: es01 environment: - cluster.name=docker-cluster - bootstrap.memory_lock=true - \u0026#34;ES_JAVA_OPTS=-Xms512m -Xmx512m\u0026#34; - \u0026#34;discovery.zen.ping.unicast.hosts=es2.xxx.com,es3.xxx.com,es4.xxx.com\u0026#34; - \u0026#34;discovery.zen.minimum_master_nodes=2\u0026#34; - \u0026#34;network.publish_host=es1..xxx.com\u0026#34; - \u0026#34;node.name=es01-XXX\u0026#34; ulimits: memlock: soft: -1 hard: -1 volumes: - ./elasticsearch/data:/usr/share/elasticsearch/data ports: - 9200:9200 - 9300:9300 networks: - fjyesnet mem_limit: 1g networks: fjyesnet: 其他节点配置均相同，修改服务发现域名配置即可，放行防火墙部分不再赘述，需要可以查看我以前发的文章\n非主节点配置只需加入node.master=false配置即可\nversion: \u0026#39;2\u0026#39; services: es01: image: docker.elastic.co/elasticsearch/elasticsearch:6.6.2 container_name: es04 environment: - cluster.name=docker-cluster - bootstrap.memory_lock=true - \u0026#34;ES_JAVA_OPTS=-Xms512m -Xmx512m\u0026#34; - \u0026#34;discovery.zen.ping.unicast.hosts=es2.xxx.com,es3.xxx.com,es1.xxx.com\u0026#34; - \u0026#34;discovery.zen.minimum_master_nodes=2\u0026#34; - \u0026#34;network.publish_host=es4..xxx.com\u0026#34; - \u0026#34;node.name=es04-XXX\u0026#34; - \u0026#34;node.master=false\u0026#34; ulimits: memlock: soft: -1 hard: -1 volumes: - ./elasticsearch/data:/usr/share/elasticsearch/data ports: - 9200:9200 - 9300:9300 networks: - fjyesnet mem_limit: 1g networks: fjyesnet: 参数解析\ndiscovery.zen.ping.unicast.hosts配置广播地址，用于节点发现，此处配置的是指向其他节点IP的域名地址，该域名在任何节点均可以被访问\ndiscovery.zen.minimum_master_nodes配置主节点最小数量，该参数十分重要，用于防止脑裂（split-brain）事件的发生。\n原理：前面提到了脑裂事件发生的原因，而避免的方法就是保证至少3个节点可靠地工作，因为一个或两个节点不能形成多数投票，这也是为什么选择使用4个节点并将该值配置为2，确保分布式环境还可以随机宕机一个节点以保证不会出现脑裂问题，因此第4个节点最好配置为无法成为主节点的slave节点。而若仅配置为3个节点的高可用状态，其实是一种“伪高可用”，此时随机宕机一个节点将可能发生“脑裂”问题，2个节点的es环境无法进行主节点的选举，并且可能出现独立的分块。\nnetwork.publish_host用于提供其他节点服务发现的主机名，默认为本地主机名映射为IP的地址，但由于使用容器提供服务发现，并且不在同一个网段，故需要手工指定\nmem_limit容器内存限制为1G\n节点数量与配置的主节点数对应关系\nMaster nodes minimum_master_nodes 备注 1 1 2 1 如果另一个节点宕机，集群将停止工作! 3 2 4 3 5 3 6 4 对可能产生疑问的解释 1.当前docker-compose已经更新到3.7了为什么还使用2这个老版本？ 原因：最新文档地址，官方文档指出3.x版本已经不再支持mem_limit参数\nNote: This replaces the older resource constraint options for non swarm mode in Compose files prior to version 3 (cpu_shares, cpu_quota, cpuset, mem_limit, memswap_limit, mem_swappiness), as described in Upgrading version 2.x to 3.x.\n但对小内存机器以及数据量不会非常的大的环境，内存限制参数尤为重要，经过实测，空载或者小负载（1000 doc）以下的es节点内存占用大约为1.5G左右，故各环境限制为1G不会影响性能，不仅如此，es官方提供的部署文档使用的也是2.2版本\n2.当前Elasticsearch已经更新到7.3了为什么还使用6.6这个老版本？ 原因：7.x版本目前尚有许多外部服务不能兼容，例如graylog，并且Spring官方组件暂时也只能兼容到6.x的版本，所以综合考虑使用6.x的最新版6.6.2\ngraylog官方文档\nCaution\nGraylog 3.x does not work with Elasticsearch 7.x!\nSpring官方文档\nCompatibility Matrix\nSpring Data Elasticsearch Elasticsearch 3.2.x (not yet released) 6.8.1 3.1.x 6.2.2 3.0.x 5.5.0 2.1.x 2.4.0 2.0.x 2.2.0 1.3.x 1.5.2 3.一般高可用不是部署3个节点就足够了吗，为什么要4个节点？ 原因：本次总共部署了3个主节点和一个非主节点，即4个都是数据节点，但只有3个参与选举。如果仅部署3个节点，那么3个都必须为主节点，此时若其中一个主节点宕机，那么不满足选举“至少3个节点可靠地工作”的条件，故新增多一个冗余数据节点，其不能竞选为主节点，但能对节点竞选做出响应，依然满足条件，详细算法可以参考Raft协议的实现原理\n节点启动 使用docker-compose命令启动即可\n$ docker-compose up -d 这种直接启动的方式会报文件夹无访问权限的错误，需要在命令执行后手动赋予文件夹访问权限\n$ chmod 777 -R elasticsearch/ $ chown 1000:1000 -R elasticsearch/ 若此时es容器已经停止则再执行一次启动命令即可\n部署结果 使用chrome插件elasticsearch-head连接任意一个节点查看情况\n4节点在线，接入业务系统后各节点情况如下图。\n注意所有索引副本数都要是2或者3，否则都无法实现高可用效果\n由于网络问题或者服务器问题导致其中一个节点下线，es会自动对各个节点的副本重新路由，此时集群颜色为黄色，仍然可用，稍等片刻会实现最终一致性\n若此时主节点下线，整个集群会立刻重新选举出新的主节点并路由各个分片，保证24小时服务都处在可用状态\n橙色表示尚未路由完成，但所有索引数据均可用\n注意\n若某些索引的副本数设置为1，则当某个节点宕机且其刚好存储该索引的副本，则整个集群的状态将立刻变成红色，意思是数据丢失\n灰色表示已经下线的分片并且无法被重新路由\n","date":"2019-09-10T22:52:00+08:00","permalink":"/archives/216/","title":"Elasticsearch Docker高可用环境搭建"},{"content":"基础环境 本地准备了4台虚拟机配置如下\n虚拟机名称 IP CPU 内存 磁盘 129 192.168.133.129 1 2G 20G 130 192.168.133.130 1 2G 20G 131 192.168.133.131 1 2G 20G 132 192.168.133.132 1 2G 20G 部署架构 部署架构图如下\n各服务详细部署情况\nZookeeper 服务器 节点 端口 版本 129 zk-1-docker 2181 3.4.14 131 zk-2-docker 2181 3.4.14 132 zk-3-docker 2181 3.4.14 MesosMaster 服务器 节点 端口 版本 132 master 5050 1.7.0 130 master 5050 1.7.0 131 master 5050 1.7.0 MesosSlave 服务器 节点 端口 版本 131 slave2 5051 1.7.0 130 slave1 5051 1.7.0 marathon 服务器 服务 端口 版本 132 marathonMaster 8080 v1.6.496 marathon-lb 服务器 服务 端口 版本 129 marathon-lb 9090 v1.14.0 环境准备 开放防火墙 本次服务各主机将可能使用到以下端口\n8080 9090 2181 2888 3888 5050 5051 逐台登陆服务器执行放行防火墙脚本\nfirewall-cmd --zone=public --add-port=8080/tcp --permanent firewall-cmd --zone=public --add-port=2181/tcp --permanent firewall-cmd --zone=public --add-port=5050/tcp --permanent firewall-cmd --zone=public --add-port=5051/tcp --permanent firewall-cmd --zone=public --add-port=9090/tcp --permanent firewall-cmd --zone=public --add-port=2888/tcp --permanent firewall-cmd --zone=public --add-port=3888/tcp --permanent firewall-cmd --reload 安装Docker 本次所有服务均采用Docker方式部署，服务调度通过Docker in Docker的方式进行\n由于Docker有可能出现版本影响环境问题，参考文章 相关文章\n故建议安装时指定版本\n$ yum-config-manager \\ --add-repo \\ https://download.docker.com/linux/centos/docker-ce.repo $ yum install docker-ce-19.03.1 docker-ce-cli-19.03.1 containerd.io 设置开机启动并运行服务\n$ systemctl enable docker $ systemctl start docker 安装docker-compose，脚本参考官方文档\n$ sudo curl -L \u0026#34;https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)\u0026#34; -o /usr/local/bin/docker-compose $ sudo chmod +x /usr/local/bin/docker-compose $ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose 安装Zookeeper 鉴于zk 3.4和3.5版本的配置有一定差别，编写脚本时注意版本，此处使用的是3.4.14，以下配置文件注意hosts的微小差别\ndocker-compose-zoo-129.yml\nversion: \u0026#39;3.1\u0026#39; services: zoo1: image: zookeeper:3.4.14 restart: always network_mode: host extra_hosts: - \u0026#34;zoo2: 192.168.133.131\u0026#34; - \u0026#34;zoo3: 192.168.133.132\u0026#34; ports: - 2181:2181 - 2888:2888 - 3888:3888 environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888 docker-compose-zoo-131.yml\nversion: \u0026#39;3.1\u0026#39; services: zoo2: image: zookeeper:3.4.14 restart: always network_mode: host extra_hosts: - \u0026#34;zoo1: 192.168.133.129\u0026#34; - \u0026#34;zoo3: 192.168.133.132\u0026#34; ports: - 2181:2181 - 2888:2888 - 3888:3888 environment: ZOO_MY_ID: 2 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=0.0.0.0:2888:3888 server.3=zoo3:2888:3888 docker-compose-zoo-132.yml\nversion: \u0026#39;3.1\u0026#39; services: zoo3: image: zookeeper:3.4.14 restart: always network_mode: host extra_hosts: - \u0026#34;zoo1: 192.168.133.129\u0026#34; - \u0026#34;zoo2: 192.168.133.131\u0026#34; ports: - 2181:2181 - 2888:2888 - 3888:3888 environment: ZOO_MY_ID: 3 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=0.0.0.0:2888:3888 安装MesosMaster MesosMaster也采用3节点高可用方案，由于要在各台机器上配置hosts过于麻烦，此处选择使用非hosts注册方式，故需要配置各自的IP地址\ndocker-compose-mesos-master-130.yml\nversion: \u0026#39;3.1\u0026#39; services: mesos-master: image: mesosphere/mesos-master:1.7.0 restart: always network_mode: host hostname: master command: --no-hostname_lookup --ip=192.168.133.130 ports: - 5050:5050 environment: MESOS_PORT: 5050 MESOS_ZK: zk://192.168.133.129:2181/mesos MESOS_QUORUM: 1 MESOS_REGISTRY: in_memory MESOS_LOG_DIR: /var/log/mesos MESOS_WORK_DIR: /var/tmp/mesos docker-compose-mesos-master-131.yml\nversion: \u0026#39;3.1\u0026#39; services: mesos-master: image: mesosphere/mesos-master:1.7.0 restart: always network_mode: host hostname: master command: --no-hostname_lookup --ip=192.168.133.131 ports: - 5050:5050 environment: MESOS_PORT: 5050 MESOS_ZK: zk://192.168.133.129:2181/mesos MESOS_QUORUM: 1 MESOS_REGISTRY: in_memory MESOS_LOG_DIR: /var/log/mesos MESOS_WORK_DIR: /var/tmp/mesos docker-compose-mesos-master-132.yml\nversion: \u0026#39;3.1\u0026#39; services: mesos-master: image: mesosphere/mesos-master:1.7.0 restart: always network_mode: host hostname: master command: --no-hostname_lookup --ip=192.168.133.132 ports: - 5050:5050 environment: MESOS_PORT: 5050 MESOS_ZK: zk://192.168.133.129:2181/mesos MESOS_QUORUM: 1 MESOS_REGISTRY: in_memory MESOS_LOG_DIR: /var/log/mesos MESOS_WORK_DIR: /var/tmp/mesos 安装MesosSlave 需要选择与master对应的版本号，同时也使用IP注册的方式\ndocker-compose-mesos-slave1.yml\nversion: \u0026#39;3.1\u0026#39; services: mesos-slave1: image: mesosphere/mesos-slave:1.7.0 restart: always network_mode: host hostname: slave1 command: --no-systemd_enable_support --no-hostname_lookup --ip=192.168.133.131 ports: - 5051:5051 volumes: - /var/run/docker.sock:/var/run/docker.sock - /cgroup:/cgroup - /sys:/sys - /usr/bin/docker:/usr/local/bin/docker environment: MESOS_PORT: 5051 MESOS_MASTER: zk://192.168.133.129:2181/mesos MESOS_SWITCH_USER: 0 MESOS_CONTAINERIZERS: docker,mesos MESOS_LOG_DIR: /var/log/mesos MESOS_WORK_DIR: /var/tmp/mesos privileged: true docker-compose-mesos-slave2.yml\nversion: \u0026#39;3.1\u0026#39; services: mesos-slave2: image: mesosphere/mesos-slave:1.7.0 restart: always network_mode: host hostname: slave2 command: --no-systemd_enable_support --no-hostname_lookup --ip=192.168.133.130 ports: - 5051:5051 volumes: - /var/run/docker.sock:/var/run/docker.sock - /cgroup:/cgroup - /sys:/sys - /usr/bin/docker:/usr/local/bin/docker environment: MESOS_PORT: 5051 MESOS_MASTER: zk://192.168.133.129:2181/mesos MESOS_SWITCH_USER: 0 MESOS_CONTAINERIZERS: docker,mesos MESOS_LOG_DIR: /var/log/mesos MESOS_WORK_DIR: /var/tmp/mesos privileged: true Marathon marathon在高版本会出现收不到mesos心跳自动停止的问题，暂时没找到可以配置的方式，故使用1.6的版本，问题链接如下\nMissing heartbeats from Mesos, eventually disconnecting and dying\ndocker-compose-marathon.yml\nversion: \u0026#39;3.1\u0026#39; services: marathon: image: mesosphere/marathon:v1.6.496 hostname: marathon restart: always command: --master zk://192.168.133.129:2181/mesos --zk zk://192.168.133.129:2181/marathon ports: - 8080:8080 安装marathon-lb marathon-lb 与marathon没有版本的对应关系，故使用当前最新的版本1.14，新版和旧版的配置也有所不同，在早期版本无需指定端口的环境变量，翻阅官网也指出其应该是自动配置的，但如果使用Docker默认运行脚本则需要手动指定\nif you use the built-in run script, you\u0026rsquo;ll need to pass an env var that contains a comma separated list of the ports HAProxy binds to\n参考ISSUES：\nunbound variable PORTS v1.1.0\ndocker-compose-marathon-lb.yml\nversion: \u0026#39;3.1\u0026#39; services: marathon-lb: image: mesosphere/marathon-lb:v1.14.0 network_mode: host hostname: marathon-lb restart: always command: sse --group external --marathon http://192.168.133.132:8080 environment: PORTS: 9090 ports: - 9090:9090 启动 启动顺序：\nzookeeper mesosMaster mesosSlave marathon marathon-lb 由于全使用docker-compose脚本编写，启动时只需指定文件名即可\n$ docker-compose -f xx.yml up -d 查看日志命令，-f表示跟踪最新的日志\n$ docker-compose -f xx.yml logs [-f] 效果 随便打开一个节点的mesos端口，可看到当前的服务注册数量和当前的leader，其采用与ZK类似的选举算法来选出leader实现高可用\nslave信息\n查看框架注册情况\n查看marathon-lb情况\nmarathon面板\nmarathon支持通过面板创建任务和通过rest接口创建任务\n使用postman创建任务测试\n请求路径http://192.168.133.132:8080/v2/apps\n内容\n{ \u0026#34;id\u0026#34;: \u0026#34;basic-0\u0026#34;, \u0026#34;cmd\u0026#34;: \u0026#34;while [ true ] ; do echo \u0026#39;Hello Marathon\u0026#39; ; sleep 5 ; done\u0026#34;, \u0026#34;cpus\u0026#34;: 0.1, \u0026#34;mem\u0026#34;: 10.0, \u0026#34;instances\u0026#34;: 1 } 结果\n面板同步显示\n运行状态正常\n查看详情可知其运行在130服务器上\n通过面板创建\n切换为json格式可以编辑更多内容\n{ \u0026#34;id\u0026#34;: \u0026#34;demo-springboot\u0026#34;, \u0026#34;cmd\u0026#34;: null, \u0026#34;cpus\u0026#34;: 0.3, \u0026#34;mem\u0026#34;: 128, \u0026#34;disk\u0026#34;: 0, \u0026#34;instances\u0026#34;: 1, \u0026#34;container\u0026#34;: { \u0026#34;docker\u0026#34;: { \u0026#34;image\u0026#34;: \u0026#34;fjy8018/demo-springboot:2.1.6.RELEASE\u0026#34;, \u0026#34;forcePullImage\u0026#34;: true }, \u0026#34;type\u0026#34;: \u0026#34;DOCKER\u0026#34;, \u0026#34;portMappings\u0026#34;: [ { \u0026#34;containerPort\u0026#34;: 8080, \u0026#34;protocol\u0026#34;: \u0026#34;tcp\u0026#34;, \u0026#34;servicePort\u0026#34;: 18080 } ] }, \u0026#34;networks\u0026#34;: [ { \u0026#34;mode\u0026#34;: \u0026#34;container/bridge\u0026#34; } ] } 运行成功\n访问成功\n查看资源调度情况\n配置扩缩容\n配置成功\n两个实例被分别调度到不同的机器上\n配置健康检查\n自动轮询接口\n错误告警\n","date":"2019-08-18T22:01:00+08:00","permalink":"/archives/215/","title":"基于Mesos+Marathon的DevOps分布式环境搭建"},{"content":"初遇DevOps 最早接触到DevOps是在2018年8月26日，那时首先接触到的就是大名鼎鼎的Jenkins，当时看了多篇关于持续集成和持续部署的文章，蠢蠢欲动，便开始了第一套DevOps环境的搭建\nJenkins 第一套DevOps是用Jenkins+Ansible+GitLab搭建的，那时只是了解Docker的好处，并没有使用Docker搭建整套环境，还是基于传统虚拟服务器搭建的整套流程。\nJenkins不仅可以编写groove脚本、pipeline 流水线等，还能与maven集成自动构建，深深感受到了Jenkins的强大，同时大量的用户为其贡献了许多插件，有很多实践问题都可以在网上搜索到解决方案\n搭建过程：\n相关文章\n相关文章\n相关文章\n相关文章\n相关文章\n相关文章\n最后顺利实现了静态网站持续集成、持续部署，详见： 相关文章\nDocker+GitLab 随着容器化技术在国内越来越流行，容器普及度越来越高，其带来的环境隔离极大推动了DevOps的落地，使得无需使用python环境隔离，可以直接利用容器技术实现跨平台和环境隔离，并且利于迁移和部署\n于是在2019年4月开始研究GitLab官方文档，发现其早已原生支持流水线，并依托其GitLab-runner可以直接实现整套DevOps环境\n因此，经过不断摸索，完整的搭建的基于Docker和GitLab的DevOps环境，搭建过程见：\n相关文章\n相关文章\n相关文章\n由于国内当时对GitLab流水线使用的并不多，因此遇到的很多坑都需要翻查官方文档和国外的实践记录，比如：\n相关文章\nDevOps 带来的好处 2019年7月将手上的一个小型项目尝试使用完整的DevOps流程开发，开发小组选择敏捷开发方法，所有的开发工具都尽量的适应快速的需求变化，同时投入非常少量的开发人员，分工明确，从一开始一周发布一次到最后一天可以发布多次\n开发人员已经完全从运维中脱离，全团队没有专门的运维人员，极大的提高了开发效率，精简了人员构成，一旦构建失败，开发人员可以根据邮件通知的内容排查上线失败问题，调整代码和配置后重新提交\n面板提供了流水线统计图，如果出现频繁的失败还是需要人工介入，查看问题原因\n当前的问题 尽管使用了基于Docker+GitLab的流水线实现了完整的DevOps流程，但对于监控和日志部分仍然处在很多问题\n尽管上线的是单体应用，服务器和应用的健康状态都无法得知，也没有相应的告警 应用的日志没有被外部服务收集，无法进行快速分析 没有实现完整的测试流程，生成测试覆盖率报告等 没有实现0停机上线，在容器切换时仍需停机1-2分钟 没有实现灰度上线、蓝绿部署等，上线失败需要手动回滚 未来探索方向 全面容器化服务治理方向 国内对容器化有非常多的实践 kubernetes作为容器调度成为大多数服务治理的选择 容器隔离、迁移和伸缩带来的巨大优势 容器与传统应用部署结合方式 以半全自动方式落地DevOps，采用Mesos+marathon方式同时兼容传统服务部署和容器化部署 兼容分布式服务治理 国外有大量的实践案例 日志收集体系完备 ","date":"2019-08-16T19:32:00+08:00","permalink":"/archives/213/","title":"DevOps实践心得一"},{"content":"配置静态IP 此处使用VMware 模拟服务器环境，为防止IP在启动时动态分配，需要手动配置静态IP，如果是物理服务器可直接跳过\n编辑虚拟机网络 编辑 -\u0026gt; 虚拟网络编辑器\n更改\n取消勾选此项\n进入NAT设置记录默认网关和子网掩码\n进入虚拟机\n设置虚拟机IP 查看虚拟机网卡，lo为本地回环口，实际网卡为ens33\n$ ip addr 1: lo: \u0026lt;LOOPBACK,UP,LOWER_UP\u0026gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:02:fe:c5 brd ff:ff:ff:ff:ff:ff ... 编辑网卡配置\n$ vi /etc/sysconfig/network-scripts/ifcfg-ens33 加入静态配置\nTYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=static DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens33 UUID=a63224f4-f210-4d84-933d-c4f0ccd68f7a DEVICE=ens33 ONBOOT=yes IPADDR=192.168.133.128 NETMASK=255.255.255.0 GATEWAY=192.168.133.2 ZONE=public 解释\nONBOOT：开机启动\nBOOTPROTO：网络分配方式，静态\nIPPADDR：手动指定ip地址\nNETMASK：子网掩码\nGATEWAY：网关ip\nDNS配置 $ vi /etc/resolv.conf 加入DNS配置\nnameserver=192.168.133.2 nameserver 8.8.8.8 nameserver 114.114.114.114 search localdomain 此处仅加入第一个也可以正常使用，但无法ping通域名\n修改主机名 没有需求此步骤可以跳过\n$ vi /etc/sysconfig/network 加入配置\n# Created by anaconda NETWORKING=yes HOSTNAME=localhost 重启网卡 $ systemctl restart network 查看网络信息\n$ ip addr 1: lo: \u0026lt;LOOPBACK,UP,LOWER_UP\u0026gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:02:fe:c5 brd ff:ff:ff:ff:ff:ff inet 192.168.133.128/24 brd 192.168.133.255 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe02:fec5/64 scope link noprefixroute valid_lft forever preferred_lft forever 静态IP配置完成\n安装KVM 此处部分参考GitHub的安装指引，官方wiki也有相关的指南，不过更加复杂\nGitHub安装指引\n官方wiki\n检测是否支持KVM $ cat /proc/cpuinfo | egrep \u0026#39;vmx|svm\u0026#39; flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp tpr_shadow vnmi ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec arat spec_ctrl intel_stibp flush_l1d arch_capabilities 关闭SELinux，将 /etc/sysconfig/selinux 中的 SELinux=enforcing 修改为 SELinux=disabled\n$ vi /etc/sysconfig/selinux 安装 KVM 环境 通过 yum 安装 kvm 基础包和管理工具\nkvm相关安装包及其作用:\nqemu-kvm 主要的KVM程序包 python-virtinst 创建虚拟机所需要的命令行工具和程序库 virt-manager GUI虚拟机管理工具 virt-top 虚拟机统计命令 virt-viewer GUI连接程序，连接到已配置好的虚拟机 libvirt C语言工具包，提供libvirt服务 libvirt-client 为虚拟客户机提供的C语言工具包 virt-install 基于libvirt服务的虚拟机创建命令 bridge-utils 创建和管理桥接设备的工具 $ yum -y install qemu-kvm python-virtinst libvirt libvirt-python virt-manager libguestfs-tools bridge-utils virt-install 重启宿主机，以便加载 kvm 模块\n$ reboot 查看KVM模块是否被正确加载\n$ lsmod | grep kvm kvm_intel 183621 0 kvm 586948 1 kvm_intel irqbypass 13503 1 kvm 配置libvirt以启用TCP服务\nLIBVIRTD_CONFIG=/etc/libvirt/libvirtd.conf LIBVIRTD_ARGS=\u0026#34;--listen\u0026#34; 同时修改/etc/libvirt/libvirtd.conf配置\n... listen_tls = 0 listen_tcp = 1 tcp_port = \u0026#34;16509\u0026#34; listen_addr = \u0026#34;0.0.0.0\u0026#34; auth_tcp = \u0026#34;none\u0026#34; ... 开启kvm服务，并且设置其开机自动启动\n$ systemctl start libvirtd $ systemctl enable libvirtd 查看状态操作结果，如Active: active (running)，说明运行情况良好\n$ systemctl status libvirtd ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since 日 2019-08-11 16:09:20 CST; 30min ago Docs: man:libvirtd(8) https://libvirt.org Main PID: 107953 (libvirtd) Tasks: 19 (limit: 32768) CGroup: /system.slice/libvirtd.service ├─105030 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper ├─105031 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper └─107953 /usr/sbin/libvirtd --listen $ systemctl is-enabled libvirtd enabled 安装WebVirtMgr管理面板 官方指南\n更新源 $ yum install -y epel-release $ yum -y install http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm 安装依赖 $ yum -y install git python-pip libvirt-python libxml2-python python-websockify supervisor nginx $ yum -y install gcc python-devel $ pip install numpy 拉取源码安装 拉取源码和安装环境\n$ git clone git://github.com/retspen/webvirtmgr.git $ cd webvirtmgr $ sudo pip install -r requirements.txt 执行安装文件并按照提示设置管理员账号密码\n$ ./manage.py syncdb WARNING:root:No local_settings file found. Creating tables ... Creating table auth_permission Creating table auth_group_permissions Creating table auth_group Creating table auth_user_groups Creating table auth_user_user_permissions Creating table auth_user Creating table django_content_type Creating table django_session Creating table django_site Creating table servers_compute Creating table instance_instance Creating table create_flavor You just installed Django\u0026#39;s auth system, which means you don\u0026#39;t have any superusers defined. Would you like to create one now? (yes/no): yes Please enter either \u0026#34;yes\u0026#34; or \u0026#34;no\u0026#34;: yes Username (leave blank to use \u0026#39;root\u0026#39;): root Email address: 123@abc.com Password: Password (again): Superuser created successfully. Installing custom SQL ... Installing indexes ... Installed 6 object(s) from 1 fixture(s) 发布到Nginx 拷贝文件到nginx web目录\n$ cd .. \u0026amp;\u0026amp; cp -r webvirtmgr/ /var/www/webvirtmgr/ 创建配置文件\n$ vi /etc/nginx/conf.d/webvirtmgr.conf 内容为\nserver { listen 80 default_server; server_name $hostname; #access_log /var/log/nginx/webvirtmgr_access_log; location /static/ { root /var/www/webvirtmgr/webvirtmgr; # or /srv instead of /var expires max; } location / { proxy_pass http://127.0.0.1:8000; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for; proxy_set_header Host $host:$server_port; proxy_set_header X-Forwarded-Proto $scheme; proxy_connect_timeout 600; proxy_read_timeout 600; proxy_send_timeout 600; client_max_body_size 1024M; # Set higher depending on your needs } } 拷贝一份副本并配置nginx.conf\n$ mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak 写入配置\n#user nobody; worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; include /etc/nginx/conf.d/*.conf; sendfile on; keepalive_timeout 65; server { listen 80; server_name localhost; location / { root html; index index.html index.htm; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } } } 重启nginx\n$ systemctl restart nginx.service 配置用户组和权限\n$ chown -R nginx:nginx /var/www/webvirtmgr 可能出现的报错——Address already in use 重启nginx服务出现错误，查看状态如下\n$ systemctl status nginx.service ● nginx.service - The nginx HTTP and reverse proxy server Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since 日 2019-08-11 12:52:42 CST; 2s ago Process: 26188 ExecStart=/usr/sbin/nginx (code=exited, status=1/FAILURE) Process: 26185 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS) Process: 26183 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS) 8月 11 12:52:40 localhost.localdomain nginx[26188]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use) 8月 11 12:52:40 localhost.localdomain nginx[26188]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use) 8月 11 12:52:41 localhost.localdomain nginx[26188]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use) 8月 11 12:52:41 localhost.localdomain nginx[26188]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use) 8月 11 12:52:42 localhost.localdomain nginx[26188]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use) 8月 11 12:52:42 localhost.localdomain nginx[26188]: nginx: [emerg] still could not bind() 8月 11 12:52:42 localhost.localdomain systemd[1]: nginx.service: control process exited, code=exited status=1 8月 11 12:52:42 localhost.localdomain systemd[1]: Failed to start The nginx HTTP and reverse proxy server. 8月 11 12:52:42 localhost.localdomain systemd[1]: Unit nginx.service entered failed state. 8月 11 12:52:42 localhost.localdomain systemd[1]: nginx.service failed. 解决方法：杀掉占用80端口的进程（不管是不是nginx占用的）\n推荐使用fuser命令快速杀掉占用端口的进程，CentOS可通过以下命令安装\n$ yum install -y psmisc 杀掉占用端口的进程\n$ fuser -k 80/tcp 80/tcp: 18869 18870 18871 18872 18873 再次重启即可\n$ service nginx restart Redirecting to /bin/systemctl restart nginx.service 更新SELinux策略 $ /usr/sbin/setsebool httpd_can_network_connect true $ chkconfig supervisord on 注意：正在将请求转发到“systemctl enable supervisord.service”。 Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service. 配置Supervisor 创建文件/etc/supervisord.d/webvirtmgr.ini\n$ /etc/supervisord.d/webvirtmgr.ini 加入配置\n[program:webvirtmgr] command=/usr/bin/python /var/www/webvirtmgr/manage.py run_gunicorn -c /var/www/webvirtmgr/conf/gunicorn.conf.py directory=/var/www/webvirtmgr autostart=true autorestart=true logfile=/var/log/supervisor/webvirtmgr.log log_stderr=true user=nginx [program:webvirtmgr-console] command=/usr/bin/python /var/www/webvirtmgr/console/webvirtmgr-console directory=/var/www/webvirtmgr autostart=true autorestart=true stdout_logfile=/var/log/supervisor/webvirtmgr-console.log redirect_stderr=true user=nginx 启动supervisor守护线程 $ systemctl start supervisord.service 配置开启启动\n$ systemctl enable supervisord.service 检查 查看端口占用，若80、8000、6080已经监听则正常\n$ netstat -lnpt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 102045/nginx: maste tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 9243/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 9543/master tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 16011/sshd: root@pt tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN 102430/python tcp 0 0 0.0.0.0:6080 0.0.0.0:* LISTEN 102429/python tcp6 0 0 :::111 :::* LISTEN 1/systemd tcp6 0 0 :::22 :::* LISTEN 9243/sshd tcp6 0 0 ::1:25 :::* LISTEN 9543/master tcp6 0 0 ::1:6010 :::* LISTEN 16011/sshd: root@pt 开放防火墙 $ firewall-cmd --zone=public --add-port=80/tcp --permanent $ firewall-cmd --reload 访问web界面 访问虚拟机地址http://192.168.133.128/login/ 可见启动成功\nTCP方式连接宿主机 连接成功\n","date":"2019-08-11T18:35:00+08:00","permalink":"/archives/212/","title":"CentOS 7 安装KVM和WebVirtMgr管理面板"},{"content":"环境 DockerHub账号、GitHub账号\n步骤 构建脚本 此处以构建mycat的docker镜像为例，在本地准备好可以运行的构建脚本Dockerfile，尽可能不使用准备好的配置文件，保证镜像的安全、纯净，并且利用环境变量配置精简构建脚本\nFROM openjdk:8-jre MAINTAINER fjy8018 fjy8018@gmail.com # Mycat 版本 ENV MYCAT_VERSION 1.6-RELEASE ENV MYCAT_PKG Mycat-server-1.6-RELEASE-20161028204710-linux.tar.gz WORKDIR /usr/local # 缓存层 RUN wget http://dl.mycat.io/$MYCAT_VERSION/$MYCAT_PKG RUN apt-get update \u0026amp;\u0026amp; \\ apt-get install -y procps \u0026amp;\u0026amp; \\ apt-get clean \u0026amp;\u0026amp; \\ tar -zxf /usr/local/$MYCAT_PKG \u0026amp;\u0026amp; \\ rm -rf /usr/local/$MYCAT_PKG VOLUME /usr/local/mycat/conf EXPOSE 8066 9066 ENTRYPOINT /usr/local/mycat/bin/mycat console 创建GitHub仓库 在GitHub上创建对应的仓库，并推送Dockerfile\n链接DockerHub与GitHub 登陆DockerHub选择repository新建仓库\n首次链接可以参照指引操作，主要是做OAuth 授权\n创建构建分支和标签 连接GitHub后可以选择需要构建的仓库，同时展开构建选项，可以指定不同分支构建出的标签，默认master分支构建为latest版本\n触发构建 第一次会自动触发构建，随后代码推送都会触发构建\n构建结果 在管理页面可以查看构建时长和构建结果\n结果 用户视角 用户视角可以看到构建的GitHub仓库\n自动构建产生的分支\n构建历史记录\n","date":"2019-07-31T22:36:41+08:00","permalink":"/archives/210/","title":"使用GitHub和DockerHub自动构建镜像"},{"content":"基础环境 SpringBoot 2.0 开始官方提供了对quartz定时任务的自动配置支持依赖spring-boot-starter-quartz组件，无需自行集成\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-quartz\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; 环境版本：\nSpringBoot 2.1.6.RELEASE Java 8 定时任务配置 添加依赖 在pom.xml中添加quartz、Jpa、mysql驱动、lombok依赖\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-data-jpa\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;mysql\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;mysql-connector-java\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;runtime\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.projectlombok\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;lombok\u0026lt;/artifactId\u0026gt; \u0026lt;optional\u0026gt;true\u0026lt;/optional\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-quartz\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; 添加配置文件 添加数据库配置\nspring: datasource: primary: driver-class-name: com.mysql.cj.jdbc.Driver username: root password: hmdt url: jdbc:mysql://db/bussines_db?useSSL=false\u0026amp;useUnicode=true\u0026amp;characterEncoding=UTF-8\u0026amp;serverTimezone=Asia/Shanghai quartz: url: jdbc:mysql://db/bussines_quartz?useSSL=false\u0026amp;useUnicode=true\u0026amp;characterEncoding=UTF-8\u0026amp;serverTimezone=Asia/Shanghai driver-class-name: com.mysql.cj.jdbc.Driver username: root password: hmdt quartz: #相关属性配置 properties: org: quartz: scheduler: instanceName: clusteredScheduler instanceId: AUTO jobStore: # 数据源名称 dataSource: quartzDataSource class: org.quartz.impl.jdbcjobstore.JobStoreTX driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate tablePrefix: QRTZ_ isClustered: true clusterCheckinInterval: 1000 useProperties: false threadPool: class: org.quartz.simpl.SimpleThreadPool threadCount: 10 threadPriority: 5 threadsInheritContextClassLoaderOfInitializingThread: true job-store-type: jdbc #初始化表结构 jdbc: initialize-schema: never jpa: hibernate: ddl-auto: update show-sql: true database-platform: org.hibernate.dialect.MySQL55Dialect properties: hibernate: format_sql: true use_sql_comments: true 数据库配置也有样例配在spring.quartz.properties中，但亲测配置无效，并且由于还需要指定连接池等信息，此处配置在spring的连接池中更容易管理\n下载官网建表语句 尽管使用了JPA，但由于没有实体关联，spring并不会帮我们自动建表，还需要去官网下载sql脚本\nquartz官网\n官网压缩包路径quartz-2.3.0-SNAPSHOT\\src\\org\\quartz\\impl\\jdbcjobstore中支持大量数据库建表语句，选择对应的使用即可\n配置多数据源 多配置源之前在文章中详细说过配置样例和原因，这里不再赘述\n相关文章\n编写数据库配置，统一使用HikariCP连接池\npackage com.gcb.invest.config; import com.zaxxer.hikari.HikariDataSource; import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties; import org.springframework.boot.autoconfigure.quartz.QuartzDataSource; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Primary; /** * 定时任务数据源配置 * * @author F嘉阳 * @date 2019-07-28 21:52 */ @Configuration public class DataSourceConfig { @Bean @Primary @ConfigurationProperties(\u0026#34;spring.datasource.primary\u0026#34;) public DataSourceProperties primaryDataSourceProperties() { return new DataSourceProperties(); } @Bean @Primary @ConfigurationProperties(\u0026#34;spring.datasource.primary.configuration\u0026#34;) public HikariDataSource firstDataSource() { return primaryDataSourceProperties().initializeDataSourceBuilder() .type(HikariDataSource.class).build(); } @Bean @ConfigurationProperties(\u0026#34;spring.datasource.quartz\u0026#34;) public DataSourceProperties quartzDataSourceProperties() { return new DataSourceProperties(); } @Bean @QuartzDataSource @ConfigurationProperties(\u0026#34;spring.datasource.quartz.configuration\u0026#34;) public HikariDataSource quartzDataSource() { return quartzDataSourceProperties().initializeDataSourceBuilder() .type(HikariDataSource.class).build(); } } 定时任务配置 配置样例1——Bean注入 编写定时任务，由于经过Spring的封装，使用变得更加方便\n此处通过内部类配置定时任务，并使用@DisallowConcurrentExecution注解指定其在分布式环境下不可并发执行\nimport lombok.extern.slf4j.Slf4j; import org.quartz.*; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.scheduling.quartz.QuartzJobBean; /** * 定时任务配置 * * @author ：F嘉阳 * @date ：2019/7/29 9:16 */ @Slf4j @Configuration public class QuartzConfig { /** * 测试定时任务构建 * * @return */ @Bean public JobDetail testTaskJobDetail() { return JobBuilder.newJob(TestTask.class) .withIdentity(TestTask.class.getName()) .storeDurably(true) .build(); } /** * 测试定时任务配置 * * @return */ @Bean public Trigger testTaskTrigger() { CronScheduleBuilder scheduleBuilder = CronScheduleBuilder.cronSchedule(\u0026#34;0/2 * * * * ?\u0026#34;); return TriggerBuilder.newTrigger() .forJob(testTaskJobDetail()) .withIdentity(TestTask.class.getName()) .withSchedule(scheduleBuilder) .build(); } @DisallowConcurrentExecution private class TestTask extends QuartzJobBean { @Override protected void executeInternal(JobExecutionContext jobExecutionContext) throws JobExecutionException { log.debug(\u0026#34;执行测试定时任务\u0026#34;); } } } 配置样例2——调度器注入 另一种方式可使用注入调度器手动构建任务，不过需要调用该方法之后调度器才会被创建\nimport org.quartz.*; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; import java.util.Date; import java.util.UUID; /** * @author ：F嘉阳 * @date ：2019/7/29 8:59 */ @Service @Transactional(rollbackFor = Exception.class) public class QuartzTimerService { @Autowired private Scheduler scheduler; public void buildGoodStockTimer() throws Exception { //任务名称 String name = UUID.randomUUID().toString(); //任务所属分组 String group = CustomQuartzJob.class.getName(); CronScheduleBuilder scheduleBuilder = CronScheduleBuilder.cronSchedule(\u0026#34;0/1 * * * * ?\u0026#34;); //创建任务 JobDetail jobDetail = JobBuilder.newJob(CustomQuartzJob.class).withIdentity(name,group).build(); //创建任务触发器 Trigger trigger = TriggerBuilder.newTrigger().withIdentity(name,group).withSchedule(scheduleBuilder).build(); //将触发器与任务绑定到调度器内 scheduler.scheduleJob(jobDetail, trigger); } } 定时任务类\nimport lombok.extern.slf4j.Slf4j; import org.quartz.DisallowConcurrentExecution; import org.quartz.JobExecutionContext; import org.quartz.JobExecutionException; import org.springframework.scheduling.quartz.QuartzJobBean; /** * @author ：F嘉阳 * @date ：2019/7/29 8:55 */ @Slf4j @DisallowConcurrentExecution public class CustomQuartzJob extends QuartzJobBean { @Override protected void executeInternal(JobExecutionContext jobExecutionContext) throws JobExecutionException { log.info(\u0026#34;执行定时任务\u0026#34;); } } 执行结果 执行成功\n在分布式环境下自动进行作业节点迁移\n","date":"2019-07-29T15:04:00+08:00","permalink":"/archives/209/","title":"SpringBoot 2.x quartz多数据定时任务源配置"},{"content":"版本变化始末 在 2019.07.24 Docker社区发布了19.03的新版本，在此前版本号为18.0x 新版本带来的坑 原本没太在意docker版本的变化，因为多数情况都是兼容旧版的所有特性和功能的\n结果在近期的CICD构建中，在ci脚本没有变化的情况下突然所有runner的构建都失败了\n随后立即采取以下措施尝试修复\n工程版本回退到上一次可用状态 更换runner服务器和容器 更改ci脚本 重启docker服务 均无效\n执行构建步骤出现了以往没有出现过的警告信息\nRunning with gitlab-runner 11.8.0 (4745a6f3) on docker-in-docker-runner 4iAG2sYM Using Docker executor with image docker:stable ... Starting service docker:dind ... Pulling docker image docker:dind ... Using docker image sha256:fd0c64832f7e46b63a180e6000dbba7ad7a63542c5764841cba73429ba74a39e for docker:dind ... Waiting for services to be up and running... *** WARNING: Service runner-4iAG2sYM-project-89-concurrent-0-docker-0 probably didn\u0026#39;t start properly. Health check error: service \u0026#34;runner-4iAG2sYM-project-89-concurrent-0-docker-0-wait-for-service\u0026#34; timeout Health check container logs: ... 2019-07-24T03:11:35.365911823Z time=\u0026#34;2019-07-24T03:11:35.365537741Z\u0026#34; level=info msg=\u0026#34;API listen on [::]:2376\u0026#34; 2019-07-24T03:11:35.366107674Z time=\u0026#34;2019-07-24T03:11:35.365626706Z\u0026#34; level=info msg=\u0026#34;API listen on /var/run/docker.sock\u0026#34; ********* Pulling docker image docker:stable ... Using docker image sha256:c4154a2b47a18fe9437956ab981bd5924b19e7ae3eb3ed60c42cf8dfa394d550 for docker:stable ... 最终报错信息\nDownloading artifacts for maven-build (280)... Downloading artifacts from coordinator... ok id=280 responseStatus=200 OK token=L9iuC5Ss $ docker build -t $CONTAINER_IMAGE:$IMAGE_VERSION . Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running? ERROR: Job failed: exit code 1 寻找解决方法 该错误在早期版本就出现过，但与本次情况均不符合，以往通过挂载容器路径即可，而本次情况是runner服务和构建脚本均不变的情况下突然出现构建失败的情况\n最终在社区ISSUS讨论中发现了相关讨论，并且问题发生时间相近\nJobs failing with \u0026ldquo;Cannot connect to the Docker daemon\u0026rdquo;\n官方开发者解释 当天几个小时前发布的新版本 19.03默认开启了TLS确实会影响已有的Docker in Docker 模式的构建环境\n同时官方开发人员建议自行部署的runner建议开启TLS功能\n该开发者在另一个讨论中提供了19.03环境下的runner启动和配置样板，同时还有样例ci脚本\nUpdate DinD example for 19.03\n解决方法 最好可以卸载旧版本的Docker，尽管运行runner和dind环境都基于docker容器中 加入TLS相关runner注册配置 此处采用更新Docker版本并使用最新版的runner镜像进行注册\n更新docker版本 官方文档\n卸载旧docker\n$ sudo yum remove docker \\ docker-client \\ docker-client-latest \\ docker-common \\ docker-latest \\ docker-latest-logrotate \\ docker-logrotate \\ docker-engine 配置生产环境的依赖仓库\n$ sudo yum-config-manager \\ --add-repo \\ https://download.docker.com/linux/centos/docker-ce.repo 官方建议的安装新版Docker命令\n$ sudo yum install docker-ce docker-ce-cli containerd.io 但经过这次事件后个人不建议使用该命令，而是应该手动指定所有Docker环境版本号，防止以后版本更新导致不兼容问题\n查看当前可以安装的所有版本\n$ sudo yum list docker-ce --showduplicates | sort -r * updates: repos.lax.quadranet.com Loaded plugins: fastestmirror * extras: mirror.hostduplex.com * epel: d2lzkl7pfhq30w.cloudfront.net * elrepo-kernel: repos.lax-noc.com docker-ce.x86_64 3:19.03.1-3.el7 docker-ce-stable docker-ce.x86_64 3:19.03.0-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.8-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.7-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.6-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.5-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.4-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.3-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.2-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.1-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.0-3.el7 docker-ce-stable docker-ce.x86_64 18.06.3.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.2.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.1.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.0.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.12.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.12.0.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.09.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.09.0.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.06.2.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.06.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.06.0.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.03.3.ce-1.el7 docker-ce-stable docker-ce.x86_64 17.03.2.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.03.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 17.03.0.ce-1.el7.centos docker-ce-stable Determining fastest mirrors * base: mirror.hostduplex.com Available Packages 指定最新版本\n$ sudo yum install docker-ce-19.03.1 docker-ce-cli-19.03.1 containerd.io 启动docker\n$ sudo systemctl start docker 设置开机自启\n$ sudo systemctl enable docker 安装Gitlab Runner 官方对新版启用TLS的注册样例\nRegister GitLab Runner from the command line to use docker and privileged mode: sudo gitlab-runner register -n \\ --url https://gitlab.com/ \\ --registration-token REGISTRATION_TOKEN \\ --executor docker \\ --description \u0026#34;My Docker Runner\u0026#34; \\ --docker-image \u0026#34;docker:stable\u0026#34; \\ --docker-privileged \\ --docker-volumes \u0026#34;/certs/client\u0026#34; 此处采用容器化注册runner命令为\n$ docker run --rm -t -i \\ -v /etc/gitlab-runner/config:/etc/gitlab-runner \\ gitlab/gitlab-runner:alpine-v12.1.0 register \\ --url \u0026#34;https://gitlab.com/\u0026#34; \\ --registration-token REGISTRATION_TOKEN \\ --executor docker \\ --non-interactive \\ --run-untagged \\ --tag-list \u0026#34;Runner,Docker\u0026#34; \\ --run-untagged=\u0026#34;true\u0026#34; \\ --locked=\u0026#34;false\u0026#34; \\ --description \u0026#34;My Docker Runner\u0026#34; \\ --docker-image \u0026#34;docker:stable\u0026#34; \\ --docker-privileged \\ --docker-volumes \u0026#34;/certs/client\u0026#34; 主要的变化至于增加了证书挂载--docker-volumes \u0026quot;/certs/client\u0026quot;配置，同时指定了runner的版本号\n启动runner\n$ docker run --name gitlab-runner --restart always -d \\ -v /etc/gitlab-runner/config:/etc/gitlab-runner \\ -v /var/run/docker.sock:/var/run/docker.sock \\ gitlab/gitlab-runner:alpine-v12.1.0 修改构建脚本 官方对新版启用TLS的构建脚本样例，指定了构建时使用的docker版本与外部一直，防止不兼容的问题\n同时加入了TLS环境变量配置\nimage: docker:19.03 variables: # When using dind service we need to instruct docker, to talk with # the daemon started inside of the service. The daemon is # available with a network connection instead of the default # /var/run/docker.sock socket. docker:19.03-dind does this # automatically by setting the DOCKER_HOST in # https://github.com/docker-library/docker/blob/d45051476babc297257df490d22cbd806f1b11e4/19.03/docker-entrypoint.sh#L23-L29 # # The \u0026#39;docker\u0026#39; hostname is the alias of the service container as described at # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services. # # Note that if you\u0026#39;re using the Kubernetes executor, the variable should be set to # tcp://localhost:2376/ because of how the Kubernetes executor connects services # to the job container # DOCKER_HOST: tcp://localhost:2376/ # # When using dind, it\u0026#39;s wise to use the overlayfs driver for # improved performance. DOCKER_DRIVER: overlay2 # Specify to Docker where to create the certificates, Docker will # create them automatically on boot, and will create # `/certs/client` that will be shared between the service and # build container. DOCKER_TLS_CERTDIR: \u0026#34;/certs\u0026#34; services: - docker:19.03-dind before_script: - echo $DOCKER_HOST - docker info build: stage: build script: - docker build -t my-docker-image . 在此前自己的构建脚本是按照官网提供的样板，没有指定版本，因此会导致每次构建都会使用当前最新的版本作为基础，最终导致外部Docker和内部Docker容器版本不一致\n官网示例DinD构建脚本\nimage: docker:stable services: - docker:dind ... 按照社区配置修改后重新构建则成功\n总结 对于Docker等云服务平台仍然需要特别注意版本变化，为了防止不兼容的情况，在生产环境尽量不要使用latest版本标签，而是应该手动指定所有参与构建的版本环境\n","date":"2019-07-28T13:09:00+08:00","permalink":"/archives/208/","title":"Docker 19.03 版本对原Gitlab-Runner的影响"},{"content":"前置步骤：\nDocker 版 GitLab CICD 实践1——GitLab 部署 [[Docker 版 GitLab CICD 实践]] Docker 版 GitLab CICD 实践2——Runner 部署 [[Docker 版 GitLab CICD 实践——Runner 部署]] 官方文档 编写结构类似Jenkins pineline流水线\nGitLab CI/CD Pipeline Configuration Reference\nGitLab CI/CD Pipeline Configuration Reference GitLab CI/CD pipelines are configured using a YAML file called .gitlab-ci.yml within each project.\nThe .gitlab-ci.yml file defines the structure and order of the pipelines and determines:\nWhat to execute using GitLab Runner. What decisions to make when specific conditions are encountered. For example, when a process succeeds or fails. This topic covers CI/CD pipeline configuration. For other CI/CD configuration information, see:\nGitLab CI/CD Variables, for configuring the environment the pipelines run in. GitLab Runner advanced configuration, for configuring GitLab Runner. We have complete examples of configuring pipelines:\nFor a quick introduction to GitLab CI, follow our quick start guide. For a collection of examples, see GitLab CI/CD Examples. To see a large .gitlab-ci.yml file used in an enterprise, see the .gitlab-ci.yml file for gitlab-ce. 目标 maven项目自动打包 自动测试 自动镜像构建并上传 官方参数表列出了作业的可用参数：\n关键词 描述 script 由Runner执行的Shell脚本。 image 使用泊坞窗图像。也可用：image:name和image:entrypoint。 services 使用docker services图像。也可用：services:name，services:alias，services:entrypoint，和services:command。 before_script 覆盖在作业之前执行的一组命令。 after_script 覆盖作业后执行的一组命令。 stages 定义管道中的阶段。 stage 定义作业阶段（默认值:) test。 only 创建作业时限制。也可用：only:refs，only:kubernetes，only:variables，和only:changes。 except 在未创建作业时限制。也可用：except:refs，except:kubernetes，except:variables，和except:changes。 tags 用于选择Runner的标签列表。 allow_failure 让工作失败。失败的作业无助于提交状态。 when 什么时候开始工作。也可用：when:manual和when:delayed。 environment 作业部署到的环境的名称。也可用：environment:name，environment:url，environment:on_stop，和environment:action。 cache 后续运行之间应缓存的文件列表。也可用：cache:paths，cache:key，cache:untracked，和cache:policy。 artifacts 成功附加到作业的文件和目录列表。也可用：artifacts:paths，artifacts:name，artifacts:untracked，artifacts:when，artifacts:expire_in，artifacts:reports，和artifacts:reports:junit。 在GitLab 企业版，这些都是可供选择：artifacts:reports:codequality，artifacts:reports:sast，artifacts:reports:dependency_scanning，artifacts:reports:container_scanning，artifacts:reports:dast，artifacts:reports:license_management，和artifacts:reports:performance。 dependencies 作业所依赖的其他作业，以便您可以在它们之间传递工件。 coverage 给定作业的代码覆盖率设置。 retry 在发生故障的情况下，可以自动重试作业的次数和次数。 parallel 应该并行运行多少个作业实例。 trigger 定义下游管道触发器。 include 允许此作业包含外部YAML文件。也可用：include:local，include:file，include:template，和include:remote。 extends 此作业将继承的配置条目。 pages 上传作业结果以用于GitLab Pages。 variables 在作业级别定义作业变量。 注： 参数types和type被弃用。\n参考 官方构建案例\nDeploy a Spring Boot application to Cloud Foundry with GitLab CI/CD\n参考其脚本\nConfigure GitLab CI/CD to deploy your application Now we need to add the GitLab CI/CD configuration file (.gitlab-ci.yml) to our project’s root. This is how GitLab figures out what commands need to be run whenever code is pushed to our repository. We will add the following .gitlab-ci.yml file to the root directory of the repository, GitLab will detect it automatically and run the steps defined once we push our code:\nimage: java:8 stages: - build - deploy build: stage: build script: ./mvnw package artifacts: paths: - target/demo-0.0.1-SNAPSHOT.jar production: stage: deploy script: - curl --location \u0026#34;https://cli.run.pivotal.io/stable?release=linux64-binary\u0026amp;source=github\u0026#34; | tar zx - ./cf login -u $CF_USERNAME -p $CF_PASSWORD -a api.run.pivotal.io - ./cf push only: - master 配置 全局变量 进入工程CI设置配置全局变量脚本，包含镜像仓库登陆名称、密码、打包名称等\n配置全局变量目的在于配置脚本中不应该包含密码等敏感信息\n若希望使用GitLab内置环境变量，可参考官方表格\nGitLab CI/CD environment variables\nCI脚本 结合可用参数和样例配置，根据已经存在的SpringBoot项目编写响应的CI脚本.gitlab-ci.yml\nimage: docker:stable services: - docker:dind variables: DOCKER_DRIVER: overlay SPRING_PROFILES_ACTIVE: gitlab-ci stages: - build - package maven-build: image: maven:3-jdk-8 stage: build script: \u0026#34;mvn package -B\u0026#34; artifacts: paths: - target/*.jar docker-build: stage: package script: - docker build -t $CONTAINER_IMAGE:latest . - docker login -u $DOCKER_HUB_USER -p $DOCKER_HUB_PASS - docker push $CONTAINER_IMAGE:latest 该脚本对于满足脚本目标的工程都可复用，若需要测试，则增加相关的测试步骤即可\n此处由于工程包含外部依赖关系，不在构建时测试\n自动构建 默认的触发构建事件为commit等\n触发后会自动执行任务\nMaven自动触发构建，自动镜像打包\n自动上传（推送）镜像\n执行结果 若执行失败则会发送邮件给开发人员\nCICD全套流程目标实现！\n总结 不管使用GitLab CICD或者是使用Jenkins+Ansible的方式，目标都在于践行DevOps，打通开发与运维流程，一旦配置好CICD流程，往后开发人员便不再操心打包、构建等操作，可以专注开发\n","date":"2019-04-03T16:32:00+08:00","permalink":"/archives/207/","title":"DevOps GitLab CICD 实践3——CI文件编写"},{"content":"前置步骤：DevOps GitLab CICD 实践1——GitLab 部署\n官方文档 文档地址\n虽然有官方指引，但个人感觉指引的不够清晰，导致初次配置可能频繁失败\nRun GitLab Runner in a container This is how you can run GitLab Runner inside a Docker container.\nGeneral GitLab Runner Docker image usage GitLab Runner Docker images (based on Ubuntu or Alpine Linux) are designed as wrappers around the standard gitlab-runner command, like if GitLab Runner was installed directly on the host.\nThe general rule is that every GitLab Runner command that normally would be executed as:\ngitlab-runner [Runner command and options...] can be executed with:\ndocker run [chosen docker options...] gitlab/gitlab-runner [Runner command and options...] For example, getting the top-level help information for GitLab Runner command could be executed as:\ndocker run --rm -t -i gitlab/gitlab-runner --help NAME: gitlab-runner - a GitLab Runner USAGE: gitlab-runner [global options] command [command options] [arguments...] VERSION: 10.7.0 (7c273476) (...) In short, the gitlab-runner part of the command is replaced with docker run [docker options] gitlab/gitlab-runner, while the rest of Runner’s command stays as it is described in the register documentation. The only difference is that the gitlab-runner command is executed inside of a Docker container.\nDocker image installation and configuration Install Docker first:\ncurl -sSL https://get.docker.com/ | sh You need to mount a config volume into the gitlab-runner container to be used for configs and other resources:\ndocker run -d --name gitlab-runner --restart always \\ -v /srv/gitlab-runner/config:/etc/gitlab-runner \\ -v /var/run/docker.sock:/var/run/docker.sock \\ gitlab/gitlab-runner:latest Tip: On macOS, use /Users/Shared instead of /srv.\nOr, you can use a config container to mount your custom data volume:\ndocker run -d --name gitlab-runner-config \\ -v /etc/gitlab-runner \\ busybox:latest \\ /bin/true And then, run the Runner:\ndocker run -d --name gitlab-runner --restart always \\ -v /var/run/docker.sock:/var/run/docker.sock \\ --volumes-from gitlab-runner-config \\ gitlab/gitlab-runner:latest Register the runner you just launched by following the instructions in the Docker section of Registering Runners. The runner won’t pick up any jobs until it’s registered.\n安装步骤 获取Gitlab Runner秘钥 可以通过Gitlab管理员账号获取，也可以让每一个用户自行配置\n普通用户查看秘钥 进入任意一个仓库的设置中，查看CICD配置\n准备注册专用Runner令牌\n管理员查看令牌 进入总设置页面，配置全局Runner令牌\nRunner注册 由于Runner一般运行复杂构建、打包任务，推荐配置在性能、带宽更大的机房\n准备Docker环境 $ curl -sSL https://get.docker.com/ | sh $ systemctl start docker 注册 可以根据需要选择注册Runner类型\n同时，为了方便配置，使用单行注册并且关闭交互\n单行注册官方文档\n命令说明：\n-v挂载的目的是为了将注册后的配置文件持久化，用于运行容器 --rm指定容器运行结束后自动删除停止的容器 -it指定使用命令行交互方式运行，便于查看注册结果 $ docker run --rm -it \\ -v /www/wwwroot/gitlab/srv/gitlab-runner/config:/etc/gitlab-runner \\ gitlab/gitlab-runner:alpine-v11.8.0 register \\ --non-interactive \\ --executor \u0026#34;docker\u0026#34; \\ --docker-image docker:stable \\ --url \u0026#34;Gitlab URL\u0026#34; \\ --registration-token \u0026#34;令牌\u0026#34; \\ --description \u0026#34;描述\u0026#34; \\ --tag-list \u0026#34;标签1,标签2\u0026#34; \\ --run-untagged \\ --docker-privileged \\ --locked=\u0026#34;false\u0026#34; 注册成功提示\n此时管理面板显示新的Runner已经注册\nRunner 启动 挂载本地配置信息并启动Runner\n$ docker run -d --name gitlab-runner --restart always \\ -v /www/wwwroot/gitlab/srv/gitlab-runner/config:/etc/gitlab-runner \\ -v /var/run/docker.sock:/var/run/docker.sock \\ gitlab/gitlab-runner:alpine-v11.8.0 此时可见Runner已经保持和Gitlab的联系\n","date":"2019-04-03T16:31:00+08:00","permalink":"/archives/206/","title":"DevOps GitLab CICD 实践2——Runner 部署"},{"content":"配置目标 邮件提示 GitHub第三方授权登陆 GitLab Runner Docker私服注册 官方介绍 目前微服务盛行环境下，服务部署优先考虑Docker方式，便于迁移和弹性伸缩\n官方镜像介绍 GitLab Docker images\nGitLab Docker images Both GitLab CE and EE are in Docker Hub:\nGitLab CE Docker image GitLab EE Docker image The GitLab Docker images are monolithic images of GitLab running all the necessary services on a single container.\nIn the following examples we are using the image of GitLab CE. To use GitLab EE instead of GitLab CE, replace the image name to gitlab/gitlab-ee:latest.\nIf you want to use the latest RC image, use gitlab/gitlab-ce:rc or gitlab/gitlab-ee:rc for GitLab CE and GitLab EE respectively.\nThe GitLab Docker images can be run in multiple ways:\nRun the image in Docker Engine Install GitLab into a cluster Install GitLab using docker-compose docker-compose 脚本 此处选择社区版(CE)安装，同时为了便于参数配置，使用docker-compose方式编写脚本文件\nInstall GitLab using docker-compose With Docker compose you can easily configure, install, and upgrade your Docker-based GitLab installation.\nInstall Docker Compose\nCreate a docker-compose.yml file (or download an example):\nweb: image: \u0026#39;gitlab/gitlab-ce:latest\u0026#39; restart: always hostname: \u0026#39;gitlab.example.com\u0026#39; environment: GITLAB_OMNIBUS_CONFIG: | external_url \u0026#39;https://gitlab.example.com\u0026#39; # Add any other gitlab.rb configuration here, each on its own line ports: - \u0026#39;80:80\u0026#39; - \u0026#39;443:443\u0026#39; - \u0026#39;22:22\u0026#39; volumes: - \u0026#39;/srv/gitlab/config:/etc/gitlab\u0026#39; - \u0026#39;/srv/gitlab/logs:/var/log/gitlab\u0026#39; - \u0026#39;/srv/gitlab/data:/var/opt/gitlab\u0026#39; Make sure you are in the same directory as docker-compose.yml and run docker-compose up -d to start GitLab\nRead “Pre-configure Docker container” to see how the GITLAB_OMNIBUS_CONFIG variable works.\nBelow is another docker-compose.yml example with GitLab running on a custom HTTP and SSH port. Notice how the GITLAB_OMNIBUS_CONFIG variables match the ports section:\nweb: image: \u0026#39;gitlab/gitlab-ce:latest\u0026#39; restart: always hostname: \u0026#39;gitlab.example.com\u0026#39; environment: GITLAB_OMNIBUS_CONFIG: | external_url \u0026#39;http://gitlab.example.com:9090\u0026#39; gitlab_rails[\u0026#39;gitlab_shell_ssh_port\u0026#39;] = 2224 ports: - \u0026#39;9090:9090\u0026#39; - \u0026#39;2224:22\u0026#39; volumes: - \u0026#39;/srv/gitlab/config:/etc/gitlab\u0026#39; - \u0026#39;/srv/gitlab/logs:/var/log/gitlab\u0026#39; - \u0026#39;/srv/gitlab/data:/var/opt/gitlab\u0026#39; This is the same as using --publish 9090:9090 --publish 2224:22.\n官方提示说明Docker CE版基于Omnibus版本，故环境配置也可参考相关文档\nOmnibus文档目录 Installation and Configuration using omnibus package Note: This section describes the commonly used configuration settings. Check configuration section of the documentation for complete configuration settings.\nInstalling GitLab Manually downloading and installing a GitLab package Setting up a domain name/URL for the GitLab Instance so that it can be accessed easily Enabling HTTPS Enabling notification EMails Enabling replying via email Installing and configuring postfix Enabling container registry on GitLab You will require SSL certificates for the domain used for container registry Enabling GitLab Pages If you want HTTPS enabled, you will have to get wildcard certificates Enabling Elasticsearch GitLab Mattermost Set up the Mattermost messaging app that ships with Omnibus GitLab package. GitLab Prometheus Set up the Prometheus monitoring included in the Omnibus GitLab package. GitLab High Availability Roles 结合配置目标编写yaml文件\n注意：\n此处邮件使用163邮箱（官方没有提供163邮箱支持案例） Docker私服公钥执行从私服上获取 由于特殊原因，目标配置未启动SSL安全连接，但GitLab可以通过简单配置支持SSL并自动更新证书 配置文档\nLet’s Encrypt Integration Primary GitLab Instance Note: Introduced in GitLab version 10.5 and disabled by default. Enabled by default in GitLab version 10.7 and later if external_url is set with the httpsprotocol and no certificates are configured.\nNote: In order for Let’s Encrypt verification to work correctly, ports 80 and 443 will need to be accessible to the Let’s Encrypt servers that run the validation. Also note that the validation currently does not work with non-standard ports.\nCaution Administrators installing or upgrading to GitLab version 10.7 or later and do not plan on using Let’s Encrypt should set the following in /etc/gitlab/gitlab.rb to disable:\nletsencrypt[\u0026#39;enable\u0026#39;] = false Add the following entries to /etc/gitlab/gitlab.rb to enable Let’s Encrypt support for the primary domain:\nletsencrypt[\u0026#39;enable\u0026#39;] = true # GitLab 10.5 and 10.6 require this option external_url \u0026#34;https://gitlab.example.com\u0026#34;\t# Must use https protocol letsencrypt[\u0026#39;contact_emails\u0026#39;] = [\u0026#39;foo@email.com\u0026#39;] # Optional 生成163邮箱授权密码\n生成GitHub授权秘钥\n最终配置\nversion: \u0026#39;3.1\u0026#39; services: gitlab: environment: GITLAB_OMNIBUS_CONFIG: | external_url \u0026#39;外部访问地址\u0026#39; gitlab_rails[\u0026#39;gitlab_shell_ssh_port\u0026#39;] = 外部SSH端口 registry_external_url \u0026#39;Docker私服地址\u0026#39; registry_nginx[\u0026#39;ssl_certificate\u0026#39;] = \u0026#34;Docker 私服CA证书 crt文件\u0026#34; registry_nginx[\u0026#39;ssl_certificate_key\u0026#39;] = \u0026#34;Docker 私服公钥 pem文件\u0026#34; gitlab_rails[\u0026#39;smtp_enable\u0026#39;] = true gitlab_rails[\u0026#39;smtp_address\u0026#39;] = \u0026#34;smtp.163.com\u0026#34; gitlab_rails[\u0026#39;smtp_port\u0026#39;] = 465 gitlab_rails[\u0026#39;smtp_user_name\u0026#39;] = \u0026#34;邮件发送者名称\u0026#34; gitlab_rails[\u0026#39;gitlab_email_from\u0026#39;] = \u0026#39;邮件发送地址\u0026#39; gitlab_rails[\u0026#39;smtp_password\u0026#39;] = \u0026#34;授权密码\u0026#34; gitlab_rails[\u0026#39;smtp_domain\u0026#39;] = \u0026#34;163.com\u0026#34; gitlab_rails[\u0026#39;smtp_authentication\u0026#39;] = \u0026#34;login\u0026#34; gitlab_rails[\u0026#39;smtp_enable_starttls_auto\u0026#39;] = true gitlab_rails[\u0026#39;smtp_tls\u0026#39;] = true gitlab_rails[\u0026#39;omniauth_enabled\u0026#39;] = true gitlab_rails[\u0026#39;omniauth_allow_single_sign_on\u0026#39;] = true gitlab_rails[\u0026#39;omniauth_block_auto_created_users\u0026#39;] = true gitlab_rails[\u0026#39;omniauth_providers\u0026#39;] = [ { \u0026#34;name\u0026#34; =\u0026gt; \u0026#34;github\u0026#34;, \u0026#34;app_id\u0026#34; =\u0026gt; \u0026#34;Client ID\u0026#34;, \u0026#34;app_secret\u0026#34; =\u0026gt; \u0026#34;Client Secret\u0026#34;, \u0026#34;url\u0026#34; =\u0026gt; \u0026#34;https://github.com/\u0026#34;, \u0026#34;args\u0026#34; =\u0026gt; { \u0026#34;scope\u0026#34; =\u0026gt; \u0026#34;user:email\u0026#34; } } ] image: gitlab/gitlab-ce:latest hostname: gitlab.fjy8018.top restart: always networks: - devops-service-bridge ports: - \u0026#39;8443:443\u0026#39; - \u0026#39;8099:8099\u0026#39; - \u0026#39;8022:22\u0026#39; volumes: - ./srv/gitlab/config:/etc/gitlab - ./srv/gitlab/logs:/var/log/gitlab - ./srv/gitlab/data:/var/opt/gitlab - /etc/docker/certs.d:/etc/docker/certs.d networks: devops-service-bridge: driver: bridge ","date":"2019-04-03T16:30:00+08:00","permalink":"/archives/205/","title":"DevOps GitLab CICD 实践1——GitLab 部署"},{"content":"无Nginx代理场景 业务层通过获取请求头参数即可拿到客户端IP\nrequest.getRemoteAddr(); 一级Nginx代理 使用代理后直接读取请求头参数会读取到代理服务器的IP地址，而非真实客户端IP\n解决方法是添加Nginx请求头参数提前保存客户端IP\nnginx.conf配置加入内容\nlocation / { ... # IP地址转发 proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Real-Port $remote_port; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } 业务层读取nginx配置的请求头参数即可\npublic static String getIpAddr(HttpServletRequest request) { String ip = request.getHeader(\u0026#34;X-Real-IP\u0026#34;); if (ip == null || ip.length() == 0 || \u0026#34; unknown \u0026#34;.equalsIgnoreCase(ip)) { ip = request.getHeader(\u0026#34;Proxy-Client-IP\u0026#34;); log.info(\u0026#34;【Proxy-Client-IP】 {}\u0026#34;, ip); } if (ip == null || ip.length() == 0 || \u0026#34; unknown \u0026#34;.equalsIgnoreCase(ip)) { ip = request.getHeader(\u0026#34;WL-Proxy-Client-IP\u0026#34;); log.info(\u0026#34;【WL-Proxy-Client-IP】{}\u0026#34;, ip); } if (ip == null || ip.length() == 0 || \u0026#34; unknown \u0026#34;.equalsIgnoreCase(ip)) { ip = request.getHeader(\u0026#34;X-Forwarded-For\u0026#34;); log.info(\u0026#34;【X-Forwarded-For】{}\u0026#34;, ip); } if (ip == null || ip.length() == 0 || \u0026#34; unknown \u0026#34;.equalsIgnoreCase(ip)) { ip = request.getRemoteAddr(); log.info(\u0026#34;【unknown】{}\u0026#34;, ip); } return ip; } 多级Nginx代理 若存在多级Nginx代理，则需要在第一级代理时获取客户端IP，在后续代理逐层传递\n第一级代理配置同上，第N级代理配置nginx.conf如下\nlocation /{ # IP地址转发 proxy_set_header X-Real-IP $X-Real-IP; proxy_set_header X-Real-Port $X-Real-Port; proxy_set_header X-Forwarded-For $X-Forwarded-For; } 业务层保持不变即可读取到传递的IP地址\n阿里云CDN转发 在已经存在nginx代理的场景下，加入CDN后源客户端IP在CDN处被转发，故第一级nginx代理使用$remote_addr参数读取到的是CDN服务的地址，而根据一般约定，CDN转发会将IP地址存放在X-Forwarded-For参数下\n修改顶级Nginx配置以支持优先获取CDN代理地址，也可以通过修改业务层读取优先级实现\nlocation / { ... # IP地址转发 # proxy_set_header X-Real-IP $remote_addr; # proxy_set_header X-Real-Port $remote_port; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } 但经过对阿里云CDN测试，发现X-Forwarded-For下不仅包含真实客户端IP也包含CDN服务IP，故业务层需要做一定的处理进行区分\n如图，第一个为真实客户端IP，第二个为CDN代理IP\npublic static String getIpAddr(HttpServletRequest request) { String ip = request.getHeader(\u0026#34;X-Real-IP\u0026#34;); if (ip == null || ip.length() == 0 || \u0026#34; unknown \u0026#34;.equalsIgnoreCase(ip)) { ip = request.getHeader(\u0026#34;Proxy-Client-IP\u0026#34;); log.info(\u0026#34;【Proxy-Client-IP】 {}\u0026#34;, ip); } if (ip == null || ip.length() == 0 || \u0026#34; unknown \u0026#34;.equalsIgnoreCase(ip)) { ip = request.getHeader(\u0026#34;WL-Proxy-Client-IP\u0026#34;); log.info(\u0026#34;【WL-Proxy-Client-IP】{}\u0026#34;, ip); } if (ip == null || ip.length() == 0 || \u0026#34; unknown \u0026#34;.equalsIgnoreCase(ip)) { ip = request.getHeader(\u0026#34;X-Forwarded-For\u0026#34;); if (ip.contains(\u0026#34;,\u0026#34;)) { // 通过阿里云CDN转发后可能读取到2个IP地址 String[] cdnMutilIp = ip.split(\u0026#34;,\u0026#34;); ip = cdnMutilIp[0]; } log.info(\u0026#34;【X-Forwarded-For】{}\u0026#34;, ip); } if (ip == null || ip.length() == 0 || \u0026#34; unknown \u0026#34;.equalsIgnoreCase(ip)) { ip = request.getRemoteAddr(); log.info(\u0026#34;【unknown】{}\u0026#34;, ip); } return ip; } } ","date":"2019-04-03T10:01:44+08:00","permalink":"/archives/204/","title":"阿里云CDN + nginx多级代理获取客户端IP"},{"content":"文档指南 官方文档\n查看官方Docker安装指南，发现只提供了镜像地址，并未给出详细操作步骤\nWith Docker Tyk has three containers that are available to set up a Docker installation:\nThe Tyk Gateway container The Tyk Dashboard container The Tyk Pump container All three are required for a full deployment. We recommend that each container is installed on a separate machine for optimum performance.\n实践 Tyk Gateway container 参考镜像指南，此处的目标是搭建带Dashboard的Tyk网关\nGateway - Pro installation with Dashboard The gateway in a Pro installation is dependent on the dashboard service. We will assume that the dashboard service is installed, up and running. If not, we would recommend that you follow the dashboard installation guide here:\nhttps://github.com/TykTechnologies/tyk-dashboard-docker\nThe gateway relies upon the dashboard service to load it\u0026rsquo;s api definitions \u0026amp; proxy configurations. As such, there is no need to mount any app directory.\ndocker run -d \\ --name tyk_gateway \\ --network tyk \\ -p 8080:8080 \\ -v $(pwd)/tyk.with_dashboard.conf:/opt/tyk-gateway/tyk.conf \\ tykio/tyk-gateway:latest 文档中说明搭建网关需要先搭建Dashboard\nTyk Dashboard container 安装指南，指南的第六步指向的连接已经失效，由此推断之前存在一键安装脚本\nQuickstart Ensure you have set up the Redis, MongoDB and Tyk Gateway containers\nSet up the docker instance IP as the Dashboard hostname (in your /etc/hosts file or as a DNS):\n127.0.0.1 dashboard.tyk.docker\nRun the Dashboard\ndocker run -d --name tyk_dashboard -p 3000:3000 --link tyk_redis:redis --link tyk_mongo:mongo --link tyk_gateway:tyk_gateway tykio/tyk-dashboard\nYou should now be able to access your Dashboard at http://dashboard.tyk.docker:3000/ (note for OSX users, replace 127.0.0.1 with whatever IP address your docker VM runs)\nEnter your Dashboard License. Go to http://dashboard.tyk.docker:3000/. You will see a screen asking for a license, enter it in the section marked “Already have a license?” and click Use this license.\nGrab the bootstrap script from our tyk-dashboard github repo and run:\n./bootstrap.sh dashboard.tyk.docker\nTo use an external configuration files, use the -v option to mount it over /opt/tyk-dashboard/tyk_analytics.conf\n经过查找发现了旧版指南链接仍然可以访问，但不建议使用，旧版指南\n按照文档的说法需要申请许可证，官方提供开发许可证，提供邮箱即可获取12个月的授权，申请地址\n申请后根据文档提供的Docker启动语句准备环境和Docker-compose文件便于一键启动\ndocker-compose.yml文件\nversion: \u0026#34;3.1\u0026#34; services: tyk_gateway: image: tykio/tyk-gateway:latest restart: always networks: - tyk-service-bridge volumes: - /www/wwwroot/tyk-free/apps:/opt/tyk-gateway/apps depends_on: - tyk_dashboard tyk_dashboard: image: tykio/tyk-dashboard restart: always networks: - tyk-service-bridge ports: - 3000:3000 depends_on: - mongo - redis mongo: image: mongo restart: always networks: - tyk-service-bridge redis: image: redis:4.0-alpine restart: always networks: - tyk-service-bridge networks: tyk-service-bridge: driver: bridge 启动\n$ docker-compose up 放行防火墙\n$ firewall-cmd --zone=public --add-port=3000/tcp --permanent $ firewall-cmd --reload 访问浏览器IP:3000输入秘钥\n许可证已经发送到注册邮件中，激活后可登陆\n但此时没有默认的用户名和密码，并且暂时未找到配置的方式（差评），需要通过官方脚本注册用户名和密码\n但官方脚本依赖Python环境，若主机没有python环境则无法执行\n启动脚本：\n#!/bin/bash # Usage ./bootstrap.sh DASHBOARD_HOSTNAME LOCALIP=$1 RANDOM_USER=$(env LC_CTYPE=C tr -dc \u0026#34;a-z0-9\u0026#34; \u0026lt; /dev/urandom | head -c 10) PASS=\u0026#34;test123\u0026#34; echo \u0026#34;Creating Organisation\u0026#34; ORGDATA=$(curl --silent --header \u0026#34;admin-auth: 12345\u0026#34; --header \u0026#34;Content-Type:application/json\u0026#34; --data \u0026#39;{\u0026#34;owner_name\u0026#34;: \u0026#34;Default Org.\u0026#34;,\u0026#34;owner_slug\u0026#34;: \u0026#34;default\u0026#34;, \u0026#34;cname_enabled\u0026#34;: true, \u0026#34;cname\u0026#34;: \u0026#34;\u0026#34;}\u0026#39; http://$LOCALIP:3000/admin/organisations 2\u0026gt;\u0026amp;1) #echo $ORGDATA ORGID=$(echo $ORGDATA | python -c \u0026#39;import json,sys;obj=json.load(sys.stdin);print(obj[\u0026#34;Meta\u0026#34;])\u0026#39;) echo \u0026#34;ORGID: $ORGID\u0026#34; echo \u0026#34;Adding new user\u0026#34; USER_DATA=$(curl --silent --header \u0026#34;admin-auth: 12345\u0026#34; --header \u0026#34;Content-Type:application/json\u0026#34; --data \u0026#39;{\u0026#34;first_name\u0026#34;: \u0026#34;John\u0026#34;,\u0026#34;last_name\u0026#34;: \u0026#34;Smith\u0026#34;,\u0026#34;email_address\u0026#34;: \u0026#34;\u0026#39;$RANDOM_USER\u0026#39;@example.com\u0026#34;,\u0026#34;password\u0026#34;:\u0026#34;\u0026#39;$PASS\u0026#39;\u0026#34;, \u0026#34;active\u0026#34;: true,\u0026#34;org_id\u0026#34;: \u0026#34;\u0026#39;$ORGID\u0026#39;\u0026#34;}\u0026#39; http://$LOCALIP:3000/admin/users 2\u0026gt;\u0026amp;1) #echo $USER_DATA USER_CODE=$(echo $USER_DATA | python -c \u0026#39;import json,sys;obj=json.load(sys.stdin);print(obj[\u0026#34;Message\u0026#34;])\u0026#39;) echo \u0026#34;USER AUTH: $USER_CODE\u0026#34; USER_LIST=$(curl --silent --header \u0026#34;authorization: $USER_CODE\u0026#34; http://$LOCALIP:3000/api/users 2\u0026gt;\u0026amp;1) #echo $USER_LIST USER_ID=$(echo $USER_LIST | python -c \u0026#39;import json,sys;obj=json.load(sys.stdin);print(obj[\u0026#34;users\u0026#34;][0][\u0026#34;id\u0026#34;])\u0026#39;) echo \u0026#34;NEW ID: $USER_ID\u0026#34; echo \u0026#34;Setting password\u0026#34; OK=$(curl --silent --header \u0026#34;authorization: $USER_CODE\u0026#34; --header \u0026#34;Content-Type:application/json\u0026#34; http://$LOCALIP:3000/api/users/$USER_ID/actions/reset --data \u0026#39;{\u0026#34;new_password\u0026#34;:\u0026#34;\u0026#39;$PASS\u0026#39;\u0026#34;}\u0026#39;) echo \u0026#34;\u0026#34; echo \u0026#34;DONE\u0026#34; echo \u0026#34;====\u0026#34; echo \u0026#34;Login at http://$LOCALIP:3000/\u0026#34; echo \u0026#34;User: $RANDOM_USER@example.com\u0026#34; echo \u0026#34;Pass: $PASS\u0026#34; echo \u0026#34;\u0026#34; 为了方便使用，此处创建了包含启动脚本的Docker镜像\n仓库地址：fjy8018/tyk-bootstrap\n使用方法：\n$ docker run --rm -it --network tyk-free_tyk-service-bridge -e TARGET_IP=tyk_dashboard fjy8018/tyk-bootstrap:0.1 随即获得登陆名和密码\n登陆成功\n","date":"2019-03-31T15:57:00+08:00","permalink":"/archives/203/","title":"Tyk网关Docker安装"},{"content":"JPA 官方文档 官方文档\n5.1.2. Annotation-based Configuration The Spring Data JPA repositories support can be activated not only through an XML namespace but also by using an annotation through JavaConfig, as shown in the following example:\nExample 51. Spring Data JPA repositories using JavaConfig\n@Configuration @EnableJpaRepositories @EnableTransactionManagement class ApplicationConfig { @Bean public DataSource dataSource() { EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder(); return builder.setType(EmbeddedDatabaseType.HSQL).build(); } @Bean public LocalContainerEntityManagerFactoryBean entityManagerFactory() { HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter(); vendorAdapter.setGenerateDdl(true); LocalContainerEntityManagerFactoryBean factory = new LocalContainerEntityManagerFactoryBean(); factory.setJpaVendorAdapter(vendorAdapter); factory.setPackagesToScan(\u0026#34;com.acme.domain\u0026#34;); factory.setDataSource(dataSource()); return factory; } @Bean public PlatformTransactionManager transactionManager(EntityManagerFactory entityManagerFactory) { JpaTransactionManager txManager = new JpaTransactionManager(); txManager.setEntityManagerFactory(entityManagerFactory); return txManager; } } 文档中说明，基于注解的配置包含三个关键Bean：\nPlatformTransactionManager——事务管理 LocalContainerEntityManagerFactoryBean——实体类管理 DataSource——数据源 同时还有一个关键注解@EnableJpaRepositories\nSpringBoot官方文档 官方文档\n多数据源使用@Primary指明优先级\nYou can even go further by leveraging what DataSourceProperties does for you — that is, by providing a default embedded database with a sensible username and password if no URL is provided. You can easily initialize a DataSourceBuilder from the state of any DataSourceProperties object, so you could also inject the DataSource that Spring Boot creates automatically. However, that would split your configuration into two namespaces: url, username, password, type, and driver on spring.datasource and the rest on your custom namespace (app.datasource). To avoid that, you can redefine a custom DataSourceProperties on your custom namespace, as shown in the following example:\n@Bean @Primary @ConfigurationProperties(\u0026#34;app.datasource\u0026#34;) public DataSourceProperties dataSourceProperties() { return new DataSourceProperties(); } @Bean @ConfigurationProperties(\u0026#34;app.datasource.configuration\u0026#34;) public HikariDataSource dataSource(DataSourceProperties properties) { return properties.initializeDataSourceBuilder().type(HikariDataSource.class) .build(); } 84.2 Configure Two DataSources If you need to configure multiple data sources, you can apply the same tricks that are described in the previous section. You must, however, mark one of the DataSource instances as @Primary, because various auto-configurations down the road expect to be able to get one by type.\nIf you create your own DataSource, the auto-configuration backs off. In the following example, we provide the exact same feature set as the auto-configuration provides on the primary data source:\n@Bean @Primary @ConfigurationProperties(\u0026#34;app.datasource.first\u0026#34;) public DataSourceProperties firstDataSourceProperties() { return new DataSourceProperties(); } @Bean @Primary @ConfigurationProperties(\u0026#34;app.datasource.first.configuration\u0026#34;) public HikariDataSource firstDataSource() { return firstDataSourceProperties().initializeDataSourceBuilder() .type(HikariDataSource.class).build(); } @Bean @ConfigurationProperties(\u0026#34;app.datasource.second\u0026#34;) public BasicDataSource secondDataSource() { return DataSourceBuilder.create().type(BasicDataSource.class).build(); } 或者使用两个EntityManagers，并结合@EnableJpaRepositories进行配置\n84.10 Use Two EntityManagers Even if the default EntityManagerFactory works fine, you need to define a new one. Otherwise, the presence of the second bean of that type switches off the default. To make it easy to do, you can use the convenient EntityManagerBuilder provided by Spring Boot. Alternatively, you can just theLocalContainerEntityManagerFactoryBean directly from Spring ORM, as shown in the following example:\n// add two data sources configured as above @Bean public LocalContainerEntityManagerFactoryBean customerEntityManagerFactory( EntityManagerFactoryBuilder builder) { return builder .dataSource(customerDataSource()) .packages(Customer.class) .persistenceUnit(\u0026#34;customers\u0026#34;) .build(); } @Bean public LocalContainerEntityManagerFactoryBean orderEntityManagerFactory( EntityManagerFactoryBuilder builder) { return builder .dataSource(orderDataSource()) .packages(Order.class) .persistenceUnit(\u0026#34;orders\u0026#34;) .build(); } The configuration above almost works on its own. To complete the picture, you need to configure TransactionManagers for the two EntityManagers as well. If you mark one of them as @Primary, it could be picked up by the default JpaTransactionManager in Spring Boot. The other would have to be explicitly injected into a new instance. Alternatively, you might be able to use a JTA transaction manager that spans both.\nIf you use Spring Data, you need to configure @EnableJpaRepositories accordingly, as shown in the following example:\n@Configuration @EnableJpaRepositories(basePackageClasses = Customer.class, entityManagerFactoryRef = \u0026#34;customerEntityManagerFactory\u0026#34;) public class CustomerConfiguration { ... } @Configuration @EnableJpaRepositories(basePackageClasses = Order.class, entityManagerFactoryRef = \u0026#34;orderEntityManagerFactory\u0026#34;) public class OrderConfiguration { ... } 源码分析 自动装配类中声明了配置Bean和条件装配顺序\npackage org.springframework.boot.autoconfigure.data.jpa; /** * {@link EnableAutoConfiguration Auto-configuration} for Spring Data\u0026#39;s JPA Repositories. * \u0026lt;p\u0026gt; * Activates when there is a bean of type {@link javax.sql.DataSource} configured in the * context, the Spring Data JPA * {@link org.springframework.data.jpa.repository.JpaRepository} type is on the classpath, * and there is no other, existing * {@link org.springframework.data.jpa.repository.JpaRepository} configured. * \u0026lt;p\u0026gt; * Once in effect, the auto-configuration is the equivalent of enabling JPA repositories * using the {@link org.springframework.data.jpa.repository.config.EnableJpaRepositories} * annotation. * \u0026lt;p\u0026gt; * This configuration class will activate \u0026lt;em\u0026gt;after\u0026lt;/em\u0026gt; the Hibernate auto-configuration. * * @author Phillip Webb * @author Josh Long * @see EnableJpaRepositories */ @Configuration @ConditionalOnBean(DataSource.class) @ConditionalOnClass(JpaRepository.class) @ConditionalOnMissingBean({ JpaRepositoryFactoryBean.class, JpaRepositoryConfigExtension.class }) @ConditionalOnProperty(prefix = \u0026#34;spring.data.jpa.repositories\u0026#34;, name = \u0026#34;enabled\u0026#34;, havingValue = \u0026#34;true\u0026#34;, matchIfMissing = true) @Import(JpaRepositoriesAutoConfigureRegistrar.class) @AutoConfigureAfter({ HibernateJpaAutoConfiguration.class, TaskExecutionAutoConfiguration.class }) public class JpaRepositoriesAutoConfiguration { @Bean @Conditional(BootstrapExecutorCondition.class) public EntityManagerFactoryBuilderCustomizer entityManagerFactoryBootstrapExecutorCustomizer( Map\u0026lt;String, AsyncTaskExecutor\u0026gt; taskExecutors) { return (builder) -\u0026gt; { AsyncTaskExecutor bootstrapExecutor = determineBootstrapExecutor( taskExecutors); if (bootstrapExecutor != null) { builder.setBootstrapExecutor(bootstrapExecutor); } }; } private AsyncTaskExecutor determineBootstrapExecutor( Map\u0026lt;String, AsyncTaskExecutor\u0026gt; taskExecutors) { if (taskExecutors.size() == 1) { return taskExecutors.values().iterator().next(); } return taskExecutors .get(TaskExecutionAutoConfiguration.APPLICATION_TASK_EXECUTOR_BEAN_NAME); } private static final class BootstrapExecutorCondition extends AnyNestedCondition { BootstrapExecutorCondition() { super(ConfigurationPhase.REGISTER_BEAN); } @ConditionalOnProperty(prefix = \u0026#34;spring.data.jpa.repositories\u0026#34;, name = \u0026#34;bootstrap-mode\u0026#34;, havingValue = \u0026#34;deferred\u0026#34;, matchIfMissing = false) static class DeferredBootstrapMode { } @ConditionalOnProperty(prefix = \u0026#34;spring.data.jpa.repositories\u0026#34;, name = \u0026#34;bootstrap-mode\u0026#34;, havingValue = \u0026#34;lazy\u0026#34;, matchIfMissing = false) static class LazyBootstrapMode { } } } 引入的配置类中声明了EnableJpaRepositories注解\npackage org.springframework.boot.autoconfigure.data.jpa; class JpaRepositoriesAutoConfigureRegistrar extends AbstractRepositoryConfigurationSourceSupport { private BootstrapMode bootstrapMode = null; @Override protected Class\u0026lt;? extends Annotation\u0026gt; getAnnotation() { return EnableJpaRepositories.class; } @Override protected Class\u0026lt;?\u0026gt; getConfiguration() { return EnableJpaRepositoriesConfiguration.class; } @Override protected RepositoryConfigurationExtension getRepositoryConfigurationExtension() { return new JpaRepositoryConfigExtension(); } @Override protected BootstrapMode getBootstrapMode() { return (this.bootstrapMode == null) ? super.getBootstrapMode() : this.bootstrapMode; } @Override public void setEnvironment(Environment environment) { super.setEnvironment(environment); configureBootstrapMode(environment); } private void configureBootstrapMode(Environment environment) { String property = environment .getProperty(\u0026#34;spring.data.jpa.repositories.bootstrap-mode\u0026#34;); if (StringUtils.hasText(property)) { this.bootstrapMode = BootstrapMode .valueOf(property.toUpperCase(Locale.ENGLISH)); } } @EnableJpaRepositories private static class EnableJpaRepositoriesConfiguration {} } EnableJpaRepositories注解定义\npackage org.springframework.data.jpa.repository.config; /** * Annotation to enable JPA repositories. Will scan the package of the annotated configuration class for Spring Data * repositories by default. * * @author Oliver Gierke * @author Thomas Darimont */ @Target(ElementType.TYPE) @Retention(RetentionPolicy.RUNTIME) @Documented @Inherited @Import(JpaRepositoriesRegistrar.class) public @interface EnableJpaRepositories { String[] value() default {}; String[] basePackages() default {}; Class\u0026lt;?\u0026gt;[] basePackageClasses() default {}; Filter[] includeFilters() default {}; Filter[] excludeFilters() default {}; String repositoryImplementationPostfix() default \u0026#34;Impl\u0026#34;; String namedQueriesLocation() default \u0026#34;\u0026#34;; Key queryLookupStrategy() default Key.CREATE_IF_NOT_FOUND; Class\u0026lt;?\u0026gt; repositoryFactoryBeanClass() default JpaRepositoryFactoryBean.class; Class\u0026lt;?\u0026gt; repositoryBaseClass() default DefaultRepositoryBaseClass.class; /** * 实体管理工厂 * @return */ String entityManagerFactoryRef() default \u0026#34;entityManagerFactory\u0026#34;; /** * 事务管理 * @return */ String transactionManagerRef() default \u0026#34;transactionManager\u0026#34;; boolean considerNestedRepositories() default false; boolean enableDefaultTransactions() default true; /** * Configures when the repositories are initialized in the bootstrap lifecycle. {@link BootstrapMode#DEFAULT} * (default) means eager initialization except all repository interfaces annotated with {@link Lazy}, * {@link BootstrapMode#LAZY} means lazy by default including injection of lazy-initialization proxies into client * beans so that those can be instantiated but will only trigger the initialization upon first repository usage (i.e a * method invocation on it). This means repositories can still be uninitialized when the application context has * completed its bootstrap. {@link BootstrapMode#DEFERRED} is fundamentally the same as {@link BootstrapMode#LAZY}, * but triggers repository initialization when the application context finishes its bootstrap. * * @return * @since 2.1 */ BootstrapMode bootstrapMode() default BootstrapMode.DEFAULT; } 查找关键Bean的注册时机\npackage org.springframework.boot.autoconfigure.orm.jpa; @Configuration @ConditionalOnClass({ LocalContainerEntityManagerFactoryBean.class, EntityManager.class }) @Conditional(HibernateEntityManagerCondition.class) @EnableConfigurationProperties(JpaProperties.class) @AutoConfigureAfter({ DataSourceAutoConfiguration.class }) @Import(HibernateJpaConfiguration.class) public class HibernateJpaAutoConfiguration {} 进入配置Bean\npackage org.springframework.boot.autoconfigure.orm.jpa; @Configuration @EnableConfigurationProperties(HibernateProperties.class) @ConditionalOnSingleCandidate(DataSource.class) class HibernateJpaConfiguration extends JpaBaseConfiguration {} 在其父类JpaBaseConfiguration中声明了装配时机和条件\npackage org.springframework.boot.autoconfigure.orm.jpa; /** * 该类在2.0版本进行了重构 * @since 2.0.0 */ @Configuration @EnableConfigurationProperties(JpaProperties.class) @Import(DataSourceInitializedPublisher.Registrar.class) public abstract class JpaBaseConfiguration implements BeanFactoryAware { @Bean @ConditionalOnMissingBean public PlatformTransactionManager transactionManager() { JpaTransactionManager transactionManager = new JpaTransactionManager(); if (this.transactionManagerCustomizers != null) { // 默认使用方法名称 this.transactionManagerCustomizers.customize(transactionManager); } return transactionManager; } @Bean @ConditionalOnMissingBean public JpaVendorAdapter jpaVendorAdapter() { AbstractJpaVendorAdapter adapter = createJpaVendorAdapter(); adapter.setShowSql(this.properties.isShowSql()); adapter.setDatabase(this.properties.determineDatabase(this.dataSource)); adapter.setDatabasePlatform(this.properties.getDatabasePlatform()); adapter.setGenerateDdl(this.properties.isGenerateDdl()); return adapter; } @Bean @ConditionalOnMissingBean public EntityManagerFactoryBuilder entityManagerFactoryBuilder( JpaVendorAdapter jpaVendorAdapter, ObjectProvider\u0026lt;PersistenceUnitManager\u0026gt; persistenceUnitManager, ObjectProvider\u0026lt;EntityManagerFactoryBuilderCustomizer\u0026gt; customizers) { EntityManagerFactoryBuilder builder = new EntityManagerFactoryBuilder( jpaVendorAdapter, this.properties.getProperties(), persistenceUnitManager.getIfAvailable()); customizers.orderedStream() .forEach((customizer) -\u0026gt; customizer.customize(builder)); return builder; } @Bean @Primary @ConditionalOnMissingBean({ LocalContainerEntityManagerFactoryBean.class, EntityManagerFactory.class }) public LocalContainerEntityManagerFactoryBean entityManagerFactory( EntityManagerFactoryBuilder factoryBuilder) { Map\u0026lt;String, Object\u0026gt; vendorProperties = getVendorProperties(); customizeVendorProperties(vendorProperties); return factoryBuilder.dataSource(this.dataSource).packages(getPackagesToScan()) .properties(vendorProperties).mappingResources(getMappingResources()) .jta(isJta()).build(); } } 测试 爬完文档和源码，对需要配置Bean有了基本了解，新建工程进行多数据源测试\n以当前最新版本SpringBoot进行测试\nmaven \u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;UTF-8\u0026#34;?\u0026gt; \u0026lt;project xmlns=\u0026#34;http://maven.apache.org/POM/4.0.0\u0026#34; xmlns:xsi=\u0026#34;http://www.w3.org/2001/XMLSchema-instance\u0026#34; xsi:schemaLocation=\u0026#34;http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd\u0026#34;\u0026gt; \u0026lt;modelVersion\u0026gt;4.0.0\u0026lt;/modelVersion\u0026gt; \u0026lt;parent\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-parent\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;2.1.3.RELEASE\u0026lt;/version\u0026gt; \u0026lt;relativePath/\u0026gt; \u0026lt;!-- lookup parent from repository --\u0026gt; \u0026lt;/parent\u0026gt; \u0026lt;groupId\u0026gt;top.fjy8018\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;jpadatasource\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;0.0.1-SNAPSHOT\u0026lt;/version\u0026gt; \u0026lt;name\u0026gt;jpadatasource\u0026lt;/name\u0026gt; \u0026lt;description\u0026gt;Demo project for Spring Boot\u0026lt;/description\u0026gt; \u0026lt;properties\u0026gt; \u0026lt;java.version\u0026gt;1.8\u0026lt;/java.version\u0026gt; \u0026lt;/properties\u0026gt; \u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-data-jpa\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-web\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;mysql\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;mysql-connector-java\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;runtime\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.projectlombok\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;lombok\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-test\u0026lt;/artifactId\u0026gt; \u0026lt;scope\u0026gt;test\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; \u0026lt;build\u0026gt; \u0026lt;plugins\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-maven-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;/plugin\u0026gt; \u0026lt;/plugins\u0026gt; \u0026lt;/build\u0026gt; \u0026lt;/project\u0026gt; 实体类 通过包路径区分主库和备份库\n主库实体类\npackage top.fjy8018.jpadatasource.entity.primary; import lombok.Data; import javax.persistence.Entity; import javax.persistence.Id; /** * @author F嘉阳 * @date 2019-03-30 9:11 */ @Data @Entity public class Product { @Id private Integer id; private String name; private Integer price; } 备份库实体类\npackage top.fjy8018.jpadatasource.entity.backup; import lombok.Data; import javax.persistence.Entity; import javax.persistence.Id; /** * @author F嘉阳 * @date 2019-03-30 9:36 */ @Data @Entity(name = \u0026#34;tb_order\u0026#34;) public class Order { @Id private Integer id; private String orderName; private Integer price; } 坑一：表名关键字冲突 若此处不指定表名为tb_order则会与MySQL关键字冲突导致建表失败\n报错：\norg.hibernate.tool.schema.spi.CommandAcceptanceException: Error executing DDL \u0026#34; create table Order ( id integer not null, orderName varchar(255), price integer, primary key (id) ) engine=InnoDB\u0026#34; via JDBC Statement ... DAO package top.fjy8018.jpadatasource.repository.backup; import org.springframework.data.jpa.repository.JpaRepository; import top.fjy8018.jpadatasource.entity.backup.Order; /** * @author F嘉阳 * @date 2019-03-30 9:12 */ public interface OrderRepository extends JpaRepository\u0026lt;Order, Integer\u0026gt; { } package top.fjy8018.jpadatasource.repository.primary; import org.springframework.data.jpa.repository.JpaRepository; import top.fjy8018.jpadatasource.entity.primary.Product; /** * @author F嘉阳 * @date 2019-03-30 9:12 */ public interface ProductRepository extends JpaRepository\u0026lt;Product, Integer\u0026gt; { } 配置Bean 官方配置样例：\nBoth data sources are also bound for advanced customizations. For instance, you could configure them as follows:\napp.datasource.first.url=jdbc:mysql://localhost/first app.datasource.first.username=dbuser app.datasource.first.password=dbpass app.datasource.first.configuration.maximum-pool-size=30 app.datasource.second.url=jdbc:mysql://localhost/second app.datasource.second.username=dbuser app.datasource.second.password=dbpass app.datasource.second.max-total=30 You can apply the same concept to the secondary DataSource as well, as shown in the following example:\n@Bean @Primary @ConfigurationProperties(\u0026#34;app.datasource.first\u0026#34;) public DataSourceProperties firstDataSourceProperties() { return new DataSourceProperties(); } @Bean @Primary @ConfigurationProperties(\u0026#34;app.datasource.first.configuration\u0026#34;) public HikariDataSource firstDataSource() { return firstDataSourceProperties().initializeDataSourceBuilder() .type(HikariDataSource.class).build(); } @Bean @ConfigurationProperties(\u0026#34;app.datasource.second\u0026#34;) public DataSourceProperties secondDataSourceProperties() { return new DataSourceProperties(); } @Bean @ConfigurationProperties(\u0026#34;app.datasource.second.configuration\u0026#34;) public BasicDataSource secondDataSource() { return secondDataSourceProperties().initializeDataSourceBuilder() .type(BasicDataSource.class).build(); } The preceding example configures two data sources on custom namespaces with the same logic as Spring Boot would use in auto-configuration. Note that each configuration sub namespace provides advanced settings based on the chosen implementation.\n参照编写配置文件，此处注意SpringBoot 2.x的驱动变化\n坑二：mysql驱动类名变化 spring: datasource: first: driver-class-name: com.mysql.cj.jdbc.Driver username: financial_adminer password: financial_adminer_pass url: jdbc:mysql://localhost/jpa_test?useSSL=false\u0026amp;useUnicode=true\u0026amp;characterEncoding=UTF-8\u0026amp;serverTimezone=UTC configuration: maximum-pool-size: 30 second: driver-class-name: com.mysql.cj.jdbc.Driver username: financial_adminer password: financial_adminer_pass url: jdbc:mysql://localhost/jpa_test_bak?useSSL=false\u0026amp;useUnicode=true\u0026amp;characterEncoding=UTF-8\u0026amp;serverTimezone=UTC configuration: maximum-pool-size: 20 jpa: hibernate: ddl-auto: update show-sql: true database-platform: org.hibernate.dialect.MySQL55Dialect properties: hibernate: format_sql: true use_sql_comments: true 配置类\npackage top.fjy8018.jpadatasource.config; import com.zaxxer.hikari.HikariDataSource; import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Primary; /** * @author F嘉阳 * @date 2019-03-30 9:24 */ @Configuration public class DataAccessConfig { @Bean @Primary @ConfigurationProperties(\u0026#34;spring.datasource.first\u0026#34;) public DataSourceProperties firstDataSourceProperties() { return new DataSourceProperties(); } @Bean @Primary @ConfigurationProperties(\u0026#34;spring.datasource.first.configuration\u0026#34;) public HikariDataSource firstDataSource() { return firstDataSourceProperties().initializeDataSourceBuilder() .type(HikariDataSource.class).build(); } @Bean @ConfigurationProperties(\u0026#34;spring.datasource.second\u0026#34;) public DataSourceProperties secondDataSourceProperties() { return new DataSourceProperties(); } @Bean @ConfigurationProperties(\u0026#34;spring.datasource.second.configuration\u0026#34;) public HikariDataSource secondDataSource() { return secondDataSourceProperties().initializeDataSourceBuilder() .type(HikariDataSource.class).build(); } } 配置完成后发现只在主库自动建表，备份库无变化，属于预期情况，因为目前仅配置了三个必须Bean的其中一个\n继续配置实体管理和事务管理Bean\n官方文档样例\n84.10 Use Two EntityManagers Even if the default EntityManagerFactory works fine, you need to define a new one. Otherwise, the presence of the second bean of that type switches off the default. To make it easy to do, you can use the convenient EntityManagerBuilder provided by Spring Boot. Alternatively, you can just theLocalContainerEntityManagerFactoryBean directly from Spring ORM, as shown in the following example:\n// add two data sources configured as above @Bean public LocalContainerEntityManagerFactoryBean customerEntityManagerFactory( EntityManagerFactoryBuilder builder) { return builder .dataSource(customerDataSource()) .packages(Customer.class) .persistenceUnit(\u0026#34;customers\u0026#34;) .build(); } @Bean public LocalContainerEntityManagerFactoryBean orderEntityManagerFactory( EntityManagerFactoryBuilder builder) { return builder .dataSource(orderDataSource()) .packages(Order.class) .persistenceUnit(\u0026#34;orders\u0026#34;) .build(); } The configuration above almost works on its own. To complete the picture, you need to configure TransactionManagers for the two EntityManagers as well. If you mark one of them as @Primary, it could be picked up by the default JpaTransactionManager in Spring Boot. The other would have to be explicitly injected into a new instance. Alternatively, you might be able to use a JTA transaction manager that spans both.\nIf you use Spring Data, you need to configure @EnableJpaRepositories accordingly, as shown in the following example:\n@Configuration @EnableJpaRepositories(basePackageClasses = Customer.class, entityManagerFactoryRef = \u0026#34;customerEntityManagerFactory\u0026#34;) public class CustomerConfiguration { ... } @Configuration @EnableJpaRepositories(basePackageClasses = Order.class, entityManagerFactoryRef = \u0026#34;orderEntityManagerFactory\u0026#34;) public class OrderConfiguration { ... } 工程配置\npackage top.fjy8018.jpadatasource.config; import com.zaxxer.hikari.HikariDataSource; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Primary; import org.springframework.data.jpa.repository.config.EnableJpaRepositories; import org.springframework.orm.jpa.JpaTransactionManager; import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean; import org.springframework.transaction.PlatformTransactionManager; import top.fjy8018.jpadatasource.entity.backup.Order; import top.fjy8018.jpadatasource.entity.primary.Product; import top.fjy8018.jpadatasource.repository.backup.OrderRepository; import top.fjy8018.jpadatasource.repository.primary.ProductRepository; import javax.sql.DataSource; /** * @author F嘉阳 * @date 2019-03-30 9:24 */ @Configuration public class DataAccessConfig { @Bean @Primary @ConfigurationProperties(\u0026#34;spring.datasource.first\u0026#34;) public DataSourceProperties firstDataSourceProperties() { return new DataSourceProperties(); } @Bean @Primary @ConfigurationProperties(\u0026#34;spring.datasource.first.configuration\u0026#34;) public HikariDataSource firstDataSource() { return firstDataSourceProperties().initializeDataSourceBuilder() .type(HikariDataSource.class).build(); } @Bean @ConfigurationProperties(\u0026#34;spring.datasource.second\u0026#34;) public DataSourceProperties secondDataSourceProperties() { return new DataSourceProperties(); } @Bean @ConfigurationProperties(\u0026#34;spring.datasource.second.configuration\u0026#34;) public HikariDataSource secondDataSource() { return secondDataSourceProperties().initializeDataSourceBuilder() .type(HikariDataSource.class).build(); } @Bean public LocalContainerEntityManagerFactoryBean firstEntityManagerFactory( EntityManagerFactoryBuilder builder, @Qualifier(\u0026#34;firstDataSource\u0026#34;) DataSource dataSource) { return builder .dataSource(dataSource) .packages(Product.class) .persistenceUnit(\u0026#34;first\u0026#34;) .build(); } @Bean public LocalContainerEntityManagerFactoryBean secondEntityManagerFactory( EntityManagerFactoryBuilder builder, @Qualifier(\u0026#34;secondDataSource\u0026#34;) DataSource dataSource) { return builder .dataSource(dataSource) .packages(Order.class) .persistenceUnit(\u0026#34;second\u0026#34;) .build(); } @Bean public PlatformTransactionManager primaryTransactionManager( @Qualifier(\u0026#34;firstEntityManagerFactory\u0026#34;) LocalContainerEntityManagerFactoryBean entityManagerFactory) { return new JpaTransactionManager(entityManagerFactory.getObject()); } @Bean public PlatformTransactionManager backupTransactionManager( @Qualifier(\u0026#34;secondEntityManagerFactory\u0026#34;) LocalContainerEntityManagerFactoryBean entityManagerFactory) { return new JpaTransactionManager(entityManagerFactory.getObject()); } @EnableJpaRepositories(basePackageClasses = ProductRepository.class, entityManagerFactoryRef = \u0026#34;firstEntityManagerFactory\u0026#34;) @Primary public class PrimaryConfiguration { } @EnableJpaRepositories(basePackageClasses = OrderRepository.class, entityManagerFactoryRef = \u0026#34;secondEntityManagerFactory\u0026#34;) public class secondConfiguration { } } 坑三：官方文档样例有误 此时配置直接启动会报错\nMethod requestMappingHandlerMapping in org.springframework.boot.autoconfigure.web.servlet.WebMvcAutoConfiguration$EnableWebMvcConfiguration required a single bean, but 2 were found: - firstEntityManagerFactory: defined by method \u0026#39;firstEntityManagerFactory\u0026#39; in class path resource [top/fjy8018/jpadatasource/config/DataAccessConfig.class] - secondEntityManagerFactory: defined by method \u0026#39;secondEntityManagerFactory\u0026#39; in class path resource [top/fjy8018/jpadatasource/config/DataAccessConfig.class] Action: Consider marking one of the beans as @Primary, updating the consumer to accept multiple beans, or using @Qualifier to identify the bean that should be consumed 意思是自动装配时发现两个类型一致的Bean，主要指定优先级，而官方文档对此处可能出现的问题并未提示\n指定firstEntityManagerFactory为优先Bean，同时文档样例创建了事务管理器但未配置到@EnableJpaRepositories注解中，此处也一并配置\n最终配置类如下\npackage top.fjy8018.jpadatasource.config; import com.zaxxer.hikari.HikariDataSource; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Primary; import org.springframework.data.jpa.repository.config.EnableJpaRepositories; import org.springframework.orm.jpa.JpaTransactionManager; import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean; import org.springframework.transaction.PlatformTransactionManager; import top.fjy8018.jpadatasource.entity.backup.Order; import top.fjy8018.jpadatasource.entity.primary.Product; import top.fjy8018.jpadatasource.repository.backup.OrderRepository; import top.fjy8018.jpadatasource.repository.primary.ProductRepository; import javax.sql.DataSource; /** * @author F嘉阳 * @date 2019-03-30 9:24 */ @Configuration public class DataAccessConfig { @Bean @Primary @ConfigurationProperties(\u0026#34;spring.datasource.first\u0026#34;) public DataSourceProperties firstDataSourceProperties() { return new DataSourceProperties(); } @Bean @Primary @ConfigurationProperties(\u0026#34;spring.datasource.first.configuration\u0026#34;) public HikariDataSource firstDataSource() { return firstDataSourceProperties().initializeDataSourceBuilder() .type(HikariDataSource.class).build(); } @Bean @ConfigurationProperties(\u0026#34;spring.datasource.second\u0026#34;) public DataSourceProperties secondDataSourceProperties() { return new DataSourceProperties(); } @Bean @ConfigurationProperties(\u0026#34;spring.datasource.second.configuration\u0026#34;) public HikariDataSource secondDataSource() { return secondDataSourceProperties().initializeDataSourceBuilder() .type(HikariDataSource.class).build(); } @Bean public LocalContainerEntityManagerFactoryBean firstEntityManagerFactory( EntityManagerFactoryBuilder builder, @Qualifier(\u0026#34;firstDataSource\u0026#34;) DataSource dataSource) { return builder .dataSource(dataSource) .packages(Product.class) .persistenceUnit(\u0026#34;first\u0026#34;) .build(); } @Bean public LocalContainerEntityManagerFactoryBean secondEntityManagerFactory( EntityManagerFactoryBuilder builder, @Qualifier(\u0026#34;secondDataSource\u0026#34;) DataSource dataSource) { return builder .dataSource(dataSource) .packages(Order.class) .persistenceUnit(\u0026#34;second\u0026#34;) .build(); } @Bean public PlatformTransactionManager primaryTransactionManager( @Qualifier(\u0026#34;firstEntityManagerFactory\u0026#34;) LocalContainerEntityManagerFactoryBean entityManagerFactory) { return new JpaTransactionManager(entityManagerFactory.getObject()); } @Bean public PlatformTransactionManager backupTransactionManager( @Qualifier(\u0026#34;secondEntityManagerFactory\u0026#34;) LocalContainerEntityManagerFactoryBean entityManagerFactory) { return new JpaTransactionManager(entityManagerFactory.getObject()); } @EnableJpaRepositories(basePackageClasses = ProductRepository.class, entityManagerFactoryRef = \u0026#34;firstEntityManagerFactory\u0026#34;, transactionManagerRef = \u0026#34;primaryTransactionManager\u0026#34;) @Primary public class PrimaryConfiguration { } @EnableJpaRepositories(basePackageClasses = OrderRepository.class, entityManagerFactoryRef = \u0026#34;secondEntityManagerFactory\u0026#34;, transactionManagerRef = \u0026#34;backupTransactionManager\u0026#34;) public class secondConfiguration { } } 坑四：JPA不会自动建表 配置完成后启动\n发现日志中建表SQL没有输出，同时查看数据库，确认表未创建\n但官方文档对多数据源的配置说明已经结束，令人疑惑\n自动建表语句没生成猜想可能由于与ddl相关的属性未配置导致\n进入配置文件查看ddl配置，发现默认为false\n将其配置为true后总配置文件如下\nspring: datasource: first: driver-class-name: com.mysql.cj.jdbc.Driver username: financial_adminer password: financial_adminer_pass url: jdbc:mysql://120.79.226.26/jpa_test?useSSL=false\u0026amp;useUnicode=true\u0026amp;characterEncoding=UTF-8\u0026amp;serverTimezone=UTC configuration: maximum-pool-size: 30 second: driver-class-name: com.mysql.cj.jdbc.Driver username: financial_adminer password: financial_adminer_pass url: jdbc:mysql://120.79.226.26/jpa_test_bak?useSSL=false\u0026amp;useUnicode=true\u0026amp;characterEncoding=UTF-8\u0026amp;serverTimezone=UTC configuration: maximum-pool-size: 20 jpa: hibernate: ddl-auto: update show-sql: true database-platform: org.hibernate.dialect.MySQL55Dialect properties: hibernate: format_sql: true use_sql_comments: true generate-ddl: true 启动工程发现自动建表生效\n查看数据库情况，与预期一致，主库只包含主库的表，备份库只包含备份库的表，多数据配置完成\n坑五：hibernate默认配置被覆盖 若在单数据源配置下应当无需配置该项，同时发现本应该默认将驼峰属性名映射为_拼接名称策略也失效，说明自定义配置源时覆盖了默认配置或者导致默认配置失效\n查询源码分析原因，发现hibernate相关配置均在HibernateProperties配置类中\npackage org.springframework.boot.autoconfigure.orm.jpa; /** * Configuration properties for Hibernate. * * @author Stephane Nicoll * @since 2.1.0 * @see JpaProperties */ @ConfigurationProperties(\u0026#34;spring.jpa.hibernate\u0026#34;) public class HibernateProperties {} 而该类在HibernateJpaConfiguration中进行注入，但配置类中进行了条件装配注解@ConditionalOnSingleCandidate标注，导致在多数据源情况下不满足该配置，类没有被装配到Spring Bean中\npackage org.springframework.boot.autoconfigure.orm.jpa; /** * {@link JpaBaseConfiguration} implementation for Hibernate. * @since 2.0.0 */ @Configuration @EnableConfigurationProperties(HibernateProperties.class) @ConditionalOnSingleCandidate(DataSource.class) class HibernateJpaConfiguration extends JpaBaseConfiguration { private final HibernateProperties hibernateProperties; HibernateJpaConfiguration(DataSource dataSource, JpaProperties jpaProperties, ConfigurableListableBeanFactory beanFactory, ObjectProvider\u0026lt;JtaTransactionManager\u0026gt; jtaTransactionManager, ObjectProvider\u0026lt;TransactionManagerCustomizers\u0026gt; transactionManagerCustomizers, HibernateProperties hibernateProperties, ObjectProvider\u0026lt;Collection\u0026lt;DataSourcePoolMetadataProvider\u0026gt;\u0026gt; metadataProviders, ObjectProvider\u0026lt;SchemaManagementProvider\u0026gt; providers, ObjectProvider\u0026lt;PhysicalNamingStrategy\u0026gt; physicalNamingStrategy, ObjectProvider\u0026lt;ImplicitNamingStrategy\u0026gt; implicitNamingStrategy, ObjectProvider\u0026lt;HibernatePropertiesCustomizer\u0026gt; hibernatePropertiesCustomizers) { super(dataSource, jpaProperties, jtaTransactionManager, transactionManagerCustomizers); this.hibernateProperties = hibernateProperties; this.defaultDdlAutoProvider = new HibernateDefaultDdlAutoProvider(providers); this.poolMetadataProvider = new CompositeDataSourcePoolMetadataProvider( metadataProviders.getIfAvailable()); this.hibernatePropertiesCustomizers = determineHibernatePropertiesCustomizers( physicalNamingStrategy.getIfAvailable(), implicitNamingStrategy.getIfAvailable(), beanFactory, hibernatePropertiesCustomizers.orderedStream() .collect(Collectors.toList())); } ... } 解决方法就只能自己工程内实现该类的配置，注意，Spring 2.x版本自动装配变化较大，当前配置基于最新版2.1.3有效，并不代表对后续版本一定兼容\n工程配置\npackage top.fjy8018.jpadatasource.config; import com.zaxxer.hikari.HikariDataSource; import org.hibernate.boot.model.naming.ImplicitNamingStrategy; import org.hibernate.boot.model.naming.PhysicalNamingStrategy; import org.hibernate.cfg.AvailableSettings; import org.springframework.beans.factory.ObjectProvider; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.beans.factory.config.ConfigurableListableBeanFactory; import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties; import org.springframework.boot.autoconfigure.orm.jpa.HibernateProperties; import org.springframework.boot.autoconfigure.orm.jpa.HibernatePropertiesCustomizer; import org.springframework.boot.autoconfigure.orm.jpa.HibernateSettings; import org.springframework.boot.autoconfigure.orm.jpa.JpaProperties; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.boot.jdbc.EmbeddedDatabaseConnection; import org.springframework.boot.jdbc.SchemaManagement; import org.springframework.boot.jdbc.SchemaManagementProvider; import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Primary; import org.springframework.data.jpa.repository.config.EnableJpaRepositories; import org.springframework.orm.hibernate5.SpringBeanContainer; import org.springframework.orm.jpa.JpaTransactionManager; import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean; import org.springframework.transaction.PlatformTransactionManager; import org.springframework.util.ClassUtils; import top.fjy8018.jpadatasource.entity.backup.Order; import top.fjy8018.jpadatasource.entity.primary.Product; import top.fjy8018.jpadatasource.repository.backup.OrderRepository; import top.fjy8018.jpadatasource.repository.primary.ProductRepository; import javax.sql.DataSource; import java.util.ArrayList; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.function.Supplier; import java.util.stream.Collectors; import java.util.stream.StreamSupport; /** * @author F嘉阳 * @date 2019-03-30 9:24 */ @Configuration public class DataAccessConfig { private final ObjectProvider\u0026lt;SchemaManagementProvider\u0026gt; providers; private final HibernateProperties hibernateProperties; private final JpaProperties properties; private final ObjectProvider\u0026lt;PhysicalNamingStrategy\u0026gt; physicalNamingStrategy; private final ObjectProvider\u0026lt;ImplicitNamingStrategy\u0026gt; implicitNamingStrategy; private final ConfigurableListableBeanFactory beanFactory; private final ObjectProvider\u0026lt;HibernatePropertiesCustomizer\u0026gt; hibernatePropertiesCustomizers; @Autowired public DataAccessConfig(ObjectProvider\u0026lt;SchemaManagementProvider\u0026gt; providers, HibernateProperties hibernateProperties, JpaProperties properties, ObjectProvider\u0026lt;PhysicalNamingStrategy\u0026gt; physicalNamingStrategy, ObjectProvider\u0026lt;ImplicitNamingStrategy\u0026gt; implicitNamingStrategy, ConfigurableListableBeanFactory beanFactory, ObjectProvider\u0026lt;HibernatePropertiesCustomizer\u0026gt; hibernatePropertiesCustomizers) { this.providers = providers; this.hibernateProperties = hibernateProperties; this.properties = properties; this.physicalNamingStrategy = physicalNamingStrategy; this.implicitNamingStrategy = implicitNamingStrategy; this.beanFactory = beanFactory; this.hibernatePropertiesCustomizers = hibernatePropertiesCustomizers; } @Bean @Primary @ConfigurationProperties(\u0026#34;spring.datasource.first\u0026#34;) public DataSourceProperties firstDataSourceProperties() { return new DataSourceProperties(); } @Bean @Primary @ConfigurationProperties(\u0026#34;spring.datasource.first.configuration\u0026#34;) public HikariDataSource firstDataSource() { return firstDataSourceProperties().initializeDataSourceBuilder() .type(HikariDataSource.class).build(); } @Bean @ConfigurationProperties(\u0026#34;spring.datasource.second\u0026#34;) public DataSourceProperties secondDataSourceProperties() { return new DataSourceProperties(); } @Bean @ConfigurationProperties(\u0026#34;spring.datasource.second.configuration\u0026#34;) public HikariDataSource secondDataSource() { return secondDataSourceProperties().initializeDataSourceBuilder() .type(HikariDataSource.class).build(); } @Bean @Primary public LocalContainerEntityManagerFactoryBean firstEntityManagerFactory( EntityManagerFactoryBuilder builder, @Qualifier(\u0026#34;firstDataSource\u0026#34;) DataSource dataSource) { return builder .dataSource(dataSource) // 加入规约配置 .properties(getVendorProperties(dataSource)) .packages(Product.class) .persistenceUnit(\u0026#34;first\u0026#34;) .build(); } @Bean public LocalContainerEntityManagerFactoryBean secondEntityManagerFactory( EntityManagerFactoryBuilder builder, @Qualifier(\u0026#34;secondDataSource\u0026#34;) DataSource dataSource) { return builder .dataSource(dataSource) .properties(getVendorProperties(dataSource)) .packages(Order.class) .persistenceUnit(\u0026#34;second\u0026#34;) .build(); } @Bean public PlatformTransactionManager primaryTransactionManager( @Qualifier(\u0026#34;firstEntityManagerFactory\u0026#34;) LocalContainerEntityManagerFactoryBean entityManagerFactory) { return new JpaTransactionManager(entityManagerFactory.getObject()); } @Bean public PlatformTransactionManager backupTransactionManager( @Qualifier(\u0026#34;secondEntityManagerFactory\u0026#34;) LocalContainerEntityManagerFactoryBean entityManagerFactory) { return new JpaTransactionManager(entityManagerFactory.getObject()); } @EnableJpaRepositories(basePackageClasses = ProductRepository.class, entityManagerFactoryRef = \u0026#34;firstEntityManagerFactory\u0026#34;, transactionManagerRef = \u0026#34;primaryTransactionManager\u0026#34;) @Primary public class PrimaryConfiguration { } @EnableJpaRepositories(basePackageClasses = OrderRepository.class, entityManagerFactoryRef = \u0026#34;secondEntityManagerFactory\u0026#34;, transactionManagerRef = \u0026#34;backupTransactionManager\u0026#34;) public class secondConfiguration { } /** * 获取配置文件信息 * * @param dataSource * @return */ private Map\u0026lt;String, Object\u0026gt; getVendorProperties(DataSource dataSource) { List\u0026lt;HibernatePropertiesCustomizer\u0026gt; hibernatePropertiesCustomizers = determineHibernatePropertiesCustomizers( physicalNamingStrategy.getIfAvailable(), implicitNamingStrategy.getIfAvailable(), beanFactory, this.hibernatePropertiesCustomizers.orderedStream() .collect(Collectors.toList())); Supplier\u0026lt;String\u0026gt; defaultDdlMode = () -\u0026gt; new HibernateDefaultDdlAutoProvider(providers) .getDefaultDdlAuto(dataSource); return new LinkedHashMap\u0026lt;\u0026gt;(this.hibernateProperties.determineHibernateProperties( properties.getProperties(), new HibernateSettings().ddlAuto(defaultDdlMode) .hibernatePropertiesCustomizers( hibernatePropertiesCustomizers))); } /** * 命名策略自动判断 * * @param physicalNamingStrategy * @param implicitNamingStrategy * @param beanFactory * @param hibernatePropertiesCustomizers * @return */ private List\u0026lt;HibernatePropertiesCustomizer\u0026gt; determineHibernatePropertiesCustomizers( PhysicalNamingStrategy physicalNamingStrategy, ImplicitNamingStrategy implicitNamingStrategy, ConfigurableListableBeanFactory beanFactory, List\u0026lt;HibernatePropertiesCustomizer\u0026gt; hibernatePropertiesCustomizers) { List\u0026lt;HibernatePropertiesCustomizer\u0026gt; customizers = new ArrayList\u0026lt;\u0026gt;(); if (ClassUtils.isPresent( \u0026#34;org.hibernate.resource.beans.container.spi.BeanContainer\u0026#34;, getClass().getClassLoader())) { customizers .add((properties) -\u0026gt; properties.put(AvailableSettings.BEAN_CONTAINER, new SpringBeanContainer(beanFactory))); } if (physicalNamingStrategy != null || implicitNamingStrategy != null) { customizers.add(new NamingStrategiesHibernatePropertiesCustomizer( physicalNamingStrategy, implicitNamingStrategy)); } customizers.addAll(hibernatePropertiesCustomizers); return customizers; } /** * 自动进行建表操作 */ class HibernateDefaultDdlAutoProvider implements SchemaManagementProvider { private final Iterable\u0026lt;SchemaManagementProvider\u0026gt; providers; HibernateDefaultDdlAutoProvider(Iterable\u0026lt;SchemaManagementProvider\u0026gt; providers) { this.providers = providers; } public String getDefaultDdlAuto(DataSource dataSource) { if (!EmbeddedDatabaseConnection.isEmbedded(dataSource)) { return \u0026#34;none\u0026#34;; } SchemaManagement schemaManagement = getSchemaManagement(dataSource); if (SchemaManagement.MANAGED.equals(schemaManagement)) { return \u0026#34;none\u0026#34;; } return \u0026#34;create-drop\u0026#34;; } @Override public SchemaManagement getSchemaManagement(DataSource dataSource) { return StreamSupport.stream(this.providers.spliterator(), false) .map((provider) -\u0026gt; provider.getSchemaManagement(dataSource)) .filter(SchemaManagement.MANAGED::equals).findFirst() .orElse(SchemaManagement.UNMANAGED); } } private static class NamingStrategiesHibernatePropertiesCustomizer implements HibernatePropertiesCustomizer { private final PhysicalNamingStrategy physicalNamingStrategy; private final ImplicitNamingStrategy implicitNamingStrategy; NamingStrategiesHibernatePropertiesCustomizer( PhysicalNamingStrategy physicalNamingStrategy, ImplicitNamingStrategy implicitNamingStrategy) { this.physicalNamingStrategy = physicalNamingStrategy; this.implicitNamingStrategy = implicitNamingStrategy; } /** * 数据库命名映射策略 * * @param hibernateProperties the JPA vendor properties to customize */ @Override public void customize(Map\u0026lt;String, Object\u0026gt; hibernateProperties) { if (this.physicalNamingStrategy != null) { hibernateProperties.put(\u0026#34;hibernate.physical_naming_strategy\u0026#34;, this.physicalNamingStrategy); } if (this.implicitNamingStrategy != null) { hibernateProperties.put(\u0026#34;hibernate.implicit_naming_strategy\u0026#34;, this.implicitNamingStrategy); } } } } 配置Bean完成后则规约配置会生效，便无需在配置文件显示配置DDL语句执行开关\n启动日志也可看出结果符合预期\n总结 由此发现，在SpringBoot版本迁移过程中不一定互相兼容，同时官方文档尽管全面，但随着版本变化文档也有不准确情况。而如果完全依靠爬源码复制源码配置也是无法实现最简单化的配置，需要结合文档、源码、日志逐一分析\n工程源码已开放，注意配置自己的数据库连接地址，工程内的连接地址已经失效\nGitHub源码\n","date":"2019-03-30T10:44:00+08:00","permalink":"/archives/202/","title":"SpringBoot2.x Data JPA 多数据源爬坑"},{"content":"第1天 Jewels and Stones （771） 题号771 Jewels and Stones\n题目描述：\n给定字符串J 代表石头中宝石的类型，和字符串 S代表你拥有的石头。 S 中每个字符代表了一种你拥有的石头的类型，你想知道你拥有的石头中有多少是宝石。\nJ 中的字母不重复，J 和 S中的所有字符都是字母。字母区分大小写，因此\u0026quot;a\u0026quot;和\u0026quot;A\u0026quot;是不同类型的石头。\n解题思路：字符串遍历比对\n心得：查看了排名较前的解法，大部分也是暴力比对，少数取巧解法获得更好的时间复杂度\n排名最前的解法\n第2天 Unique Email Addresse （929） 题号929 Unique Email Addresses\n题目描述：\n每封电子邮件都由一个本地名称和一个域名组成，以 @ 符号分隔。\n例如，在 alice@leetcode.com中， alice 是本地名称，而 leetcode.com 是域名。\n除了小写字母，这些电子邮件还可能包含 '.' 或 '+'。\n如果在电子邮件地址的本地名称部分中的某些字符之间添加句点（'.'），则发往那里的邮件将会转发到本地名称中没有点的同一地址。例如，\u0026quot;alice.z@leetcode.com” 和 “alicez@leetcode.com” 会转发到同一电子邮件地址。 （请注意，此规则不适用于域名。）\n如果在本地名称中添加加号（'+'），则会忽略第一个加号后面的所有内容。这允许过滤某些电子邮件，例如 m.y+name@email.com 将转发到 my@email.com。 （同样，此规则不适用于域名。）\n可以同时使用这两个规则。\n给定电子邮件列表 emails，我们会向列表中的每个地址发送一封电子邮件。实际收到邮件的不同地址有多少？\n示例：\n输入：[\u0026#34;test.email+alex@leetcode.com\u0026#34;,\u0026#34;test.e.mail+bob.cathy@leetcode.com\u0026#34;,\u0026#34;testemail+david@lee.tcode.com\u0026#34;] 输出：2 解释：实际收到邮件的是 \u0026#34;testemail@leetcode.com\u0026#34; 和 \u0026#34;testemail@lee.tcode.com\u0026#34;。 提示：\n1 \u0026lt;= emails[i].length \u0026lt;= 100 1 \u0026lt;= emails.length \u0026lt;= 100 每封 emails[i] 都包含有且仅有一个 '@' 字符。 解题思路：字符串替换+去除重复元素\n心得体会：网站测试用例不全导致排名较前的解法有问题，并不能完全符合题目要求，自己的代码时间复杂度还是较高\n国外Java精简正则表达式解法 第3天 Sort Array By Parity （905） 题号905 Sort Array By Parity 题目描述：给定一个非负整数数组 A，返回一个由 A 的所有偶数元素组成的数组，后面跟 A 的所有奇数元素。 解题思路：双指针 心得体会：经典解法，时间复杂度为O(n)，遍历数组，将偶数从前往后存放，奇数则从后往前存放 第4天 Encode and Decode TinyURL（535） 题号535 Encode and Decode TinyURL\n题目描述：\nTinyURL是一种URL简化服务， 比如：当你输入一个URL https://leetcode.com/problems/design-tinyurl 时，它将返回一个简化的URL http://tinyurl.com/4e9iAk.\n解题思路：短网址算法简化版\n心得体会：哈希运算后再转16进制即可明显缩短网址长度，题目没有要求生成的长度，故没有实现真正的短网址算法，也有不运算通过集合直接存储返回下标的，目前暂时没有在讨论区看到更好的Java解法\n还有更加取巧的解法，直接暴力返回。。然而还显示739 / 739 个通过测试用例。。排名较前的多为该解法\n第5天 Subsets（78） 题号78 Subsets 题目描述：给定一组不含重复元素的整数数组 nums，返回该数组所有可能的子集（幂集）。 说明：解集不能包含重复的子集。 解题思路：深度优先或者广度优先算法 心得体会：思考了一段时间，之后在讨论区看到BFS、DFS解法，最后看到使用排序和递归结合的DFS简洁解法，拓宽了思路 排序和递归结合的DFS解法\n第6天 Minimum Add to Make Parentheses Valid（921） 题号921 Minimum Add to Make Parentheses Valid\n题目描述：\n给定一个由 '(' 和 ')' 括号组成的字符串 S，我们需要添加最少的括号（ '(' 或是 ')'，可以在任何位置），以使得到的括号字符串有效。\n从形式上讲，只有满足下面几点之一，括号字符串才是有效的：\n它是一个空字符串，或者 它可以被写成 AB （A 与 B 连接）, 其中 A 和 B 都是有效字符串，或者 它可以被写作 (A)，其中 A 是有效字符串。 给定一个括号字符串，返回为使结果字符串有效而必须添加的最少括号数。\n解题思路：栈，遇到左括号则右括号入栈，遇到右括号则弹出栈顶元素并判断其是否为左括号，栈为空或者判断为false都将计数器自增，最终返回栈大小与计数器之和即可\n心得体会：按照题目标签选题为栈，故使用了栈来实现，但耗时其实比直接使用左右括号计数器要长\n左右括号计数器解法\n第7天 Reverse String（344） 题号 344 Reverse String 题目描述：编写一个函数，其作用是将输入的字符串反转过来。 解题思路：双指针 心得体会：简单题，两个指针分别从前往后和从后往前遍历并交换，但从讨论区也看到了更多的思路，例如使用ACII位运算解法等 ACII位运算解法\n第8天 Remove Linked List Elements（203） 题号 203 Remove Linked List Elements 题目描述：删除链表中等于给定值 val 的所有节点。 解题思路： 单链表迭代删除元素 带虚拟头结点单链表迭代删除元素 递归实现 心得体会：简单题，使用指针逐个判断下一个元素是否为待删除元素，迭代直至链表尾部。题目虽然难度不大，但可以增强对链表和递归用法的理解。 非虚拟头结点实现\n虚拟头结点实现\n递归实现\n第9天 Valid Parentheses（20） 题号 20 Valid Parentheses\n题目描述：\n给定一个只包括 '('，')'，'{'，'}'，'['，']' 的字符串，判断字符串是否有效。\n有效字符串需满足：\n左括号必须用相同类型的右括号闭合。 左括号必须以正确的顺序闭合。 注意空字符串可被认为是有效字符串。\n解题思路：栈\n心得体会：简单题，栈的基础操作，但注意出栈后仍需判断括号是否匹配以及遍历完毕还需判断栈是否为空\n第10天 Binary Tree Inorder Traversal（94） 题号 94 Binary Tree Inorder Traversal\n题目描述：\n给定一个二叉树，返回它的中序 遍历。\n示例:\n输入: [1,null,2,3] 1 \\ 2 / 3 输出: [1,3,2] 解题思路：考察二叉树中序遍历递归实现和非递归实现\n心得体会：简单题，二叉树的基础操作\n递归实现\n非递归实现\n第11天 Search in a Binary Search Tree（700） 题号 700 Search in a Binary Search Tree\n题目描述：\n给定二叉搜索树（BST）的根节点和一个值。 你需要在BST中找到节点值等于给定值的节点。 返回以该节点为根的子树。 如果节点不存在，则返回 NULL。\n例如，\n给定二叉搜索树: 4 / \\ 2 7 / \\ 1 3 和值: 2 你应该返回如下子树:\n2 / \\ 1 3 在上述示例中，如果要找的值是 5，但因为没有节点值为 5，我们应该返回 NULL。\n解题思路：二叉树节点值比对、查找、递归实现\n心得体会：简单题，二叉树的查找操作\n第12天 Binary Tree Level Order Traversal II（107） 题号 107 Binary Tree Level Order Traversal II\n题目描述：\n给定一个二叉树，返回其节点值自底向上的层次遍历。 （即按从叶子节点所在层到根节点所在的层，逐层从左向右遍历）\n例如： 给定二叉树 [3,9,20,null,null,15,7],\n3 / \\ 9 20 / \\ 15 7 返回其自底向上的层次遍历为：\n[ [15,7], [9,20], [3] ] 解题思路：深度优先遍历、广度优先遍历\n心得体会：由于题目要求自底向上层次遍历，反向输出使用栈实现，考虑使用深度优先并结合队列实现，故使用非递归方式，但每层都要组合为一个集合，考虑使用双队列，解题过程中清楚这并不是最优解，后在评论区看到更优解法\n评论区赞同数较高的解法\nDFS solution:\npublic class Solution { public List\u0026lt;List\u0026lt;Integer\u0026gt;\u0026gt; levelOrderBottom(TreeNode root) { Queue\u0026lt;TreeNode\u0026gt; queue = new LinkedList\u0026lt;TreeNode\u0026gt;(); List\u0026lt;List\u0026lt;Integer\u0026gt;\u0026gt; wrapList = new LinkedList\u0026lt;List\u0026lt;Integer\u0026gt;\u0026gt;(); if(root == null) return wrapList; queue.offer(root); while(!queue.isEmpty()){ // 记录队列中根节点数量 int levelNum = queue.size(); List\u0026lt;Integer\u0026gt; subList = new LinkedList\u0026lt;Integer\u0026gt;(); for(int i=0; i\u0026lt;levelNum; i++) { if(queue.peek().left != null) queue.offer(queue.peek().left); if(queue.peek().right != null) queue.offer(queue.peek().right); subList.add(queue.poll().val); } wrapList.add(0, subList); } return wrapList; } } BFS solution:\npublic class Solution { public List\u0026lt;List\u0026lt;Integer\u0026gt;\u0026gt; levelOrderBottom(TreeNode root) { List\u0026lt;List\u0026lt;Integer\u0026gt;\u0026gt; wrapList = new LinkedList\u0026lt;List\u0026lt;Integer\u0026gt;\u0026gt;(); levelMaker(wrapList, root, 0); return wrapList; } public void levelMaker(List\u0026lt;List\u0026lt;Integer\u0026gt;\u0026gt; list, TreeNode root, int level) { if(root == null) return; if(level \u0026gt;= list.size()) { list.add(0, new LinkedList\u0026lt;Integer\u0026gt;()); } levelMaker(list, root.left, level+1); levelMaker(list, root.right, level+1); list.get(list.size()-level-1).add(root.val); } } 第13天 Binary Tree Preorder Traversal（144） 题号 144 Binary Tree Preorder Traversal\n题目描述：\n给定一个二叉树，返回它的 前序 遍历。\n示例:\n输入: [1,null,2,3] 1 \\ 2 / 3 输出: [1,2,3] 解题思路：二叉树先序遍历递归实现、非递归实现\n心得体会：简单题，二叉树的基础操作\n递归实现\n非递归实现\n第14天 Binary Tree Postorder Traversal（145） 题号 145 Binary Tree Postorder Traversal\n题目描述：\n给定一个二叉树，返回它的 后序 遍历。\n示例:\n输入: [1,null,2,3] 1 \\ 2 / 3 输出: [3,2,1] 解题思路：二叉树递归操作、双栈使用\n心得体会：递归实现较为简单，但非递归实现需要对栈有深刻的理解，通过双栈使用让每个节点在第三次遍历时输出\n递归实现\n非递归实现\n第15天 Merge Two Sorted Lists（21） 题号 21 Merge Two Sorted Lists\n题目描述：\n将两个有序链表合并为一个新的有序链表并返回。新链表是通过拼接给定的两个链表的所有节点组成的。\n示例：\n输入：1-\u0026gt;2-\u0026gt;4, 1-\u0026gt;3-\u0026gt;4 输出：1-\u0026gt;1-\u0026gt;2-\u0026gt;3-\u0026gt;4-\u0026gt;4 解题思路：链表插入操作\n心得体会：简单题，注意边界值以及可能出现两个链表长度不等的情况\n第16天 Remove Nth Node From End of List（19） 题号 19 Remove Nth Node From End of List\n题目描述：\n给定一个链表，删除链表的倒数第 n 个节点，并且返回链表的头结点。\n示例：\n给定一个链表: 1-\u0026gt;2-\u0026gt;3-\u0026gt;4-\u0026gt;5, 和 n = 2. 当删除了倒数第二个节点后，链表变为 1-\u0026gt;2-\u0026gt;3-\u0026gt;5. 说明：\n给定的 n 保证是有效的。\n进阶：\n使用一趟扫描实现\n解题思路：第一趟扫描得到链表大小、用链表大小减去n得到节点顺位下标，按链表删除元素操作即可\n心得体会：两趟扫描比较容易想到，解题后在讨论区看到评价较高的一趟扫描排序实现\n两趟扫描实现\n一趟扫描实现\nA one pass solution can be done using pointers. Move one pointer fast \u0026ndash;\u0026gt; n+1 places forward, to maintain a gap of n between the two pointers and then move both at the same speed. Finally, when the fast pointer reaches the end, the slow pointer will be n+1 places behind - just the right spot for it to be able to skip the next node.\nSince the question gives that n is valid, not too many checks have to be put in place. Otherwise, this would be necessary.\npublic ListNode removeNthFromEnd(ListNode head, int n) { ListNode start = new ListNode(0); ListNode slow = start, fast = start; slow.next = head; //Move fast in front so that the gap between slow and fast becomes n for(int i=1; i\u0026lt;=n+1; i++) { fast = fast.next; } //Move fast to the end, maintaining the gap while(fast != null) { slow = slow.next; fast = fast.next; } //Skip the desired node slow.next = slow.next.next; return start.next; } 第17天 Unique Morse Code Words（804） 题号 804 Unique Morse Code Words\n题目描述：\n国际摩尔斯密码定义一种标准编码方式，将每个字母对应于一个由一系列点和短线组成的字符串， 比如: \u0026quot;a\u0026quot; 对应 \u0026quot;.-\u0026quot;, \u0026quot;b\u0026quot; 对应 \u0026quot;-...\u0026quot;, \u0026quot;c\u0026quot; 对应 \u0026quot;-.-.\u0026quot;, 等等。\n为了方便，所有26个英文字母对应摩尔斯密码表如下：\n[\u0026#34;.-\u0026#34;,\u0026#34;-...\u0026#34;,\u0026#34;-.-.\u0026#34;,\u0026#34;-..\u0026#34;,\u0026#34;.\u0026#34;,\u0026#34;..-.\u0026#34;,\u0026#34;--.\u0026#34;,\u0026#34;....\u0026#34;,\u0026#34;..\u0026#34;,\u0026#34;.---\u0026#34;,\u0026#34;-.-\u0026#34;,\u0026#34;.-..\u0026#34;,\u0026#34;--\u0026#34;,\u0026#34;-.\u0026#34;,\u0026#34;---\u0026#34;,\u0026#34;.--.\u0026#34;,\u0026#34;--.-\u0026#34;,\u0026#34;.-.\u0026#34;,\u0026#34;...\u0026#34;,\u0026#34;-\u0026#34;,\u0026#34;..-\u0026#34;,\u0026#34;...-\u0026#34;,\u0026#34;.--\u0026#34;,\u0026#34;-..-\u0026#34;,\u0026#34;-.--\u0026#34;,\u0026#34;--..\u0026#34;] 给定一个单词列表，每个单词可以写成每个字母对应摩尔斯密码的组合。例如，\u0026ldquo;cab\u0026rdquo; 可以写成 \u0026ldquo;-.-..\u0026ndash;\u0026hellip;\u0026quot;，(即 \u0026ldquo;-.-.\u0026rdquo; + \u0026ldquo;-\u0026hellip;\u0026rdquo; + \u0026ldquo;.-\u0026ldquo;字符串的结合)。我们将这样一个连接过程称作单词翻译。\n返回我们可以获得所有词不同单词翻译的数量。\n例如: 输入: words = [\u0026#34;gin\u0026#34;, \u0026#34;zen\u0026#34;, \u0026#34;gig\u0026#34;, \u0026#34;msg\u0026#34;] 输出: 2 解释: 各单词翻译如下: \u0026#34;gin\u0026#34; -\u0026gt; \u0026#34;--...-.\u0026#34; \u0026#34;zen\u0026#34; -\u0026gt; \u0026#34;--...-.\u0026#34; \u0026#34;gig\u0026#34; -\u0026gt; \u0026#34;--...--.\u0026#34; \u0026#34;msg\u0026#34; -\u0026gt; \u0026#34;--...--.\u0026#34; 共有 2 种不同翻译, \u0026#34;--...-.\u0026#34; 和 \u0026#34;--...--.\u0026#34;. 注意:\n单词列表words 的长度不会超过 100。 每个单词 words[i]的长度范围为 [1, 12]。 每个单词 words[i]只包含小写字母。 解题思路：通过给定的摩斯密码表翻译成字符串再利用Set去重\n心得体会：翻译过程注意数组下标从0开始，故原本a对应96则需要-97，同时该问题不存在大写字母，可减少一部分逻辑\n第18天 Intersection of Two Arrays（349） 题号 349 Intersection of Two Arrays\n题目描述：\n给定两个数组，编写一个函数来计算它们的交集。\n示例 1:\n输入: nums1 = [1,2,2,1], nums2 = [2,2] 输出: [2] 示例 2:\n输入: nums1 = [4,9,5], nums2 = [9,4,9,8,4] 输出: [9,4] 说明:\n输出结果中的每个元素一定是唯一的。 我们可以不考虑输出结果的顺序。 解题思路：使用集合对数组去重，并遍历另一个数组，遇到相同元素则添加进结果集，同时删除第一个数组集合中的元素，防止下次再次遇到相同的元素\n心得体会：最容易想到的方式是对两个数组都进行去重后比较，但通过使用第一个去重集合在第二个数组遍历过程中遇到相同元素时去除该元素的方式，减少对第二个数组的去重操作，提高性能\n第19天 Intersection of Two Arrays II（350） 题号 350 Intersection of Two Arrays II\n题目描述：\n给定两个数组，编写一个函数来计算它们的交集。\n示例 1:\n输入: nums1 = [1,2,2,1], nums2 = [2,2] 输出: [2,2] 示例 2:\n输入: nums1 = [4,9,5], nums2 = [9,4,9,8,4] 输出: [4,9] 说明：\n输出结果中每个元素出现的次数，应与元素在两个数组中出现的次数一致。 我们可以不考虑输出结果的顺序。 进阶:\n如果给定的数组已经排好序呢？你将如何优化你的算法？ 如果 nums1 的大小比 nums2 小很多，哪种方法更优？ 如果 nums2 的元素存储在磁盘上，磁盘内存是有限的，并且你不能一次加载所有的元素到内存中，你该怎么办？ 解题思路：首先对第一个数组进行词频统计，在对第二个数组遍历过程中遇到相同元素对词频减一并加入结果集，当词频为0则移除元素，由此实现每个元素出现的次数，应与元素在两个数组中出现的次数一致\n心得体会：Map作为词频统计的经典用法，但注意当词频为0时需要移除元素，否则词频计数便没有意义\n第20天 N-Repeated Element in Size 2N Array（961） 题号 961 N-Repeated Element in Size 2N Array\n题目描述：\n在大小为 2N 的数组 A 中有 N+1 个不同的元素，其中有一个元素重复了 N 次。\n返回重复了 N 次的那个元素。\n示例 1：\n输入：[1,2,3,3] 输出：3 示例 2：\n输入：[2,1,2,5,3,2] 输出：2 示例 3：\n输入：[5,1,5,2,5,3,5,4] 输出：5 提示：\n4 \u0026lt;= A.length \u0026lt;= 10000 0 \u0026lt;= A[i] \u0026lt; 10000 A.length 为偶数 解题思路：Map词频统计用法，一旦某个元素重复了N次则不再继续统计\n心得体会：词频统计的经典用法\n第21天 Single Number（136） 题号 136 Single Number\n题目描述：\n给定一个非空整数数组，除了某个元素只出现一次以外，其余每个元素均出现两次。找出那个只出现了一次的元素。\n说明：\n你的算法应该具有线性时间复杂度。 你可以不使用额外空间来实现吗？\n示例 1:\n输入: [2,2,1] 输出: 1 示例 2:\n输入: [4,1,2,1,2] 输出: 4 解题思路：Map词频统计用法，统计完成后寻找词频为1的即可\n心得体会：Map词频统计法比较容易想到，解题后在官方解答区看到两个更灵活的解法，分别通过数学计算和位运算实现，开阔了解题思路\nMap词频统计法\n数学运算法\n原理 $$ 2∗(a+b+c)−(a+a+b+b+c)=c $$ class Solution { public int singleNumber(int[] nums) { return (2 * sum(set(nums)) - sum(nums)); } private int[] set(int[] nums) { HashSet\u0026lt;Integer\u0026gt; set = new HashSet\u0026lt;\u0026gt;(); for (int num : nums) { set.add(num); } int[] res = new int[set.size()]; int i = 0; for (Integer integer : set) { res[i++] = integer; } return res; } private int sum(int[] nums) { int sum = 0; for (int num : nums) { sum += num; } return sum; } } 位运算法（已知性能最佳解法）\nclass Solution { public int singleNumber(int[] nums) { int result = 0; for (int i : nums) { // 一个数和0异或还是自己，一个数和自己异或是0 result ^= i; } return result; } } 第22天 Top K Frequent Elements（347） 题号 347 Top K Frequent Elements\n题目描述：\n给定一个非空的整数数组，返回其中出现频率前 k 高的元素。\n示例 1:\n输入: nums = [1,1,1,2,2,3], k = 2 输出: [1,2] 示例 2:\n输入: nums = [1], k = 1 输出: [1] 说明：\n你可以假设给定的 k 总是合理的，且 1 ≤ k ≤ 数组中不相同的元素的个数。 你的算法的时间复杂度必须优于 O(n log n) , n 是数组的大小。 解题思路：使用优先队列，维护出现频率前 k 高的元素\n心得体会：Java内置的优先队列为小根堆，同时支持定义比较器，结合优先队列特性即可解答\n第23天 Top K Frequent Words（692） 题号 692 Top K Frequent Words\n题目描述：\n给一非空的单词列表，返回前 k 个出现次数最多的单词。\n返回的答案应该按单词出现频率由高到低排序。如果不同的单词有相同出现频率，按字母顺序排序。\n示例 1：\n输入: [\u0026#34;i\u0026#34;, \u0026#34;love\u0026#34;, \u0026#34;leetcode\u0026#34;, \u0026#34;i\u0026#34;, \u0026#34;love\u0026#34;, \u0026#34;coding\u0026#34;], k = 2 输出: [\u0026#34;i\u0026#34;, \u0026#34;love\u0026#34;] 解析: \u0026#34;i\u0026#34; 和 \u0026#34;love\u0026#34; 为出现次数最多的两个单词，均为2次。 注意，按字母顺序 \u0026#34;i\u0026#34; 在 \u0026#34;love\u0026#34; 之前。 示例 2：\n输入: [\u0026#34;the\u0026#34;, \u0026#34;day\u0026#34;, \u0026#34;is\u0026#34;, \u0026#34;sunny\u0026#34;, \u0026#34;the\u0026#34;, \u0026#34;the\u0026#34;, \u0026#34;the\u0026#34;, \u0026#34;sunny\u0026#34;, \u0026#34;is\u0026#34;, \u0026#34;is\u0026#34;], k = 4 输出: [\u0026#34;the\u0026#34;, \u0026#34;is\u0026#34;, \u0026#34;sunny\u0026#34;, \u0026#34;day\u0026#34;] 解析: \u0026#34;the\u0026#34;, \u0026#34;is\u0026#34;, \u0026#34;sunny\u0026#34; 和 \u0026#34;day\u0026#34; 是出现次数最多的四个单词， 出现次数依次为 4, 3, 2 和 1 次。 注意：\n假定 k 总为有效值， 1 ≤ k ≤ 集合元素数。 输入的单词均由小写字母组成。 扩展练习：\n尝试以 O(n log k) 时间复杂度和 O(n) 空间复杂度解决。 解题思路：使用优先队列，维护前 k 个出现次数最多的单词，同时维护字典顺序\n心得体会：与347题类似，但需要处理单词有相同出现频率，按字母顺序排序的情况，按照Java默认字符串排序方式和频率大小定义比较器即可\n第24天 Range Sum Query - Immutable（303） 题号 303 Range Sum Query - Immutable\n题目描述：\n给定一个整数数组 nums，求出数组从索引 i 到 j (i ≤ j) 范围内元素的总和，包含 i, j 两点。\n示例：\n给定 nums = [-2, 0, 3, -5, 2, -1]，求和函数为 sumRange() sumRange(0, 2) -\u0026gt; 1 sumRange(2, 5) -\u0026gt; -1 sumRange(0, 5) -\u0026gt; -3 说明:\n你可以假设数组不可变。 会多次调用 sumRange 方法。 解题思路：典型的线段树应用场景，但该题可以使用数组维护区间总和，对于该题来说时间复杂度更低\n心得体会：线段树解法从读题即可想到，而后由于说明为Immutable，故通过维护数组实现时间复杂度更低\n数组实现\n线段树实现\nclass NumArray { private SegmentTree\u0026lt;Integer\u0026gt; tree; public NumArray(int[] nums) { Integer[] list = new Integer[nums.length]; for (int i = 0; i \u0026lt; nums.length; i++) { list[i] = nums[i]; } tree = new SegmentTree\u0026lt;\u0026gt;(list,(a, b) -\u0026gt; a+b); } public int sumRange(int i, int j) { return tree.query(i,j); } class SegmentTree\u0026lt;E\u0026gt; { private E[] data; private E[] tree; private Merger\u0026lt;E\u0026gt; merger; public SegmentTree(E[] arr, Merger\u0026lt;E\u0026gt; merger) { this.merger = merger; this.data = (E[]) new Object[arr.length]; // 拷贝副本 for (int i = 0; i \u0026lt; arr.length; i++) { data[i] = arr[i]; } // 创建线段树存储空间，4倍空间可以存储最坏情况下的线段树 this.tree = (E[]) new Object[4 * arr.length]; // 创建线段树 buildSegmentTree(0, 0, data.length - 1); } /** * 在treeIndex的位置创建表示区间[l...r]的线段树 * * @param treeIndex * @param left * @param right */ private void buildSegmentTree(int treeIndex, int left, int right) { if (data.length==0){ return; } // 递归终止条件为叶子节点 if (left == right) { tree[treeIndex] = data[left]; return; } // 左子树索引 int leftIndex = leftChild(treeIndex); // 右子树索引 int rightIndex = rightChild(treeIndex); // 等价于 (left + right)/2 int mid = left + (right - left) / 2; buildSegmentTree(leftIndex, left, mid); buildSegmentTree(rightIndex, mid + 1, right); // 线段值由业务情况决定 tree[treeIndex] = merger.merge(tree[leftIndex], tree[rightIndex]); } public E query(int queryL, int queryR) { if (queryL \u0026lt; 0 || queryL \u0026gt;= data.length || queryR \u0026lt; 0 || queryR \u0026gt;= data.length || queryL \u0026gt; queryR) { throw new IllegalArgumentException(\u0026#34;Index is illegal.\u0026#34;); } return query(0, 0, data.length - 1, queryL, queryR); } private E query(int treeIndex, int left, int right, int queryL, int queryR) { // 查询区间与线段树区间相等则直接返回 if (left == queryL \u0026amp;\u0026amp; right == queryR) { return tree[treeIndex]; } int leftIndex = leftChild(treeIndex); int rightIndex = rightChild(treeIndex); int mid = left + (right - left) / 2; // 若全在左子树 if (queryR \u0026lt;= mid) { return query(leftIndex, left, mid, queryL, queryR); } else if (queryL \u0026gt;= mid + 1) { // 全在右子树 return query(rightIndex, mid + 1, right, queryL, queryR); } // 分散在左右子树 E leftResult = query(leftIndex, left, mid, queryL, mid); E rightResult = query(rightIndex, mid + 1, right, mid + 1, queryR); return merger.merge(leftResult, rightResult); } public int getSize() { return data.length; } public E get(int index) { isLegal(index); return data[index]; } private void isLegal(int index) { if (index \u0026lt; 0 || index \u0026gt; data.length - 1) { throw new IllegalArgumentException(\u0026#34;index out of bound\u0026#34;); } } /** * 返回平衡二叉树的数组表示中，一个索引所表示的元素的左孩子节点的索引 * * @param index * @return */ private int leftChild(int index) { return index * 2 + 1; } /** * 返回平衡二叉树的数组表示中，一个索引所表示的元素的右孩子节点的索引 * * @param index * @return */ private int rightChild(int index) { return index * 2 + 2; }} @FunctionalInterface public interface Merger\u0026lt;E\u0026gt; { /** * 合并a、b两个对象 * * @param a * @param b * @return */ E merge(E a, E b); } } 性能差异\n解决方式 耗时 内存 数组 52 ms 41.2 MB 线段树 59 ms 42.2 MB 第25天 Range Sum Query - Mutable（307） 题号 307 Range Sum Query - Mutable\n题目描述：\n给定一个整数数组 nums，求出数组从索引 i 到 j (i ≤ j) 范围内元素的总和，包含 i, j 两点。\nupdate(i, val) 函数可以通过将下标为 i 的数值更新为 val，从而对数列进行修改。\n示例:\nGiven nums = [1, 3, 5] sumRange(0, 2) -\u0026gt; 9 update(1, 2) sumRange(0, 2) -\u0026gt; 8 说明:\n数组仅可以在 update 函数下进行修改。 你可以假设 update 函数与 sumRange 函数的调用次数是均匀分布的。 解题思路：使用线段树或者数组动态维护区间总和\n心得体会：与303题类似，也是线段树的典型应用场景，但此时数组为Mutable，虽然也可以通过数组维护区间总和，但更新时需要同时维护多个数值，此时的效率不如线段树\n线段树实现\nclass NumArray { private SegmentTree\u0026lt;Integer\u0026gt; tree; public NumArray(int[] nums) { Integer[] arr = new Integer[nums.length]; for (int i = 0; i \u0026lt; nums.length; i++) { arr[i] = nums[i]; } tree = new SegmentTree\u0026lt;\u0026gt;(arr,(a, b) -\u0026gt; a+b); } public void update(int i, int val) { tree.set(i,val); } public int sumRange(int i, int j) { return tree.query(i,j); } class SegmentTree\u0026lt;E\u0026gt; { private E[] data; private E[] tree; private Merger\u0026lt;E\u0026gt; merger; public SegmentTree(E[] arr, Merger\u0026lt;E\u0026gt; merger) { this.merger = merger; this.data = (E[]) new Object[arr.length]; // 拷贝副本 for (int i = 0; i \u0026lt; arr.length; i++) { data[i] = arr[i]; } // 创建线段树存储空间，4倍空间可以存储最坏情况下的线段树 this.tree = (E[]) new Object[4 * arr.length]; // 创建线段树 buildSegmentTree(0, 0, data.length - 1); } /** * 在treeIndex的位置创建表示区间[l...r]的线段树 * * @param treeIndex * @param left * @param right */ private void buildSegmentTree(int treeIndex, int left, int right) { if (data.length==0){ return; } // 递归终止条件为叶子节点 if (left == right) { tree[treeIndex] = data[left]; return; } // 左子树索引 int leftIndex = leftChild(treeIndex); // 右子树索引 int rightIndex = rightChild(treeIndex); // 等价于 (left + right)/2 int mid = left + (right - left) / 2; buildSegmentTree(leftIndex, left, mid); buildSegmentTree(rightIndex, mid + 1, right); // 线段值由业务情况决定 tree[treeIndex] = merger.merge(tree[leftIndex], tree[rightIndex]); } public E query(int queryL, int queryR) { if (queryL \u0026lt; 0 || queryL \u0026gt;= data.length || queryR \u0026lt; 0 || queryR \u0026gt;= data.length || queryL \u0026gt; queryR) { throw new IllegalArgumentException(\u0026#34;Index is illegal.\u0026#34;); } return query(0, 0, data.length - 1, queryL, queryR); } /** * 在以treeIndex为根的线段树中[l...r]的范围里，搜索区间[queryL...queryR]的值 * * @param treeIndex * @param left * @param right * @param queryL * @param queryR * @return */ private E query(int treeIndex, int left, int right, int queryL, int queryR) { // 查询区间与线段树区间相等则直接返回 if (left == queryL \u0026amp;\u0026amp; right == queryR) { return tree[treeIndex]; } int leftIndex = leftChild(treeIndex); int rightIndex = rightChild(treeIndex); int mid = left + (right - left) / 2; // 若全在左子树 if (queryR \u0026lt;= mid) { return query(leftIndex, left, mid, queryL, queryR); } else if (queryL \u0026gt;= mid + 1) { // 全在右子树 return query(rightIndex, mid + 1, right, queryL, queryR); } // 分散在左右子树 E leftResult = query(leftIndex, left, mid, queryL, mid); E rightResult = query(rightIndex, mid + 1, right, mid + 1, queryR); return merger.merge(leftResult, rightResult); } /** * 将index位置的值，更新为e * * @param index * @param e */ public void set(int index,E e){ if (data.length==0){ return; } isLegal(index); data[index] = e; set(0,0,data.length-1,index,e); } /** * 在以treeIndex为根的线段树中更新index的值为e * * @param treeIndex * @param left * @param right * @param index * @param e */ private void set(int treeIndex,int left,int right,int index,E e){ if (left == right){ // 更新叶子结点 tree[treeIndex] = e; return; } int mid = left+(right-left)/2; int leftTreeIndex = leftChild(treeIndex); int rightTreeIndex = rightChild(treeIndex); if (index\u0026lt;=mid){ set(leftTreeIndex,left,mid,index,e); }else { set(rightTreeIndex,mid+1,right,index,e); } // 更新线段树节点值 tree[treeIndex] = merger.merge(tree[leftTreeIndex],tree[rightTreeIndex]); } public int getSize() { return data.length; } public E get(int index) { isLegal(index); return data[index]; } private void isLegal(int index) { if (index \u0026lt; 0 || index \u0026gt; data.length - 1) { throw new IllegalArgumentException(\u0026#34;index out of bound\u0026#34;); } } /** * 返回平衡二叉树的数组表示中，一个索引所表示的元素的左孩子节点的索引 * * @param index * @return */ private int leftChild(int index) { return index * 2 + 1; } /** * 返回平衡二叉树的数组表示中，一个索引所表示的元素的右孩子节点的索引 * * @param index * @return */ private int rightChild(int index) { return index * 2 + 2; }} @FunctionalInterface public interface Merger\u0026lt;E\u0026gt; { /** * 合并a、b两个对象 * * @param a * @param b * @return */ E merge(E a, E b); } } /** * Your NumArray object will be instantiated and called as such: * NumArray obj = new NumArray(nums); * obj.update(i,val); * int param_2 = obj.sumRange(i,j); */ 数组实现\nclass NumArray { int[] nums; int[] origin; public NumArray(int[] nums) { origin = new int[nums.length]; if (nums.length\u0026gt;0){ origin[0] = nums[0]; } // 第i个位置存储索引[0..i]的和，最后一个位置则为整个数组的和 for (int i = 1; i \u0026lt; nums.length; i++) { origin[i] = nums[i]; nums[i] += nums[i - 1]; } this.nums = nums; } public void update(int i, int val) { // 更新位置之后的值均要变化 int oldVal = origin[i]; int diff = val - oldVal; for (int j = i ; j \u0026lt; nums.length; j++) { nums[j] += diff; } origin[i] = val; } public int sumRange(int i, int j) { if (i == 0) { return nums[j]; } // 计算区间和 return nums[j] - nums[i - 1]; } } 性能差异\n解决方式 耗时 内存 线段树 60 ms 47.3 MB 数组 199 ms 49.3 MB 第26天 Implement Trie (Prefix Tree)（208） 题号 208 Implement Trie (Prefix Tree)\n题目描述：\n实现一个 Trie (前缀树)，包含 insert, search, 和 startsWith 这三个操作。\n示例:\nTrie trie = new Trie(); trie.insert(\u0026#34;apple\u0026#34;); trie.search(\u0026#34;apple\u0026#34;); // 返回 true trie.search(\u0026#34;app\u0026#34;); // 返回 false trie.startsWith(\u0026#34;app\u0026#34;); // 返回 true trie.insert(\u0026#34;app\u0026#34;); trie.search(\u0026#34;app\u0026#34;); // 返回 true 说明:\n你可以假设所有的输入都是由小写字母 a-z 构成的。 保证所有输入均为非空字符串。 解题思路：字典树（前缀树）实现\n心得体会：字典树（前缀树）实现\n字典树实现\nclass Trie { /** * 多叉树结构 */ private class Node { private boolean isWord; private HashMap\u0026lt;Character, Node\u0026gt; next; public Node(boolean isWord) { this.isWord = isWord; this.next = new HashMap\u0026lt;\u0026gt;(); } public Node() { this(false); } } private Node root; private int size; /** Initialize your data structure here. */ public Trie() { // 带虚拟头结点 this.root = new Node(); this.size = 0; } /** Inserts a word into the trie. */ public void insert(String word) { Node cur = root; for (int i = 0; i \u0026lt; word.length(); i++) { char c = word.charAt(i); // 不存在当前字母节点 if (cur.next.get(c) == null) { cur.next.put(c, new Node()); } // 已经存在当前字母节点，继续向下寻找 cur = cur.next.get(c); } // 字母添加完毕，查看当前单词末尾是否已经被标记，若被标记则说明重复添加 if (!cur.isWord) { // 标记为单词末尾 cur.isWord = true; size++; } } /** Returns if the word is in the trie. */ public boolean search(String word) { Node cur = root; for (int i = 0; i \u0026lt; word.length(); i++) { char c = word.charAt(i); if (cur.next.get(c) == null) { return false; } cur = cur.next.get(c); } return cur.isWord; } /** Returns if there is any word in the trie that starts with the given prefix. */ public boolean startsWith(String prefix) { Node cur = root; for (int i = 0; i \u0026lt; prefix.length(); i++) { char c = prefix.charAt(i); if (cur.next.get(c)==null){ return false; } cur = cur.next.get(c); } return true; } } 第27天 Add and Search Word - Data structure design（211） 题号 211 Add and Search Word - Data structure design\n题目描述：\n设计一个支持以下两种操作的数据结构：\nvoid addWord(word) bool search(word) search(word) 可以搜索文字或正则表达式字符串，字符串只包含字母 . 或 a-z 。 . 可以表示任何一个字母。\n示例:\naddWord(\u0026#34;bad\u0026#34;) addWord(\u0026#34;dad\u0026#34;) addWord(\u0026#34;mad\u0026#34;) search(\u0026#34;pad\u0026#34;) -\u0026gt; false search(\u0026#34;bad\u0026#34;) -\u0026gt; true search(\u0026#34;.ad\u0026#34;) -\u0026gt; true search(\u0026#34;b..\u0026#34;) -\u0026gt; true 说明:\n你可以假设所有单词都是由小写字母 a-z 组成的。\n解题思路：字典树实现\n心得体会：典型字典树（前缀树）应用场景\n字典树实现\nclass WordDictionary { /** * 多叉树结构 */ private class Node { private boolean isWord; private HashMap\u0026lt;Character, Node\u0026gt; next; public Node(boolean isWord) { this.isWord = isWord; this.next = new HashMap\u0026lt;\u0026gt;(); } public Node() { this(false); } } private Node root; /** Initialize your data structure here. */ public WordDictionary() { // 带虚拟头结点 this.root = new Node(); } /** Adds a word into the data structure. */ public void addWord(String word) { Node cur = root; for (int i = 0; i \u0026lt; word.length(); i++) { char c = word.charAt(i); // 不存在当前字母节点 if (cur.next.get(c) == null) { cur.next.put(c, new Node()); } // 已经存在当前字母节点，继续向下寻找 cur = cur.next.get(c); } // 字母添加完毕，查看当前单词末尾是否已经被标记，若被标记则说明重复添加 if (!cur.isWord) { // 标记为单词末尾 cur.isWord = true; } } /** Returns if the word is in the data structure. A word could contain the dot character \u0026#39;.\u0026#39; to represent any one letter. */ public boolean search(String word) { return match(word,root,0); } private boolean match(String word,Node node,int index){ // 终止条件 if (index == word.length()){ return node.isWord; } char c = word.charAt(index); // 判断通配符 if (c != \u0026#39;.\u0026#39;){ if (node.next.get(c)==null){ return false; } return match(word,node.next.get(c),index+1); } // 遍历当前节点所有的子节点 for (Character w : node.next.keySet()) { // 如果有一个节点返回true则为true if (match(word,node.next.get(w),index+1)){ return true; } } return false; } } /** * Your WordDictionary object will be instantiated and called as such: * WordDictionary obj = new WordDictionary(); * obj.addWord(word); * boolean param_2 = obj.search(word); */ 第28天 Map Sum Pairs（677） 题号 677 Map Sum Pairs\n题目描述：\n实现一个 MapSum 类里的两个方法，insert 和 sum。\n对于方法 insert，你将得到一对（字符串，整数）的键值对。字符串表示键，整数表示值。如果键已经存在，那么原来的键值对将被替代成新的键值对。\n对于方法 sum，你将得到一个表示前缀的字符串，你需要返回所有以该前缀开头的键的值的总和。\n示例 1:\n输入: insert(\u0026#34;apple\u0026#34;, 3), 输出: Null 输入: sum(\u0026#34;ap\u0026#34;), 输出: 3 输入: insert(\u0026#34;app\u0026#34;, 2), 输出: Null 输入: sum(\u0026#34;ap\u0026#34;), 输出: 5 解题思路：字典树\n心得体会：使用字典树记录字母节点，到节点尾部维护权重值，更新时同时更新权重值即可\n字典树实现\n第29天 Validate Binary Search Tree（98） 题号 98 Validate Binary Search Tree\n题目描述：\n给定一个二叉树，判断其是否是一个有效的二叉搜索树。\n假设一个二叉搜索树具有如下特征：\n节点的左子树只包含小于当前节点的数。 节点的右子树只包含大于当前节点的数。 所有左子树和右子树自身必须也是二叉搜索树。 示例 1:\n输入: 2 / \\ 1 3 输出: true 示例 2:\n输入: 5 / \\ 1 4 / \\ 3 6 输出: false 解释: 输入为: [5,1,4,null,null,3,6]。 根节点的值为 5 ，但是其右子节点值为 4 。 解题思路：二叉树中序遍历\n心得体会：利用二分搜索树中序遍历有序性的特点，对中序遍历集合进行检查，若发现无序则返回。该解法可进一步优化为在执行中序遍历过程中保留上一次递归结果进行判断，若发现无序则无需遍历所有节点。\n第30天 Delete Node in a BST（450） 题号 450 Delete Node in a BST\n题目描述：\n给定一个二叉搜索树的根节点 root 和一个值 key，删除二叉搜索树中的 key 对应的节点，并保证二叉搜索树的性质不变。返回二叉搜索树（有可能被更新）的根节点的引用。\n一般来说，删除节点可分为两个步骤：\n首先找到需要删除的节点； 如果找到了，删除它。 说明： 要求算法时间复杂度为 O(h)，h 为树的高度。\n示例:\nroot = [5,3,6,2,4,null,7] key = 3 5 / \\ 3 6 / \\ \\ 2 4 7 给定需要删除的节点值是 3，所以我们首先找到 3 这个节点，然后删除它。 一个正确的答案是 [5,4,6,2,null,null,7], 如下图所示。 5 / \\ 4 6 / \\ 2 7 另一个正确答案是 [5,2,6,null,4,null,7]。 5 / \\ 2 6 \\ \\ 4 7 解题思路：二分搜索树删除节点\n心得体会：对于二分搜索树删除节点，此处选择该节点右子树中最小的节点替代被删除节点以保证二分搜索树的性质，相反，也可以通过使用该节点的左子树中最大的节点替代。\n第31天 Binary Tree Level Order Traversal（102） 题号 102 Binary Tree Level Order Traversal\n题目描述：\n给定一个二叉树，返回其按层次遍历的节点值。 （即逐层地，从左到右访问所有节点）。\n例如: 给定二叉树: [3,9,20,null,null,15,7],\n3 / \\ 9 20 / \\ 15 7 返回其层次遍历结果：\n[ [3], [9,20], [15,7] ] 解题思路：二分搜索树层次遍历\n心得体会：与[107题](#第12天 Binary Tree Level Order Traversal II（107)类似\n第32天 Binary Search Tree Iterator（173） 题号 173 Binary Search Tree Iterator\n题目描述：\n实现一个二叉搜索树迭代器。你将使用二叉搜索树的根节点初始化迭代器。\n调用 next() 将返回二叉搜索树中的下一个最小的数。\n示例：\nBSTIterator iterator = new BSTIterator(root); iterator.next(); // 返回 3 iterator.next(); // 返回 7 iterator.hasNext(); // 返回 true iterator.next(); // 返回 9 iterator.hasNext(); // 返回 true iterator.next(); // 返回 15 iterator.hasNext(); // 返回 true iterator.next(); // 返回 20 iterator.hasNext(); // 返回 false 提示：\nnext() 和 hasNext() 操作的时间复杂度是 O(1)，并使用 O(h) 内存，其中 h 是树的高度。 你可以假设 next() 调用总是有效的，也就是说，当调用 next() 时，BST 中至少存在一个下一个最小的数。 解题思路：使用栈或者队列实现\n心得体会：比较简单的方式是使用队列存储中序遍历集合，依次出队则保证为树中最小的元素，也评论区看到栈的解法，时间复杂度更低，但实现相对复杂\n队列解法\n栈解法\n/** * Definition for binary tree * public class TreeNode { * int val; * TreeNode left; * TreeNode right; * TreeNode(int x) { val = x; } * } */ public class BSTIterator { private Stack\u0026lt;TreeNode\u0026gt; stack; public BSTIterator(TreeNode root) { stack = new Stack\u0026lt;\u0026gt;(); TreeNode cur = root; while(cur != null){ stack.push(cur); if(cur.left != null) cur = cur.left; else break; } } /** @return whether we have a next smallest number */ public boolean hasNext() { return !stack.isEmpty(); } /** @return the next smallest number */ public int next() { TreeNode node = stack.pop(); TreeNode cur = node; // traversal right branch if(cur.right != null){ cur = cur.right; while(cur != null){ stack.push(cur); if(cur.left != null) cur = cur.left; else break; } } return node.val; } } /** * Your BSTIterator will be called like this: * BSTIterator i = new BSTIterator(root); * while (i.hasNext()) v[f()] = i.next(); */ ","date":"2019-01-06T19:43:00+08:00","permalink":"/archives/201/","title":"LeetCode刷题记录"},{"content":"mysql驱动问题 问题描述 问题发生时间（2018-12-27）\n环境版本：\nSpringBoot 2.1.1 Mybatis 1.3.2 原始配置\nspring: datasource: driver-class-name: com.mysql.jdbc.Driver username: hikaricp password: hikaricp url: jdbc:mysql://localhost:3306/db_test?characterEncoding=utf-8\u0026amp;useSSL=false 报错信息\nRegistered driver with driverClassName=com.mysql.jdbc.Driver was not found, 报错提示\nLoading class `com.mysql.jdbc.Driver\u0026#39;. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver\u0026#39;. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 解决办法 从提示中可看出，驱动名称有变，更正驱动名称即可\n更正后配置\nspring: datasource: driver-class-name: com.mysql.cj.jdbc.Driver username: hikaricp password: hikaricp url: jdbc:mysql://localhost:3306/db_test?characterEncoding=utf-8\u0026amp;useSSL=false 同时若使用IDEA自带提示也可看出问题，有两种驱动都存在的问题，希望后续修复\n另一种解决方案，从SpringBoot2.0之后连接池使用hikari，可根据url自动识别驱动，即如下配置也可成功运行\nspring: datasource: username: hikaricp password: hikaricp url: jdbc:mysql://localhost:3306/db_test?characterEncoding=utf-8\u0026amp;useSSL=false 注解使用简化 前景提要 进入SpringBoot时代后“注解驱动开发”就已经成为趋势，mybatis官方也提供注解替代xml配置的方案\n问题描述 问题发现时间（2018-12-27）\n环境版本：\nSpringBoot 2.1.1 Mybatis 1.3.2 在早期版本的mybatis-starter还需要进行实体映射xml文件配置，这次在使用1.3.2版本时再次查了下官方源码，发现已经无需使用映射文件，其底层会自动通过反射识别对象属性类型，并且去除了不少早期版本使用的注解（当前仍可以使用，但不使用功能正常）\n无需手动创建SqlSession或DataSource 官方文档说明：\nAs you may already know, to use MyBatis with Spring you need at least an SqlSessionFactory and at least one mapper interface.\nMyBatis-Spring-Boot-Starter will:\nAutodetect an existing DataSource. Will create and register an instance of a SqlSessionFactory passing that DataSource as an input using the SqlSessionFactoryBean. Will create and register an instance of a SqlSessionTemplate got out of the SqlSessionFactory. Autoscan your mappers, link them to the SqlSessionTemplate and register them to Spring context so they can be injected into your beans. 支持版本为Spring 4.3+\nYou just need to create a normal Spring boot application and let the mapper be injected like follows(available on Spring 4.3+):\n@MapperScan -\u0026gt; @Mapper 早期版本需要在启动类上加上@MapperScan(basePackages = \u0026quot;xxx\u0026quot;)注解扫描mapper类，新版本可直接在mapper包上使用@Mapper注解即可\nThe MyBatis-Spring-Boot-Starter will search, by default, for mappers marked with the @Mapper annotation.\n除非需要自定义注解或者不使用@Mapper才需要加上@MapperScan\nYou may want to specify a custom annotation or a marker interface for scanning. If so, you must use the @MapperScan annotation.\n@Results与Mapper.xml 早期版本需要手动指定类型映射，并在resources目录下进行xml配置，新版本自动通过反射识别属性类型，无需进行xml配置\n","date":"2018-12-27T12:37:00+08:00","permalink":"/archives/198/","title":"mybatis中遇到的坑"},{"content":"Hadoop伪分布式安装 基础环境 Hadoop：版本2.7.3，下载地址 HBase：版本1.2.4，下载地址 JDK8 jdk 安装命令\n$ yum -y install java-devel java 查看当前系统jre和jdk路径，输入update-alternatives --config java和update-alternatives --config javac\n$ update-alternatives --config java 共有 1 个提供“java”的程序。 选项 命令 ----------------------------------------------- *+ 1 java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/bin/java) $ update-alternatives --config javac 共有 1 个提供“javac”的程序。 选项 命令 ----------------------------------------------- *+ 1 java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/bin/javac) 查看环境\n$ ll 总用量 311132 -rw-------. 1 root root 1241 8月 11 10:53 anaconda-ks.cfg -rw-r--r--. 1 root root 214092195 8月 26 2016 hadoop-2.7.3.tar.gz -rw-r--r--. 1 root root 104497899 10月 26 2016 hbase-1.2.4-bin.tar.gz $ java -version openjdk version \u0026#34;1.8.0_191\u0026#34; OpenJDK Runtime Environment (build 1.8.0_191-b12) OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) $ javac -version javac 1.8.0_191 免密登录 新建hadoop用户，忽略密码强度提示\n$ adduser hadoop $ passwd hadoop 为方便远程连接和学习，同时授予root权限，输入whereis sudoers查看权限控制文件位置\n$ whereis sudoers sudoers: /etc/sudoers /etc/sudoers.d /usr/libexec/sudoers.so /usr/share/man/man5/sudoers.5.gz vi /etc/sudoers修改内容为\nroot ALL=(ALL) ALL hadoop ALL=(ALL) ALL 切换至hadoop用户，使用 ssh-keygen -t rsa生成私钥信息，位置在/home/hadoop/，并且忽略所有提示输入的信息\n查看生成的文件，该SSH文件夹为隐藏文件夹，故用ls -a查看，可见自动生成了.ssh目录和id_rsa,id_rsa.pub两个文件\n$ ls /home/hadoop/.ssh/ id_rsa id_rsa.pub 复制公钥到.ssh目录\n$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys 免密测试\n$ ssh localhost The authenticity of host \u0026#39;localhost (::1)\u0026#39; can\u0026#39;t be established. ECDSA key fingerprint is SHA256:3UEr3lx2jznmN3FL3SflViR05IZe6AWweb3TeSsfX0M. ECDSA key fingerprint is MD5:0f:31:c1:cc:54:69:7f:d8:8d:7c:8c:22:95:2e:03:4e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added \u0026#39;localhost\u0026#39; (ECDSA) to the list of known hosts. Last login: Sat Dec 15 11:14:27 2018 $ exit 登出 Connection to localhost closed. 安装Hadoop 解压文件\n$ tar -zxvf hadoop-2.7.3.tar.gz 修改配置文件\n$ vi /hadoop-2.7.3/etc/hadoop/hadoop-env.sh 配置Java路径\n# The java implementation to use. export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64 修改HDFS配置\n$ vi /hadoop-2.7.3/etc/hadoop/hdfs-site.xml 配置内容\n\u0026lt;configuration\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;dfs.replication\u0026lt;/name\u0026gt; \u0026lt;!-- 配置副本数量--\u0026gt; \u0026lt;value\u0026gt;1\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;dfs.namenode.name.dir\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;file:/home/hadoop/hadoop_data/dfs/name\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;dfs.namenode.data.dir\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;file:/home/hadoop/hadoop_data/dfs/data\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;/configuration\u0026gt; 手动创建相关目录\n$ mkdir -p /home/hadoop/hadoop_data/dfs/name $ mkdir -p /home/hadoop/hadoop_data/dfs/data 配置core-site.xml，内容为\n\u0026lt;configuration\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;fs.defaultFS\u0026lt;/name\u0026gt; \u0026lt;!-- 配置主机IP--\u0026gt; \u0026lt;value\u0026gt;hdfs://0.0.0.0:8020\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;!-- 配置临时文件位置--\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;hadoop.tmp.dir\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;file:/home/hadoop/hadoop_data\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;/configuration\u0026gt; 放行端口并重载\n$ firewall-cmd --zone=public --add-port=8020/tcp --add-port=50070/tcp --permanent $ firewall-cmd --reload 启动Hadoop 初始化 进入/hadoop-2.7.3/bin目录，使用如下语句初始化Hadoop，该操作仅执行一次即可，以后使用无需重复执行，否则执行后会删除当前hadoop上的所有文件\n$ hadoop namenode -format 滚动均为info级别的日志并且status 0表示已初始化成功\n... 18/12/15 17:07:32 INFO namenode.FSImage: Allocated new BlockPoolId: BP-99731851-127.0.0.1-1544864852133 18/12/15 17:07:32 INFO common.Storage: Storage directory /home/hadoop/hadoop_data/dfs/name has been successfully formatted. 18/12/15 17:07:32 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/hadoop_data/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression 18/12/15 17:07:32 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/hadoop_data/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 353 bytes saved in 0 seconds. 18/12/15 17:07:32 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid \u0026gt;= 0 18/12/15 17:07:32 INFO util.ExitUtil: Exiting with status 0 18/12/15 17:07:32 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1 ************************************************************/ 启动HDFS 进入/hadoop-2.7.3/sbin目录输入./start-dfs.sh启动HDFS服务\n$ ./start-dfs.sh Starting namenodes on [0.0.0.0] The authenticity of host \u0026#39;0.0.0.0 (0.0.0.0)\u0026#39; can\u0026#39;t be established. ECDSA key fingerprint is SHA256:3UEr3lx2jznmN3FL3SflViR05IZe6AWweb3TeSsfX0M. ECDSA key fingerprint is MD5:0f:31:c1:cc:54:69:7f:d8:8d:7c:8c:22:95:2e:03:4e. Are you sure you want to continue connecting (yes/no)? yes 0.0.0.0: Warning: Permanently added \u0026#39;0.0.0.0\u0026#39; (ECDSA) to the list of known hosts. 0.0.0.0: starting namenode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-namenode-localhost.localdomain.out localhost: starting datanode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-datanode-localhost.localdomain.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-secondarynamenode-localhost.localdomain.out 进程验证，若出现如下四个进程表示HDFS服务启动成功\n$ jps 28930 Jps 28821 SecondaryNameNode 28663 DataNode 28541 NameNode 配置hdfs命令 使用vi ~/.bash_profile进入配置文件，按如下配置\nexport HADOOP_HOME=/home/hadoop/hadoop-2.7.3 export PATH=$HADOOP_HOME/bin:$PATH 使配置文件生效source ~/.bash_profile 输入hdfs回车测试命令是否生效，若出现如下画面则表示命令已经生效\n$ hdfs Usage: hdfs [--config confdir] [--loglevel loglevel] COMMAND where COMMAND is one of: dfs run a filesystem command on the file systems supported in Hadoop. classpath prints the classpath namenode -format format the DFS filesystem secondarynamenode run the DFS secondary namenode namenode run the DFS namenode journalnode run the DFS journalnode zkfc run the ZK Failover Co ... 命令测试 $ hadoop fs -mkdir /test $ hadoop fs -ls / Found 1 items drwxr-xr-x - hadoop supergroup 0 2018-12-15 17:14 /test HBase伪分布式搭建 HBase安装 解压安装包，目录如下\n$ ll 总用量 320 drwxr-xr-x. 4 hadoop hadoop 4096 1月 29 2016 bin -rw-r--r--. 1 hadoop hadoop 122439 10月 26 2016 CHANGES.txt drwxr-xr-x. 2 hadoop hadoop 178 1月 29 2016 conf drwxr-xr-x. 12 hadoop hadoop 4096 10月 26 2016 docs drwxr-xr-x. 7 hadoop hadoop 80 10月 26 2016 hbase-webapps -rw-rw-r--. 1 hadoop hadoop 261 10月 26 2016 LEGAL drwxrwxr-x. 3 hadoop hadoop 8192 12月 15 11:24 lib -rw-rw-r--. 1 hadoop hadoop 130696 10月 26 2016 LICENSE.txt -rw-rw-r--. 1 hadoop hadoop 42025 10月 26 2016 NOTICE.txt -rw-r--r--. 1 hadoop hadoop 1477 12月 27 2015 README.txt 拷贝Hadoop配置文件到/conf目录\n$ cp /home/hadoop/hadoop-2.7.3/etc/hadoop/hdfs-site.xml . $ cp /home/hadoop/hadoop-2.7.3/etc/hadoop/core-site.xml . 修改hbase-env.sh文件，增加JDK路径\n# The java implementation to use. Java 1.7+ required. export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64 由于使用的是jdk8，主要注释内存配置\n# Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+ #export HBASE_MASTER_OPTS=\u0026#34;$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m\u0026#34; #export HBASE_REGIONSERVER_OPTS=\u0026#34;$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m\u0026#34; 另外打开文件最后zookeeper配置，设置为使用内置的zookeeper\n# Tell HBase whether it should manage it\u0026#39;s own instance of Zookeeper or not. export HBASE_MANAGES_ZK=true 修改hbase-site.xml文件，内容如下\n\u0026lt;configuration\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;hbase.rootdir\u0026lt;/name\u0026gt; \u0026lt;!-- 配置文件根目录--\u0026gt; \u0026lt;value\u0026gt;hdfs://localhost:8020/hbase\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;hbase.zookeeper.property.dataDir\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;/home/hadoop/hadoop_data/zookeeper\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;hbase.cluster.distributed\u0026lt;/name\u0026gt; \u0026lt;!-- 配置以集群方式运行--\u0026gt; \u0026lt;value\u0026gt;true\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;/configuration\u0026gt; 参考文档\n启动HBase 在/bin目录下执行./start-hbase.sh启动HBase\n$ ./start-hbase.sh OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N localhost: starting zookeeper, logging to /home/hadoop/hbase-1.2.4/bin/../logs/hbase-hadoop-zookeeper-localhost.localdomain.out localhost: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N starting master, logging to /home/hadoop/hbase-1.2.4/bin/../logs/hbase-hadoop-master-localhost.localdomain.out OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N starting regionserver, logging to /home/hadoop/hbase-1.2.4/bin/../logs/hbase-hadoop-1-regionserver-localhost.localdomain.out 由于虚拟机只分配了单核单线程，GC警告暂时忽略\n进程验证，确认启动成功\n$ jps 28821 SecondaryNameNode 28663 DataNode 29911 HRegionServer 29785 HMaster 29724 HQuorumPeer 28541 NameNode 30303 Jps 进入HBase命令行测试\n$ ./hbase shell OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.4/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] HBase Shell; enter \u0026#39;help\u0026lt;RETURN\u0026gt;\u0026#39; for list of supported commands. Type \u0026#34;exit\u0026lt;RETURN\u0026gt;\u0026#34; to leave the HBase Shell Version 1.2.4, r67592f3d062743907f8c5ae00dbbe1ae4f69e5af, Tue Oct 25 18:10:20 CDT 2016 hbase(main):001:0\u0026gt; status 1 active master, 0 backup masters, 1 servers, 0 dead, 2.0000 average load hbase(main):002:0\u0026gt; 查看其是否自动在HDFS上创建目录/hbase\n$ hadoop fs -ls / Found 2 items drwxr-xr-x - hadoop supergroup 0 2018-12-15 18:43 /hbase drwxr-xr-x - hadoop supergroup 0 2018-12-15 17:14 /test ","date":"2018-12-15T19:18:00+08:00","permalink":"/archives/195/","title":"HBase伪分布式安装"},{"content":"概述 HBase 简介 Hbase是一个分布式的、面向列的开源数据库 列式数据库：将列值统一存储 行式数据库：关系型数据库，将每一行值统一存储 Hbase在Hadoop之上提供了类似于Bigtable的能力 Bigtable是压缩、高性能、高可扩展性的基于Google GFS文件系统的数据库，用于存储大规模的结构化数据，在扩展性和性能上有很大的优势 Hbase不同于一般的关系数据库，它适合非结构化数据存储 非结构化数据：不能用二维表进行存储的数据，例如图片、文档等 Hbase地位 Apache基金会顶级项目 基于Hadoop核心HDFS系统进行数据存储，类似于Hive 可以存储超大数据并适合用来进行大数据的的实时查询 Hbase与HDFS Hbase建立在Hadoop文件系统上，利用了Hadoop文件系统的容错能力 Hbase提供对数据随机实时读/写访问功能 Hbase内部使用哈希表存储索引，可将HDFS文件中的数据进行快速查找 适用场景 瞬间写入量很大，常用数据库不好支持或者需要很高的成本支撑 数据需要长久保存，且数据量会持久增长到比较大的场景 Hbase不适合用于有join、多级索引、表关系复杂的数据模型 存储模型与原理 CAP定理 分布式系统的CAP理论：理论首先把分布式系统中的三个特性进行了如下归纳：\n一致性（C）：所有节点在同一时间具有相同的数据\n可用性（A）：保证每个请求不管成功或者失败（包含节点宕机）都有响应，但不保证获取数据的正确性\n分区容错性（P）：系统中任意信息的丢失或失败不会影响系统的继续运作。系统如果不能在时限内达成数据一致性，必须就当前操作在C和A之间做出选择。\nHbase为CP特性\nACID ACID的定义：\nAtomic原子性:所有的步骤要么全部完成要么一个也不会完成 Consistent一致性: 通过各种途径包括外键约束等任何写入数据库的数据都是有效的，不能发生表与表之间存在外键约束 Isolated隔离性: 一个未完成事务不会影响另外一个未完成事务 Durable持久性: 一旦一个事务被提交，它应该持久保存 HBase非强ACID\nHBase概念 字段 概念 备注 NameSpace 类似RDBMS的“数据库” Table 表名必须是能用在文件路径里的合法名字 原因：每个表都会映射为HDFS的文件 Row 在表里，每一行代表一个数据对象，每一行都以一个行键（Row Key）来唯一标识的，行键没有特定的数据类型，以二进制的字节进行存储 Row Key 唯一标识一行记录，不可改变，只能删除 Column 由Column family和Column qualifier组成，由（：）进行分隔 例如：family：qualifier Column family 在定义HBase表的时候需要提前设置好列族，表中所有的列都需要组织在列族里面 同一个列族的成员有相同的前缀，物理上，一个列族的成员都是储存在一起，用于存储优化 Column qualifier（列限定符） 列族中的数据通过列标识来进行映射，可以理解为一个键值对，Column qualifier就是Key 类似具体的列名 Cell 每一个行键，列族和列标识共同组成的一个单元 无特定的数据类型，以二进制字节存储 Timestamp 每个值都有一个Timestamp，作为该值特定版本的标识符 读未指定版本则返回最新版，写未指定时间则使用当前时间，HBase默认保留3个版本数据 HBase与传统关系数据库的区别 Hbase RDBMS 数据库大小 PB GB、TB 数据类型 Bytes 丰富的数据类型 事务支持 ACID只支持单个Row级别 全面的ACID支持 索引 只支持Row-Key 支持 吞吐量 百万查询/每秒 数千查询/每秒 HBase基础架构 HBase分布式系统简介 HMaster HBase主/从集群架构中的中央节点 负责将region（HBase中存储最小单元，表格的基本单位）分配给RegionServer，协调RegionServer的负载并维护集群状态 维护表和Region的元数据，不参与数据的输入/输出过程 RegionServer 维护HMaster分配给他的region，处理对这些region的IO请求 负责切分正在运行过程中变得过大的region Zookeeper 集群协调器，保证至少一个节点出于active状态 HMaster启动将系统表加载到ZK（用于服务发现） 提供HBase RegionServer状态信息 HBase原理 HBase写流程 Client先访问zookeeper，得到对应的RegionServer地址 Client对RegionServer发起写请求，RegionServer接受数据写入内存 当MemStore的大小达到一定值后，flush到StoreFile并存储到HDFS HBase特殊点在于其先写内存再写日志，通过类似MySQL中MVCC机制保证一致性\nHBase读流程 Client先访问zookeeper，得到对应的RegionServer地址 Client对RegionServer发起读请求 当RegionServer收到client的读请求后，先扫描自己的Memstore，再扫描BlockCache（加速读内容缓存区）如果还没找到则从StoreFile中读取数据，然后将数据返回给Client 读特点 读过程与HMaster无关，只和ZK有关，有效的减小HMaster负载 HBase模块协作 有关HBase的三个问题\nHBase启动时发生了什么 当RegionServer失效后会发生什么 当HMaster失效后会发生什么 HBase启动 HBase启动，注册到zookeeper，等待RegionServer汇报 RegionServer注册到zookeeper，并向HMaster汇报 对各个RegionServer（包括失效的）的数据进行整理，分配Region和meta信息（所有表的索引） RegionServer失效 HMaster将失效RegionServer上的Region分配到其他节点 HMaster更新HBase：meta表保证数据正常访问 HMaster失效 配置高可用后\n出于Backup状态的其他HMaster节点选举出一个转为Active状态 未配置高可用\n数据能正常读写，但不能创建删除表，也不能更改表结构，因为涉及meta表更新 ","date":"2018-12-15T12:06:00+08:00","permalink":"/archives/194/","title":"HBase 学习笔记"},{"content":"[toc]\n官方文档\nJDK命令行参数 标准化参数，各个jvm版本中不变 -参数 -help -version -server -client\n非标准化参数，不同jvm版本可能会变化 -X参数 -Xint:解释执行 -Xcomp:第一次全部编译成本地代码（首次运行速度慢） -Xmixed:混合模式（默认），JVM自己决定是否编译成本地代码\nPS C:\\Users\\11860\u0026gt; java -version java version \u0026#34;1.8.0_151\u0026#34; Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) -XX参数 布尔类型 格式：-XX:[+-]表示启用或禁用 -XX:+UseConcMarkSweepGC 表示启用CMS垃圾回收器 -XX:+UseG1GC 表示启用G1垃圾回收器\n值类型 格式：-XX:=表示name属性的值是value -XX:MaxGCPauseMillis=500 表示GC最大停顿时间为500 -XX:GCTimeRatio=19\n-Xms -Xmx -Xms等价于-XX:InitialHeapSize 即初始化堆大小 -Xmx等价于-XX:MaxHeapSize 最大堆大小\nJVM运行时参数 -XX:+PrintFlagsInitial 查看初始值 -XX:+PrintFlagsFinal 查看最终值 -XX:+UnlockDiagnosticVMOptions 解锁诊断参数 -XX:+PrintCommandLineFlags 打印命令行参数\n+PrintFlagsFinal = 即默认值 := 即用户或JVM修改后的值\nPS C:\\Users\\11860\u0026gt; java -XX:+PrintFlagsFinal -version [Global flags] uintx AdaptiveSizeDecrementScaleFactor = 4 {product} uintx AdaptiveSizeMajorGCDecayTimeScale = 10 {product} uintx AdaptiveSizePausePolicy = 0 {product} uintx AdaptiveSizePolicyCollectionCostMargin = 50 {product} uintx InitialCodeCacheSize = 2555904 {pd product} uintx InitialHeapSize := 268435456 {product} bool UseG1GC = false {product} jps 专用于查看java进程\nPS C:\\Users\\11860\u0026gt; jps 14624 Jps 10248 14168 proxyee-down.exe 2120 RemoteMavenServer 5192 main\\proxyee-down-core.jar jps -l显示全称\nPS C:\\Users\\11860\u0026gt; jps -l 10248 14168 proxyee-down-2.54\\proxyee-down.exe 1800 sun.tools.jps.Jps 2120 org.jetbrains.idea.maven.server.RemoteMavenServer 5192 /proxyee-down-2.54/main\\proxyee-down-core.jar 命令参考地址\njinfo 查看正在运行的java程序参数\nPS C:\\Users\\11860\u0026gt; jinfo Usage: jinfo [option] \u0026lt;pid\u0026gt; (to connect to running process) jinfo [option] \u0026lt;executable \u0026lt;core\u0026gt; (to connect to a core file) jinfo [option] [server_id@]\u0026lt;remote server IP or hostname\u0026gt; (to connect to remote debug server) where \u0026lt;option\u0026gt; is one of: -flag \u0026lt;name\u0026gt; to print the value of the named VM flag -flag [+|-]\u0026lt;name\u0026gt; to enable or disable the named VM flag -flag \u0026lt;name\u0026gt;=\u0026lt;value\u0026gt; to set the named VM flag to the given value -flags to print VM flags -sysprops to print Java system properties \u0026lt;no option\u0026gt; to print both of the above -h | -help to print this help message 查看最大内存\njinfo -flag MaxHeapSize\nPS C:\\Users\\11860\u0026gt; jinfo -flag MaxHeapSize 2120 -XX:MaxHeapSize=805306368 查看垃圾回收器\njinfo -flag UseG1GC\njstat 查看jvm统计信息\nPS C:\\Users\\11860\u0026gt; jstat -help Usage: jstat -help|-options jstat -\u0026lt;option\u0026gt; [-t] [-h\u0026lt;lines\u0026gt;] \u0026lt;vmid\u0026gt; [\u0026lt;interval\u0026gt; [\u0026lt;count\u0026gt;]] Definitions: \u0026lt;option\u0026gt; An option reported by the -options option \u0026lt;vmid\u0026gt; Virtual Machine Identifier. A vmid takes the following form: \u0026lt;lvmid\u0026gt;[@\u0026lt;hostname\u0026gt;[:\u0026lt;port\u0026gt;]] Where \u0026lt;lvmid\u0026gt; is the local vm identifier for the target Java virtual machine, typically a process id; \u0026lt;hostname\u0026gt; is the name of the host running the target Java virtual machine; and \u0026lt;port\u0026gt; is the port number for the rmiregistry on the target host. See the jvmstat documentation for a more complete description of the Virtual Machine Identifier. \u0026lt;lines\u0026gt; Number of samples between header lines. \u0026lt;interval\u0026gt; Sampling interval. The following forms are allowed: \u0026lt;n\u0026gt;[\u0026#34;ms\u0026#34;|\u0026#34;s\u0026#34;] Where \u0026lt;n\u0026gt; is an integer and the suffix specifies the units as milliseconds(\u0026#34;ms\u0026#34;) or seconds(\u0026#34;s\u0026#34;). The default units are \u0026#34;ms\u0026#34;. \u0026lt;count\u0026gt; Number of samples to take before terminating. -J\u0026lt;flag\u0026gt; Pass \u0026lt;flag\u0026gt; directly to the runtime system. 官方文档\n可选参数\n-statOption Determines the statistics information the jstat command displays. The following lists the available options. Use the -options general option to display the list of options for a particular platform installation. See Stat Options and Output.\nclass: Displays statistics about the behavior of the class loader. compiler: Displays statistics about the behavior of the Java HotSpot VM Just-in-Time compiler. gc: Displays statistics about the behavior of the garbage collected heap. gccapacity: Displays statistics about the capacities of the generations and their corresponding spaces. gccause: Displays a summary about garbage collection statistics (same as -gcutil), with the cause of the last and current (when applicable) garbage collection events. gcnew: Displays statistics of the behavior of the new generation. gcnewcapacity: Displays statistics about the sizes of the new generations and its corresponding spaces. gcold: Displays statistics about the behavior of the old generation and metaspace statistics. gcoldcapacity: Displays statistics about the sizes of the old generation. gcmetacapacity: Displays statistics about the sizes of the metaspace. gcutil: Displays a summary about garbage collection statistics. printcompilation: Displays Java HotSpot VM compilation method statistics. 类加载 -class Stat Options and Output The following information summarizes the columns that the jstat command outputs for each statOption.\n-class option Class loader statistics.\nLoaded: Number of classes loaded. Bytes: Number of kBs loaded. Unloaded: Number of classes unloaded. Bytes: Number of Kbytes unloaded. Time: Time spent performing class loading and unloading operations. PS C:\\Users\\11860\u0026gt; jstat -class 2120 Loaded Bytes Unloaded Bytes Time 4271 7387.1 98 137.6 3.28 垃圾收集 -gc 输出含义 S0C、S1C、S0U、S1U：S0和S1的总用量与使用量 EC、EU：Eden区总量与使用量 OC、OU：Old区总量与使用量 MC、MU：Metaspace区总量与使用量 CCSC、CCSU：压缩类空间总量与使用量 YGC、YGCT：YoungGC的次数与时间 FGC、FGCT：FullGC的次数与时间 GCT：总的GC时间\n解释：S0=S1，同一时间只启用s1或s0，S0+S1+Eden=Young，内存堆=Young+Old 默认命令只输出一次\nPS C:\\Users\\11860\u0026gt; jstat -gc 2120 S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT 35840.0 51712.0 0.0 0.0 158720.0 11900.5 102400.0 9199.6 24192.0 22964.4 2944.0 2550.2 8 0.314 2 0.154 0.468 指定输出参数为每隔一秒输出一次，共输出10次\nPS C:\\Users\\11860\u0026gt; jstat -gc 5192 1000 10 S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT 4352.0 4352.0 2228.1 0.0 34944.0 19156.1 87424.0 75913.1 36864.0 35975.9 0.0 0.0 2338 7.004 8 0.459 7.463 4352.0 4352.0 0.0 2131.3 34944.0 7184.2 87424.0 76050.5 36864.0 35975.9 0.0 0.0 2339 7.006 8 0.459 7.466 4352.0 4352.0 0.0 2131.3 34944.0 31117.2 87424.0 76050.5 36864.0 35975.9 0.0 0.0 2339 7.006 8 0.459 7.466 4352.0 4352.0 2320.3 0.0 34944.0 8482.5 87424.0 76050.5 36864.0 35975.9 0.0 0.0 2340 7.009 8 0.459 7.468 4352.0 4352.0 2320.3 0.0 34944.0 29236.0 87424.0 76050.5 36864.0 35975.9 0.0 0.0 2340 7.009 8 0.459 7.468 4352.0 4352.0 0.0 2058.5 34944.0 11809.7 87424.0 76348.8 36864.0 35975.9 0.0 0.0 2341 7.012 8 0.459 7.471 4352.0 4352.0 2339.3 0.0 34944.0 0.0 87424.0 76348.8 36864.0 35975.9 0.0 0.0 2342 7.015 8 0.459 7.474 4352.0 4352.0 2339.3 0.0 34944.0 11533.3 87424.0 76348.8 36864.0 35975.9 0.0 0.0 2342 7.015 8 0.459 7.474 4352.0 4352.0 0.0 2076.1 34944.0 3826.1 87424.0 76520.6 36864.0 35975.9 0.0 0.0 2343 7.017 8 0.459 7.476 4352.0 4352.0 0.0 2076.1 34944.0 11102.7 87424.0 76520.6 36864.0 35975.9 0.0 0.0 2343 7.017 8 0.459 7.476 JIT编译 -complier\nPS C:\\Users\\11860\u0026gt; jstat -compiler 2120 Compiled Failed Invalid Time FailedType FailedMethod 5234 0 0 16.70 0 JVM内存结构 内存溢出 堆区内存溢出 非堆区内存溢出 导出内存映像文件 当内存溢出时自动导出 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./\njmap命令手动导出\nPS C:\\Users\\11860\u0026gt; jmap -help Usage: jmap [option] \u0026lt;pid\u0026gt; (to connect to running process) jmap [option] \u0026lt;executable \u0026lt;core\u0026gt; (to connect to a core file) jmap [option] [server_id@]\u0026lt;remote server IP or hostname\u0026gt; (to connect to remote debug server) where \u0026lt;option\u0026gt; is one of: \u0026lt;none\u0026gt; to print same info as Solaris pmap -heap to print java heap summary -histo[:live] to print histogram of java object heap; if the \u0026#34;live\u0026#34; suboption is specified, only count live objects -clstats to print class loader statistics -finalizerinfo to print information on objects awaiting finalization -dump:\u0026lt;dump-options\u0026gt; to dump java heap in hprof binary format dump-options: live dump only live objects; if not specified, all objects in the heap are dumped. format=b binary format file=\u0026lt;file\u0026gt; dump heap to \u0026lt;file\u0026gt; Example: jmap -dump:live,format=b,file=heap.bin \u0026lt;pid\u0026gt; -F force. Use with -dump:\u0026lt;dump-options\u0026gt; \u0026lt;pid\u0026gt; or -histo to force a heap dump or histogram when \u0026lt;pid\u0026gt; does not respond. The \u0026#34;live\u0026#34; suboption is not supported in this mode. -h | -help to print this help message -J\u0026lt;flag\u0026gt; to pass \u0026lt;flag\u0026gt; directly to the runtime system 手动导出映像文件\nPS C:\\Users\\11860\u0026gt; jmap -dump:format=b,file=heap.hprof 11720 Dumping heap to L:\\heap.hprof ... Heap dump file created jmap相关命令\nPS C:\\Users\\11860\u0026gt; jmap -heap 11720 Attaching to process ID 11720, please wait... Debugger attached successfully. Server compiler detected. JVM version is 25.151-b12 using thread-local object allocation. Parallel GC with 8 thread(s) Heap Configuration: MinHeapFreeRatio = 0 MaxHeapFreeRatio = 100 MaxHeapSize = 33554432 (32.0MB) NewSize = 11010048 (10.5MB) MaxNewSize = 11010048 (10.5MB) OldSize = 22544384 (21.5MB) NewRatio = 2 SurvivorRatio = 8 MetaspaceSize = 21807104 (20.796875MB) CompressedClassSpaceSize = 1073741824 (1024.0MB) MaxMetaspaceSize = 17592186044415 MB G1HeapRegionSize = 0 (0.0MB) Heap Usage: PS Young Generation Eden Space: capacity = 7864320 (7.5MB) used = 6748264 (6.435646057128906MB) free = 1116056 (1.0643539428710938MB) 85.80861409505208% used From Space: capacity = 1572864 (1.5MB) used = 950272 (0.90625MB) free = 622592 (0.59375MB) 60.416666666666664% used To Space: capacity = 1572864 (1.5MB) used = 0 (0.0MB) free = 1572864 (1.5MB) 0.0% used PS Old Generation capacity = 22544384 (21.5MB) used = 21187512 (20.20598602294922MB) free = 1356872 (1.2940139770507812MB) 93.9813303392987% used 16212 interned Strings occupying 2160560 bytes. MAT分析内存溢出 工具下载 导入文件显示怀疑内存溢出（Leak suspects）\n选择直方图\n找到影响最大的项目，右键查看强引用以判断这些占用内存大的对象由谁引用，由此定位内存溢出问题\n选择实体树\n也可看出存在大量对象加载\njstack 分析线程状态\nPS C:\\Users\\11860\u0026gt; jstack Usage: jstack [-l] \u0026lt;pid\u0026gt; (to connect to running process) jstack -F [-m] [-l] \u0026lt;pid\u0026gt; (to connect to a hung process) jstack [-m] [-l] \u0026lt;executable\u0026gt; \u0026lt;core\u0026gt; (to connect to a core file) jstack [-m] [-l] [server_id@]\u0026lt;remote server IP or hostname\u0026gt; (to connect to a remote debug server) Options: -F to force a thread dump. Use when jstack \u0026lt;pid\u0026gt; does not respond (process is hung) -m to print both java and native frames (mixed mode) -l long listing. Prints additional information about locks -h or -help to print this help message 手动导出\njstack 23276 \u0026gt; RemoteMavenServerJstack 线程状态官方文档\n[线程状态互相转化]https://mp.weixin.qq.com/s/GsxeFM7QWuR\u0026ndash;Kbpb7At2w)\njstack定位CPU飙高问题 tomcat 远程监控 开启tomcat远程监控 startup.sh 在最后一行的\u0026rsquo;start \u0026ldquo;$@\u0026quot;\u0026lsquo;前加上\u0026rsquo;jpda\u0026rsquo;\ncatalina.sh 搜索jpda,可看到jpda使用说明\n","date":"2018-12-09T17:05:00+08:00","permalink":"/archives/193/","title":"Java8生产环境下性能监控与调优详解笔记"},{"content":"jpa方法命名规则 关键字 方法命名 sql where字句 And findByNameAndPwd where name= ? and pwd =? Or findByNameOrSex where name= ? or sex=? Is,Equals findById,findByIdEquals where id= ? Between findByIdBetween where id between ? and ? LessThan findByIdLessThan where id \u0026lt; ? LessThanEquals findByIdLessThanEquals where id \u0026lt;= ? GreaterThan findByIdGreaterThan where id \u0026gt; ? GreaterThanEquals findByIdGreaterThanEquals where id \u0026gt; = ? After findByIdAfter where id \u0026gt; ? Before findByIdBefore where id \u0026lt; ? IsNull findByNameIsNull where name is null isNotNull,NotNull findByNameNotNull where name is not null Like findByNameLike where name like ? NotLike findByNameNotLike where name not like ? StartingWith findByNameStartingWith where name like \u0026lsquo;?%\u0026rsquo; EndingWith findByNameEndingWith where name like \u0026lsquo;%?\u0026rsquo; Containing findByNameContaining where name like \u0026lsquo;%?%\u0026rsquo; OrderBy findByIdOrderByXDesc where id=? order by x desc Not findByNameNot where name \u0026lt;\u0026gt; ? In findByIdIn(Collection c) where id in (?) NotIn findByIdNotIn(Collection c) where id not in (?) TRUE findByAaaTue where aaa = true FALSE findByAaaFalse where aaa = false IgnoreCase findByNameIgnoreCase where UPPER(name)=UPPER(?) 常用注解 自动生成时间和修改者 @CreatedDate、@CreatedBy、@LastModifiedDate、@LastModifiedBy\n在spring jpa中，支持在字段或者方法上进行注解@CreatedDate、@CreatedBy、@LastModifiedDate、@LastModifiedBy，从字面意思可以很清楚的了解，这几个注解的用处。\n@CreatedDate 表示该字段为创建时间时间字段，在这个实体被insert的时候，会设置值\n@CreatedBy 表示该字段为创建人，在这个实体被insert的时候，会设置值\n@LastModifiedDate、@LastModifiedBy同理。\n自动建表引擎设置 使用SpringDataJpa自动建表默认为MyISAM，而主流大多数场景都需要使用InnoDB，可通过配置文件配置即可\nspring: jpa: database-platform: org.hibernate.dialect.MySQL55Dialect 注意：类名有所变化\n早期通过网上搜索的类名都说为org.hibernate.dialect.MySQL5InnoDBDialect，当实际测试时发现引擎依然为MyISAM，通过进一步查询源码发现早期类名已经弃用\n/* * Hibernate, Relational Persistence for Idiomatic Java * * License: GNU Lesser General Public License (LGPL), version 2.1 or later. * See the lgpl.txt file in the root directory or \u0026lt;http://www.gnu.org/licenses/lgpl-2.1.html\u0026gt;. */ package org.hibernate.dialect; /** A Dialect for MySQL 5 using InnoDB engine * * @author Gavin King, * @author Scott Marlow * @deprecated Use \u0026#34;hibernate.dialect.storage_engine=innodb\u0026#34; environment variable or JVM system property instead. */ @Deprecated public class MySQL5InnoDBDialect extends MySQL5Dialect { @Override protected MySQLStorageEngine getDefaultMySQLStorageEngine() { return InnoDBStorageEngine.INSTANCE; } } 但注释里并没有指明新的替代类名，而是建议通过配置实现，但SpringDataJpa自动装配的类里没有支持该配置，故如果不单独引入hibernate依赖是无法使用该配置的\n最后在stackoverflow上看到解决方案：where to put hibernate.dialect.storage_engine property\n即使用新的类名org.hibernate.dialect.MySQL55Dialect替换即可，尽管两个类名初看没差别，实际上差了一个数字5，影响比较大\nMySQL驱动变化 在使用了新版的驱动类后，不管是MyBatis还是Jpa，都会报时区不一致和报错信息乱码的错误需要将连接URL更新\n新配置\nspring: datasource: driver-class-name: com.mysql.cj.jdbc.Driver username: root password: url: jdbc:mysql://localhost/pw_manager?useSSL=false\u0026amp;useUnicode=true\u0026amp;characterEncoding=UTF-8\u0026amp;serverTimezone=UTC 原配置\nspring: datasource: driver-class-name: com.mysql.jdbc.Driver username: cmsspringcloud password: cmsdbadmin url: jdbc:mysql://localhost:3306/cms_springcloud?characterEncoding=utf-8\u0026amp;useSSL=false ","date":"2018-12-09T17:04:00+08:00","permalink":"/archives/192/","title":"Spring Data JPA 知识记录"},{"content":"查看方法的具体实现 ctrl+alt+鼠标点击\n查找接口实现类 ctrl + alt +B\n自动补全函数返回值 IDEA的ctrl+alt+v相当于eclipse的CTRL+F1\n查找源码类 ctrl+N\n查找方法 ctrl+F12\n复制当前行到下一行 ctrl+D\n","date":"2018-12-09T17:02:00+08:00","permalink":"/archives/190/","title":"IDEA常用快捷键-windows"},{"content":"伴生类与伴生对象 特点\n同名的class与同名的object互为伴生类和伴生对象 若object中有名为apply的方法，则通过类名()会在类初始化完成后调用object的apply方法 若class有名为apply的方法，则通过对象()默认调用class的apply方法 object A{ def apply(){ ... } } class A{ def apply(){ ... } } val b = A() //此处默认调用object 的apply方法 val c = new A() c() // 此处调用class 的apply方法 最佳实践：在object的apply方法中new class\ntrait 类似java的接口 用法\nclass A extends B with C with D ... 集合 List 特点\nNil 为不可变的空List List由head和tail组成，head为第一个元素，剩余元素为tail List组装通过::实现 val l1 = 1 :: Nil // 返回包含一个元素的集合List(1)，即把1作为head，Nil作为tail List可通过:_*转成可变参数 l1.tail:_* 模式匹配 特点\n类似java中的switch 支持多重过滤 支持内容、类型匹配 函数高级操作 字符串高级操作 插值 字符串前通过字母s声明，内容通过$接变量名即可\nval h = \u0026#34;hello\u0026#34; val name = \u0026#34;FJY\u0026#34; println(h +\u0026#34; \u0026#34;+ name) // hello FJY println(s\u0026#34;hello $name\u0026#34;) // hello FJY 输出结果相同\n多行字符串 IDEA创建多行字符串：Shift+’ 按3次即可，创建结果为\nval b = \u0026#34;\u0026#34;\u0026#34; | \u0026#34;\u0026#34;\u0026#34;.stripMargin 样例 val b = \u0026#34;\u0026#34;\u0026#34; |多行字符串样例 |Hello Scala \u0026#34;\u0026#34;\u0026#34;.stripMargin println(b) 最终结果为原样输出\n匿名函数 函数可以命名，也可以不命名\n定义方式：（参数名：参数类型\u0026hellip;）=\u0026gt; 函数体\n样例1 val m = (x:Int) =\u0026gt; x+1 println(m(10)) // 11 (x:Int) =\u0026gt; x+1为匿名函数，等价于{(x:Int) =\u0026gt; x+1}\n匿名函数可传给变量，也可传给函数\n样例2 def plus = (x:Int,y:Int)=\u0026gt; x+y println(plus(2,3)) // 5 柯里化函数 样例 // 原函数 def plus1(x:Int,y:Int)= x+y println(plus1(2,3)) // 柯里化后 def plus2(x:Int)(y:Int)= x+y println(plus2(2)(3)) 即将多参数分拆\n高阶函数 map 作用：逐个操作集合中的元素\nlist.map((x)=\u0026gt;x+1) // 对集合中的每个元素执行+1操作 list.map(_ * 2) // 集合中每个元素*2，_ 为占位符 其中x=\u0026gt;x+1为匿名函数，只有一个元素参数的函数括号可以省略\nfilter 作用：过滤\nlist.filter(_ \u0026gt; 2) // 只拿出大于2的元素 reduce 作用：操作两个元素\nlist.reduce(_ + _) // 集合中相邻的元素两两相加，即求和 reduceXX方法 reduceLeft,reduceRight,foldLeft,foldRight方法\nval a = List(1,7,2,9) val a1 = a.reduceLeft(_ - _)// ((1-7) - 2) - 9 = -17 val a2 = a.foldLeft(0)(_ - _) // 0-1-7-2-9 = -19 对于foldLeft方法还有一种简写，这种写法的本意是让你通过/:来联想一棵树的样子 对/:操作符来说，初始值是第一个操作元本题中是0，：后是第二个操作元a\nval a3 = (0 /: a)(_ - _) // 等价于a.foldLeft(0)(_ - _ foldRight或:\\的变体 val a4 = a.foldRight(0)(_ - _)// 1-(7-(2-(9-0))) = -13 val a5 = (a :\\ 0)(_ - _) // 等价于a.foldRight(0)(_ - _) flatten 作用：将多个集合元素转成一个集合元素\nflatMap 作用：将多个集合元素转成一个集合后对每个元素执行操作\nSpark word count text_file.flatMap(lambda line : line.split()).map(lambda word:(word,1)).reduceByKey(lambda a, b:a+b) 即将文件中的元素拆分，再映射上一个初值，最后两两相加\n原生scala实现wordcount txts.flatMap(_.split(\u0026#34;,\u0026#34;)).map(x =\u0026gt; (x,1)).groupBy(_._1).mapValues(_.size).foreach(println) 偏函数 含义：被包在花括号内没有match的一组case语句\n一般通过match实现方式\nval names = Array(\u0026#34;A\u0026#34;,\u0026#34;B\u0026#34;,\u0026#34;C\u0026#34;) val name = names(Random.nextInt(names.length)) name match { case \u0026#34;A\u0026#34; =\u0026gt; \u0026#34;AAAAAAAA\u0026#34; case \u0026#34;B\u0026#34; =\u0026gt; \u0026#34;BBBBBBBB\u0026#34; case _ =\u0026gt; \u0026#34;CCCCCCCC\u0026#34; } 通过偏函数实现\n// -A 输入参数类型 -B 输出参数类型 def say:PartialFunction[String,String]={ case \u0026#34;A\u0026#34; =\u0026gt; \u0026#34;AAAAAAAA\u0026#34; case \u0026#34;B\u0026#34; =\u0026gt; \u0026#34;BBBBBBBB\u0026#34; case _ =\u0026gt; \u0026#34;CCCCCCCC\u0026#34; } 能自动进行模式匹配\n隐式转换 解决问题：为一个已经存在的类添加一个方法\nJava实现方式：动态代理\nScala实现方式：隐式转换\n注意：该特性为双刃剑，容易使代码难以理解\nScala 2.10引入了一种叫做隐式类的新特性。隐式类指的是用implicit关键字修饰的类。在对应的作用域内，带有这个关键字的类的主构造函数可用于隐式转换。\n样例 object ImplicitApp extends App { // 定义隐式转换函数，传入源对象，输出目标对象 implicit def man2superman(man: Man):Superman=new Superman(man.name) val man = new Man(\u0026#34;FJY\u0026#34;) man.fly() } class Man(val name:String){ def eat(): Unit ={ printf(s\u0026#34;man[ $name ] eat...\u0026#34;) } } class Superman(val name:String){ def fly(): Unit ={ printf(s\u0026#34;superman[ $name ] fly...\u0026#34;) } } 样例2：为Java IO类增加读取文件内容方法 object RichFileApp extends App { // 目的在于为Java IO类增加读取文件内容方法 implicit def file2RichFile(file: File):RichFile=new RichFile(file) val f = new File(\u0026#34;tmp/hello.txt\u0026#34;) println(f.readConent) } class RichFile(file:File){ def readConent={ Source.fromFile(file.getPath).mkString } } 隐式参数 隐式参数会自动寻找方法、类内部、伴生类内类型匹配的唯一参数，不推荐使用\n样例 object ImplicitParaApp { implicit val test = \u0026#34;test\u0026#34; def main(args: Array[String]): Unit = { // 隐式参数会自动寻找方法、类内部、伴生类内类型匹配的唯一参数 // 若外部定义了多个隐式参数则报错 def testPara(implicit name:String): Unit ={ println(name + \u0026#34;...\u0026#34;) } testPara } } 隐式类 使用隐式类时，类名必须在当前作用域内可见且无歧义，这一要求与隐式值等其他隐式类型转换方式类似。\nobject ImplicitClassApp extends App { implicit class Calculator(x:Int){ def add(a:Int) = a+x } // 原本整数1不具有add方法，使用隐式类后拥有该方法 println(1.add(2)) } 操作外部数据 scala.io.Source 该对象支持操作文件、网络流等，例如读取文件方法源码支持通过柯里化指定字符集\ndef fromFile(name: String)(implicit codec: Codec): BufferedSource = fromFile(new JFile(name))(codec) 样例——读取文件 val file = Source.fromFile(\u0026#34;tmp/file.txt\u0026#34;)(Codec.UTF8) 样例——逐行读取 def readLine(file: BufferedSource): Unit ={ for (line \u0026lt;- file.getLines()){ println(line) } } 样例——逐个字符读取 def readChar(file: BufferedSource): Unit ={ for (ele \u0026lt;- file){ print(ele) } } 样例——网络读取 def readNet(url:String): Unit ={ val file = Source.fromURL(url) for (line \u0026lt;- file.getLines()){ println(line) } } ","date":"2018-12-09T16:59:00+08:00","permalink":"/archives/189/","title":"Scala 学习笔记"},{"content":"宝塔 宝塔从2017年底开始接触并全面在自己服务器上推广使用，到目前为止手上的服务器需要建站数据库之类的都部署了宝塔，同时开发测试环境的服务器为了方便管理也部署了宝塔，可以说用了也算比较长时间了，宝塔的优势也比较明显\n1. 界面简单，逻辑明确 系统信息一目了然\n2. 软件安装简单 基本都是无脑安装，之前在开发测试环境搭建Redis和MySQL就是通过宝塔快速安装的，5分钟内搭建完毕快速测试，而且卸载的比较干净，特别是如果小白自己下载MySQL编译包安装很容易出现卸载残留导致无法安装新的MySQL\n3. 快捷的防火墙管理 宝塔提供的防火墙管理基本满足需求，比使用防火墙命令逐个放行便捷\n总结 总的来说宝塔适合建站管理，快速应用部署，但对于复杂的操作还是需要自行通过终端解决，而且防火墙功能有限，抗CC攻击等都需要付费或者搭建其他软件\nAPPNODE 在最近看到APPNODE这个管理面板，在虚拟机跑了一下发现其功能强大，还是有点意思的。\n以下逐一说明（无广告费）\n1. 简单安装 其实APPNODE和宝塔都是一键安装，自选服务器环境这些共同点，从这点上来说没什么差别，两者甚至连默认端口号都是一样的。。\n装好后的首页，默认展示建站指引（好评，第一次在宝塔建站还是摸索了一会的）\n更详细的服务器信息 展示的服务器信息比宝塔详细很多，各项指标都有展示\n更多的系统软件 可以安装许多软件目前都是免费的，软件也和宝塔类型类似，但有一些特别的，比如frp内网穿透之类还是不错的，比自己部署简单不少，不知道以后会不会收费\n网络测试软件看看就好\n更丰富的软件市场 和宝塔类似的软件市场，同样种类更多，有些2017年才兴起的小众软件也在里面，也有不少刚需平台例如owncloud、nextcloud，而同样用宝塔搭建私有云还是挺折腾的\n功能完备的防火墙管理（重点） 在我看来这个是与宝塔最大的不同，前面那些只要宝塔想加入其实也不难，但这个支持自己配置的防火墙规则，其实内部实现应该是通过nginx代理配置实现，但比宝塔收费的要良心，通过简单设置参数即可抵抗简单的DDOS攻击\n攻击测试，通过Apache ab压测进行DDOS攻击\n未开启之前所有请求顺利通过\n在配置1秒内最大访问次数后成功拦截，1000个请求只有70个有效，说明DDOS流量劫持被成功拦截\n更灵活的网站控制 日志、SSL、负载情况、网站更新自动提醒重载nginx等\n另外还提供免费SSL证书自动续租，十分好评\n总结 APPNODE适合对服务器操作熟悉，需要更大灵活度的定制服务器功能的用户，对于建站灵活度也非常高，而缺点也比较明显，功能多必然带来操作复杂，许多功能小白可能没听过，乱操作一波可能会翻车。。\n","date":"2018-12-09T16:35:04+08:00","permalink":"/archives/188/","title":"Linux面板APPNODE与宝塔横评"},{"content":"总览 三种启动方式：\n动态参数启动（指定端口号） 配置文件启动（生产环境推荐） 最简单启动 官方安装指南 安装前置环境gcc-c++\n$ yum install -y gcc-c++ 官方下载安装文档\n$ wget http://download.redis.io/releases/redis-5.0.0.tar.gz $ tar xzf redis-5.0.0.tar.gz $ cd redis-5.0.0 $ make MALLOC=libc $ make install 编译后在src目录下生成可执行文件\n$ ls | grep redis- redis-benchmark ——性能测试 redis-benchmark.c redis-benchmark.o redis-check-aof ——aof修复 redis-check-aof.c redis-check-aof.o redis-check-rdb ——rdb修复 redis-check-rdb.c redis-check-rdb.o redis-cli ——客户端 redis-cli.c redis-cli.o redis-sentinel ——集群 redis-server ——服务器 redis-trib.rb 启动 简单启动 $ redis-server 2660:C 21 Nov 2018 20:01:10.958 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 2660:C 21 Nov 2018 20:01:10.958 # Redis version=5.0.0, bits=64, commit=00000000, modified=0, pid=2660, just started 2660:C 21 Nov 2018 20:01:10.958 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf 2660:M 21 Nov 2018 20:01:10.959 * Increased maximum number of open files to 10032 (it was originally set to 1024). _._ _.-``__ \u0026#39;\u0026#39;-._ _.-`` `. `_. \u0026#39;\u0026#39;-._ Redis 5.0.0 (00000000/0) 64 bit .-`` .-```. ```\\/ _.,_ \u0026#39;\u0026#39;-._ ( \u0026#39; , .-` | `, ) Running in standalone mode |`-._`-...-` __...-.``-._|\u0026#39;` _.-\u0026#39;| Port: 6379 | `-._ `._ / _.-\u0026#39; | PID: 2660 `-._ `-._ `-./ _.-\u0026#39; _.-\u0026#39; |`-._`-._ `-.__.-\u0026#39; _.-\u0026#39;_.-\u0026#39;| | `-._`-._ _.-\u0026#39;_.-\u0026#39; | http://redis.io `-._ `-._`-.__.-\u0026#39;_.-\u0026#39; _.-\u0026#39; |`-._`-._ `-.__.-\u0026#39; _.-\u0026#39;_.-\u0026#39;| | `-._`-._ _.-\u0026#39;_.-\u0026#39; | `-._ `-._`-.__.-\u0026#39;_.-\u0026#39; _.-\u0026#39; `-._ `-.__.-\u0026#39; _.-\u0026#39; `-._ _.-\u0026#39; `-.__.-\u0026#39; 2660:M 21 Nov 2018 20:01:10.961 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 2660:M 21 Nov 2018 20:01:10.961 # Server initialized 2660:M 21 Nov 2018 20:01:10.961 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add \u0026#39;vm.overcommit_memory = 1\u0026#39; to /etc/sysctl.conf and then reboot or run the command \u0026#39;sysctl vm.overcommit_memory=1\u0026#39; for this to take effect. 2660:M 21 Nov 2018 20:01:10.961 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command \u0026#39;echo never \u0026gt; /sys/kernel/mm/transparent_hugepage/enabled\u0026#39; as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 2660:M 21 Nov 2018 20:01:10.961 * Ready to accept connections 客户端连接测试\n$ redis-cli -h 127.0.0.1 -p 6379 127.0.0.1:6379\u0026gt; ping PONG 127.0.0.1:6379\u0026gt; set hello world OK 127.0.0.1:6379\u0026gt; get hello \u0026#34;world\u0026#34; 动态参数启动 $ redis-server --port 6380 连接默认端口测试\n$ redis-cli -h 127.0.0.1 -p 6379 Could not connect to Redis at 127.0.0.1:6379: Connection refused not connected\u0026gt; exit 连接新端口测试\n$ redis-cli -h 127.0.0.1 -p 6380 127.0.0.1:6380\u0026gt; ping PONG 127.0.0.1:6380\u0026gt; exit 通过进程查看\n$ ps -ef | grep redis-server |grep -v grep root 2666 1469 0 20:04 pts/0 00:00:00 redis-server *:6380 配置文件启动 拷贝默认配置\n$ pwd /root/redis-5.0.0 $ mkdir config $ cp redis.conf config/ $ cd config/ $ ls redis.conf 查看默认配置，以下命令用于去除注释和空格\n$ cat redis.conf | grep -v \u0026#34;#\u0026#34; | grep -v \u0026#34;^$\u0026#34; \u0026gt; redis-default.conf $ cat redis-default.conf bind 127.0.0.1 protected-mode yes port 6379 #端口号 tcp-backlog 511 timeout 0 tcp-keepalive 300 daemonize no #默认非守护进程启动 supervised no pidfile /var/run/redis_6379.pid loglevel notice logfile \u0026#34;\u0026#34; #日志文件名称 databases 16 always-show-logo yes save 900 1 save 300 10 save 60 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb dir ./ #工作目录 replica-serve-stale-data yes replica-read-only yes repl-diskless-sync no repl-diskless-sync-delay 5 repl-disable-tcp-nodelay no replica-priority 100 lazyfree-lazy-eviction no lazyfree-lazy-expire no lazyfree-lazy-server-del no replica-lazy-flush no appendonly no appendfilename \u0026#34;appendonly.aof\u0026#34; appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb aof-load-truncated yes aof-use-rdb-preamble yes lua-time-limit 5000 slowlog-log-slower-than 10000 slowlog-max-len 128 latency-monitor-threshold 0 notify-keyspace-events \u0026#34;\u0026#34; hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-size -2 list-compress-depth 0 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 hll-sparse-max-bytes 3000 stream-node-max-bytes 4096 stream-node-max-entries 100 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit replica 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 hz 10 dynamic-hz yes aof-rewrite-incremental-fsync yes rdb-save-incremental-fsync yes 由于默认配置redis为单线程，故一台服务器一般起多个redis实例，配置文件以端口号区分\n$ cp redis-default.conf redis-6379.conf $ ll 总用量 72 -rw-r--r--. 1 root root 65 11月 21 20:46 redis-6379.conf -rw-r--r--. 1 root root 62155 11月 21 20:36 redis.conf -rw-r--r--. 1 root root 1425 11月 21 20:41 redis-default.conf 简化配置如下：\nport 6380 daemonize yes logfile \u0026#34;6380.log\u0026#34; dir \u0026#34;/root/tmp/redis\u0026#34; 配置文件启动\n$ redis-server config/redis-6380.conf ","date":"2018-12-09T11:19:49+08:00","permalink":"/archives/187/","title":"redis多种启动配置方式"},{"content":"地址 5折圣诞优惠码：CHRISTMASANDNEWYEAR ，账户内现有产品可使用优惠码 RENEW2019 获得 35% off 续约折扣\nAAEX\n原因 从最开始使用免费账号到坚持自建服务器搭建专线梯子，一直认为专线梯子才是王道，何必花钱购买别人的昂贵加速服务，然而。。直到发现AAEX。。其与自建服务器区别如下\n默认套餐 默认套餐对自建服务器的吸引不大而且和其他服务提供商产品类型差不多，适用于视频搬运。另外，因为自建服务器成本和套餐差距不大，除了需要自己维护之外没太大区别，而且自建服务器还有台服务器可以折腾\n对于本人来说一般就偶尔看个视频，自建服务器一个月才3-5刀，还不限流量，但缺点也是节点有限，而且带宽流量极大浪费，然后发现了另一个隐藏套餐\n固定流量包 这个就比较有意思了，按量付费，和买运营商流量包一样，按需购买，还不清零，永久有效，对于查阅资料，游戏玩家来说还是非常友好的，而且所有节点均可用\n本人试了下买了200多G的流量套餐100块，38个节点随便用，全平台支持（好评，特别是苹果用户，下载客户端非常麻烦，而且一般还要付费），AAEX官方提供appleID下载梯子app（满分）\n效果 常见的控制台，多个香港节点，延迟低带宽大\n使用RSS订阅客户端自动更新节点，同时带有流量统计，比原生的SSR体验好了不少\n全节点均可用\n通过在线视频和网页测试基本秒开，在线4K无压力，带宽也能跑满本地带宽（20M）\n总结 一趟测试下来，优点如下\n无需自己购买和搭建服务器 无需担心IP被墙 多节点选择，延迟低（自建服务器香港、日本机房贼贵） 固定流量包，不清零 全平台支持 用了这个之后基本就告别自建服务器的梯子了。。\n","date":"2018-12-09T11:17:00+08:00","permalink":"/archives/186/","title":"极力推荐的网络加速（梯子）服务"},{"content":"基础环境 4台服务器，一台外部客户机\n安装指南 官方支持C++源码编译安装和Docker安装\nC++源码编译安装地址\nDocker镜像仓库\n停止所有容器命令 docker stop $(docker ps -a -q) docker rm $(docker ps -a -q) 安装 mesos主节点 node1\ndocker pull mesosphere/mesos-master:1.7.0 node2(host)\ndocker run --name mesos-zookeeper -p 2181:2181 --restart always -d zookeeper:3.5 node1执行脚本(需替换zk IP地址)\nMESOS_QUORUM=1即1为单机运行，3个节点为2，计算方式为 n÷2+1 \\\n若不手动指定hostname映射的IP，则所有节点默认通过主机名进行通信\n使用host通信脚本\ndocker run -d --net=host \\ -e MESOS_PORT=5050 \\ -e MESOS_ZK=zk://192.168.0.32:2181/mesos \\ -e MESOS_QUORUM=1 \\ -e MESOS_REGISTRY=in_memory \\ -e MESOS_LOG_DIR=/var/log/mesos \\ -e MESOS_WORK_DIR=/var/tmp/mesos \\ -v \u0026#34;$(pwd)/log/mesos:/var/log/mesos\u0026#34; \\ -v \u0026#34;$(pwd)/work/mesos:/var/tmp/mesos\u0026#34; \\ -p 5050:5050 \\ --name=mesos-master \\ mesosphere/mesos-master:1.7.0 \\ 使用IP通信脚本\ndocker run -d --net=host \\ --hostname=192.168.0.32 \\ -e MESOS_PORT=5050 \\ -e MESOS_ZK=zk://192.168.0.32:2181/mesos \\ -e MESOS_QUORUM=1 \\ -e MESOS_REGISTRY=in_memory \\ -e MESOS_LOG_DIR=/var/log/mesos \\ -e MESOS_WORK_DIR=/var/tmp/mesos \\ -v \u0026#34;$(pwd)/log/mesos:/var/log/mesos\u0026#34; \\ -v \u0026#34;$(pwd)/work/mesos:/var/tmp/mesos\u0026#34; \\ -p 5050:5050 \\ --name=mesos-master \\ mesosphere/mesos-master:1.7.0 \\ --no-hostname_lookup \\ --ip=192.168.0.33 查看日志\ndocker logs mesos-master 成功输出日志\nI1003 08:41:29.868827 14 group.cpp:341] Group process (zookeeper-group(2)@192.168.0.33:5050) connected to ZooKeeper I1003 08:41:29.868902 14 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I1003 08:41:29.868921 14 group.cpp:419] Trying to create path \u0026#39;/mesos\u0026#39; in ZooKeeper I1003 08:41:29.902194 14 contender.cpp:268] New candidate (id=\u0026#39;1\u0026#39;) has entered the contest for leadership I1003 08:41:29.904409 8 detector.cpp:152] Detected a new leader: (id=\u0026#39;1\u0026#39;) I1003 08:41:29.904513 8 group.cpp:700] Trying to get \u0026#39;/mesos/json.info_0000000001\u0026#39; in ZooKeeper I1003 08:41:29.912968 8 zookeeper.cpp:262] A new leading master (UPID=master@192.168.0.33:5050) is detected I1003 08:41:29.913091 8 master.cpp:2113] Elected as the leading master! I1003 08:41:29.913115 8 master.cpp:1668] Recovering from registrar I1003 08:41:29.916913 11 registrar.cpp:383] Successfully fetched the registry (0B) in 3.734016ms I1003 08:41:29.917054 11 registrar.cpp:487] Applied 1 operations in 26us; attempting to update the registry I1003 08:41:29.918582 11 registrar.cpp:544] Successfully updated the registry in 1.449984ms I1003 08:41:29.918630 11 registrar.cpp:416] Successfully recovered registrar I1003 08:41:29.918772 8 master.cpp:1782] Recovered 0 agents from the registry (121B); allowing 10mins for agents to reregister 访问IP:5050，对于PWD，可以访问http://SSHIP-5050.direct.labs.play-with-docker.com/#/，将SSH ip替换成实际提供的IP即可\nmesos从节点 node3 (slave-1)、node4(slave-2)\n启动脚本\n使用host通信脚本\nmkdir /cgroup docker run -d --net=host --privileged \\ -e MESOS_PORT=5051 \\ -e MESOS_MASTER=zk://192.168.0.32:2181/mesos \\ -e MESOS_SWITCH_USER=0 \\ -e MESOS_CONTAINERIZERS=docker,mesos \\ -e MESOS_LOG_DIR=/var/log/mesos \\ -e MESOS_WORK_DIR=/var/tmp/mesos \\ -v \u0026#34;$(pwd)/log/mesos:/var/log/mesos\u0026#34; \\ -v \u0026#34;$(pwd)/tmp/mesos:/var/tmp/mesos\u0026#34; \\ -v /var/run/docker.sock:/var/run/docker.sock \\ -v /cgroup:/cgroup \\ -v /sys:/sys \\ -v \u0026#34;$(which docker):/usr/local/bin/docker\u0026#34; \\ --name=mesos-slave \\ -p 5051:5051 \\ mesosphere/mesos-slave:1.7.0 \\ --no-systemd_enable_support 使用IP通信脚本，--ip需要改成实际IP\nmkdir /cgroup docker run -d --net=host --privileged \\ -e MESOS_PORT=5051 \\ -e MESOS_MASTER=zk://192.168.0.32:2181/mesos \\ -e MESOS_SWITCH_USER=0 \\ -e MESOS_CONTAINERIZERS=docker,mesos \\ -e MESOS_LOG_DIR=/var/log/mesos \\ -e MESOS_WORK_DIR=/var/tmp/mesos \\ -v \u0026#34;$(pwd)/log/mesos:/var/log/mesos\u0026#34; \\ -v \u0026#34;$(pwd)/tmp/mesos:/var/tmp/mesos\u0026#34; \\ -v /var/run/docker.sock:/var/run/docker.sock \\ -v /cgroup:/cgroup \\ -v /sys:/sys \\ -v \u0026#34;$(which docker):/usr/local/bin/docker\u0026#34; \\ --name=mesos-slave \\ -p 5051:5051 \\ mesosphere/mesos-slave:1.7.0 \\ --no-systemd_enable_support \\ --no-hostname_lookup \\ --ip=192.168.0.31 查看日志\ndocker logs mesos-slave 启动成功日志\nI1003 09:05:08.615039 12 group.cpp:341] Group process (zookeeper-group(1)@192.168.0.31:5051) connected to ZooKeeper I1003 09:05:08.615104 12 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I1003 09:05:08.615121 12 group.cpp:419] Trying to create path \u0026#39;/mesos\u0026#39; in ZooKeeper I1003 09:05:08.618291 13 state.cpp:66] Recovering state from \u0026#39;/var/tmp/mesos/meta\u0026#39; I1003 09:05:08.618405 15 slave.cpp:6930] Finished recovering checkpointed state from \u0026#39;/var/tmp/mesos/meta\u0026#39;, beginning agent recovery I1003 09:05:08.618475 8 task_status_update_manager.cpp:207] Recovering task status update manager I1003 09:05:08.618628 8 docker.cpp:908] Recovering Docker containers I1003 09:05:08.626359 13 detector.cpp:152] Detected a new leader: (id=\u0026#39;3\u0026#39;) I1003 09:05:08.626487 15 group.cpp:700] Trying to get \u0026#39;/mesos/json.info_0000000003\u0026#39; in ZooKeeper I1003 09:05:08.628664 15 zookeeper.cpp:262] A new leading master (UPID=master@192.168.0.33:5050) is detected I1003 09:05:08.997036 15 docker.cpp:921] Got the list of Docker containers I1003 09:05:08.997324 15 docker.cpp:1157] Finished processing orphaned Docker containers I1003 09:05:08.997436 8 slave.cpp:7159] Recovering executors I1003 09:05:08.997553 8 slave.cpp:7312] Finished recovery I1003 09:05:08.998044 11 task_status_update_manager.cpp:181] Pausing sending task status updates I1003 09:05:08.998045 8 slave.cpp:1275] New master detected at master@192.168.0.33:5050 I1003 09:05:08.998159 8 slave.cpp:1329] No credentials provided. Attempting to register without authentication I1003 09:05:08.998181 8 slave.cpp:1340] Detecting new master I1003 09:05:09.446756 11 slave.cpp:1500] Registered with master master@192.168.0.33:5050; given agent ID b597695d-6e37-418a-9879-4ffe46117e17-S0 I1003 09:05:09.449091 10 task_status_update_manager.cpp:188] Resuming sending task status updates 访问主节点web界面可看见资源邀约有效\n节点情况\nmarathon安装 node5 启动脚本\ndocker run -d --net=host \\ -p 8080:8080 \\ mesosphere/marathon:latest \\ --master zk://192.168.0.32:2181/mesos \\ --zk zk://192.168.0.32:2181/marathon 通过IP:8080查看运行结果\nmarathon-lb安装 node6\n启动脚本\ndocker run -d -e PORTS=9090 --net=host \\ -p 9090:9090 \\ mesosphere/marathon-lb sse \\ --group external \\ --marathon http://192.168.0.29:8080 访问IP:9090/haproxy?stats可查看启动状态\n","date":"2018-10-03T22:10:00+08:00","permalink":"/archives/185/","title":"Mesos集群安装"},{"content":"服务器信息 服务商 VPS为hiformance廉价VPS，OpenVZ虚拟化方式，可以通过virt-what命令查询，CentOS安装命令\nyum install -y virt-what 查询虚拟化方式\n[root@iYdQ3O-server ~]# virt-what openvz 硬件信息 内存：8G（连续买3年） CPU：2核 硬盘：20G SSD 流量：4T/月 带宽：1Gbps IPv4：1个 优惠价：8.99美元/年（QN网络） 另外补充一点，hiformance运营商比较迷的是，如果购买的是CN2网络，即亚洲优化网络，则有两个虚拟化同时并存的情况\n[root@BYr3tC-server ~]# virt-what openvz lxc 而两种虚拟化并存就导致其既不能装OpenVZ平台的BBR加速方式，又因为不是KVM虚拟化，没有系统内核，无法通过升级内核的方式安装KVM平台的BBR加速，所以目前建议购买hiformance的还是买QN网络比较好\n加速方式 目前（2018-09-15）OpenVZ平台上有5种可选方案\n参考链接\n方案简单分析 本人并没有尝试全部方案，从方案实现上大致分析下\nUML\n其Alpine linux 小内存版对低内存机器较为友好，但UML是装完 ssr 和 bbr 就已经运行了，不确定SSR账号的安全性，以及安装的SSR不清楚是否是原版，故有安全隐患\nLKL\nLKL是比较多人使用的解决方案，但亲自尝试了几次，发现并不容易安装成功，而且安装失败没有友好的报错提示\nRinetd\nRinetd原版通过全配置文件配置，使用Rinetd可以不用开启服务器的TUN/TAP功能，更为友好，但不支持端口号段配置\nNanQinLang\nNanQinLang有两个分支，一个是基于Rinetd修改版，一个是基于haproxy修改版，本次采用其Rinetd修改版，由于haproxy修改版虽然支持端口号段，并且加速也没问题，但存在haproxy经常挂掉的问题，而根据其作者在GitHub上的回复，换用 rinetd 没有这些毛病\nBBR安装 本次采用方案为NanQinLang基于Rinetd修改版，官方开源地址\n系统环境 系统版本为最新版\n[root@BYr3tC-server ~]# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) 安装 傻瓜式命令安装，中文引导\nwget https://github.com/tcp-nanqinlang/lkl-rinetd/releases/download/1.1.0/tcp_nanqinlang-rinetd-centos.sh bash tcp_nanqinlang-rinetd-centos.sh 安装好后可以通过配置文件增加端口，配置文件在目录/home/tcp_nanqinlang\n[root@iYdQ3O-server ~]# cd /home/tcp_nanqinlang [root@iYdQ3O-server tcp_nanqinlang]# ls config-port.conf config-rinetd.sh nohup.out rinetd rinetd.1 [root@iYdQ3O-server tcp_nanqinlang]# vi config-port.conf 端口配置文件config-port.conf\n0.0.0.0 8001 0.0.0.0 8001 0.0.0.0 8888 0.0.0.0 8888 0.0.0.0 8080 0.0.0.0 8080 0.0.0.0 8085 0.0.0.0 8085 0.0.0.0 8086 0.0.0.0 8086 0.0.0.0 8087 0.0.0.0 8087 0.0.0.0 8088 0.0.0.0 8088 0.0.0.0 8089 0.0.0.0 8089 0.0.0.0 9000 0.0.0.0 9000 速度测试 使用superbench进行网速测试\n使用前 如下可见，启用前国内的速度平均在4Mbps左右，十足的小水管，而国外基本有百兆以上的性能\n使用后 加速后提升明显，国内速度平均45Mbps，对于大多数家庭20M宽带来说已经稳定跑满\n","date":"2018-09-15T22:45:00+08:00","permalink":"/archives/183/","title":"OpenVZ安装BBR加速"},{"content":"测试命令 采用superbench\nwget -qO- --no-check-certificate https://raw.githubusercontent.com/oooldking/script/master/superbench.sh | bash OpenVZ 以手上的HiFormance VPS为例，superbench自动识别虚拟化平台和硬件信息，并对国内不同地点、不同运营商的上行、下行速度测试和延迟测试\nHiFormance 亚洲优化线路，由于HiFormance 亚洲优化线路虚拟化平台是OpenVZ+lxc，故无法安装BBR加速\nHiFormance 普通线路，启用了BBR加速\nKVM 以搬瓦工VPS为例，平台识别正常\n搬瓦工VPS，亚洲优化，启用BBR\n阿里云ECS 1Mbps带宽，识别正常\n物理机 识别平台为独立主机\n","date":"2018-09-13T13:00:22+08:00","permalink":"/archives/182/","title":"VPS性能测试"},{"content":"环境准备 参考前几篇文章配置后实现以下环境\n部署好Jenkins服务器一台(192.168.163.133),并在Jenkins服务器上部署ansible，且与目标主机(192.168.163.132)做了免密登录 可以访问的GitLab服务器一台，并注册了Jenkins相关账号且在Jenkins系统设置中配置完毕 一台工作环境主机，用于代码编写，该主机可以通过域名访问目标主机(192.168.163.132) 编写ansible playbook 搭建基础工程目录结构 从仓库下载之前编写的ansible工程目录结构 若没有也可自行创建，最终文件目录结构如下\n└─ansible-playbook-repo │ deploy.yml │ ├─inventory │ dev │ prod │ └─roles └─nginx ├─files │ │ health_check.sh │ │ index.html │ │ │ └─website │ ├─tasks │ main.yml │ └─templates nginx.conf.j2 文件编写 工程入口deploy.yml编写\n- hosts: \u0026#34;nginx\u0026#34; gather_facts: true remote_user: root roles: - nginx 编写开发环境配置文件inventory/dev，此处dev与prod配置一致，若实际需求可自行修改，用于最后jenkins参数化构建的选项 inventory/dev文件内容\n[nginx] test.fjy8018.top [nginx:vars] server_name=test.fjy8018.top port=80 user=ansible worker_processes=4 max_open_file=65505 root=/www inventory/prod文件内容\n[nginx] test.fjy8018.top [nginx:vars] server_name=test.fjy8018.top port=80 user=ansible worker_processes=4 max_open_file=65505 root=/www 编写健康检查shell脚本health_check.sh roles/nginx/files/health_check.sh文件内容\n#!/bin/bash # 接受命令行参数 URL=$1 # 访问目标地址，判断是否能正常访问并输出 curl -Is http://$URL \u0026gt; /dev/null \u0026amp;\u0026amp; echo \u0026#34;远程主机状态正常\u0026#34; || echo \u0026#34;远程主机已停止\u0026#34; 添加静态网站文件 /roles/nginx/files/website为静态网页模板目录，里面存放从网上下载的静态网站模板 配置nginx指向该资源 /roles/nginx/templates/nginx.conf.j2文件内容\n# For more information on configuration, see: user {{ user }}; worker_processes {{ worker_processes }}; error_log /var/log/nginx/error.log; pid /var/run/nginx.pid; events { worker_connections {{ max_open_file }}; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main \u0026#39;$remote_addr - $remote_user [$time_local] \u0026#34;$request\u0026#34; \u0026#39; \u0026#39;$status $body_bytes_sent \u0026#34;$http_referer\u0026#34; \u0026#39; \u0026#39;\u0026#34;$http_user_agent\u0026#34; \u0026#34;$http_x_forwarded_for\u0026#34;\u0026#39;; access_log /var/log/nginx/access.log main; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; #gzip on; # Load config files from the /etc/nginx/conf.d directory # The default server is in conf.d/default.conf #include /etc/nginx/conf.d/*.conf; server { listen {{ port }} default_server; server_name {{ server_name }}; #charset koi8-r; #access_log logs/host.access.log main; location / { root {{ root }}/website; index index.html index.htm; } error_page 404 /404.html; location = /404.html { root /usr/share/nginx/html; } # redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/share/nginx/html; } } } 编写主任务文件 主任务文件main.yml内容如下\n- name: 关闭系统防火墙 service: name=firewalld state=stopped - name: 关闭强制访问控制策略 selinux: state=disabled - name: 设置yum源 yum: pkg=epel-release state=latest - name: 写入nginx配置文件 template: src=roles/nginx/templates/nginx.conf.j2 dest=/etc/nginx/nginx.conf - name: 创建nginx根目录文件夹 file: \u0026#39;path={{ root }} state=directory owner={{ user }} group={{ user }} mode=0755\u0026#39; - name: 文件拷贝 copy: \u0026#39;remote_src=no src=roles/nginx/files/website dest=/www/website owner={{ user }} mode=0755\u0026#39; - name: 启动nginx service: name=nginx state=restarted - name: 运行健康检查 shell: \u0026#34;sh roles/nginx/files/health_check.sh {{ server_name }}\u0026#34; # 本地执行 delegate_to: localhost # 参数传递 register: health_status - debug: msg=\u0026#34;{{ health_status.stdout }}\u0026#34; Git提交 提交变更到版本仓库GitLab\ngit add . git commit -m \u0026#34;添加模板文件\u0026#34; git push origin master Jenkins创建并执行 创建自由风格构建任务，添加参数化构建、添加选项参数和文本参数 添加Git仓库地址和账号 添加Shell构建，Shell内容为\n#!/bin/bash # 关闭命令行扩展环境，避免环境参数日志干扰 set +x # 加载虚拟环境 source /home/deploy/.py3-a2.5-env/bin/activate # 加载ansible到当前用户目录 source /home/deploy/.py3-a2.5-env/ansible/hacking/env-setup -q cd $WORKSPACE ls -a # 查看版本 ansible --version ansible-playbook --version # 引入资源清单，测试远程命令 ansible-playbook -i inventory/$deploy_env ./deploy.yml -e project=nginx -e branch=$branch -e env=$deploy_env 保存并构建\n结果 构建成功 访问测试 在工作主机上访问目标地址+资源路径即可，访问通过 后续构建 后期只要在工作主机变更代码推送到git仓库，再到Jenkins的web界面点击参数化构建即可让Jenkins自动拉取最新分支并执行Ansible脚本\n","date":"2018-09-01T12:45:00+08:00","permalink":"/archives/181/","title":"Jenkins+GitLab+Ansible实现静态网站自动部署"},{"content":"环境准备 安装jdk yum -y install java-devel java 检测安装结果\n[root@localhost maven-freestyle-job]# javac -version javac 1.8.0_181 [root@localhost maven-freestyle-job]# java -version openjdk version \u0026#34;1.8.0_181\u0026#34; OpenJDK Runtime Environment (build 1.8.0_181-b13) OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode) 安装maven环境 wget http://apache.osuosl.org/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz 解压，并指定解压路径\ntar -zxvf apache-maven-3.5.4-bin.tar.gz -C /opt 查看安装结果\n[root@localhost ~]# cd /opt/ [root@localhost opt]# cd apache-maven-3.5.4/ [root@localhost apache-maven-3.5.4]# cd bin/ [root@localhost bin]# ./mvn -v Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00) Maven home: /opt/apache-maven-3.5.4 Java version: 1.8.0_181, vendor: Oracle Corporation, runtime: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: \u0026#34;linux\u0026#34;, version: \u0026#34;3.10.0-862.el7.x86_64\u0026#34;, arch: \u0026#34;amd64\u0026#34;, family: \u0026#34;unix\u0026#34; [root@localhost bin]# pwd /opt/apache-maven-3.5.4/bin 配置环境 进入Jenkins系统管理中的全局工具配置，配置本地的maven地址 此处的jdk报错正常，若不配置则会自动调用系统环境里的jdk 指定maven名称和路径 准备maven工程 提前在gitlab中装备一个简单的Springboot web工程，通过springboot脚手架快速生成，pom文件如下 新建构建 新建构建并填写目标仓库地址 添加构建设置为maven构建 添加构建参数，即在命令行mvn之后的参数 选择maven版本 构建 构建成功 ","date":"2018-08-26T17:15:00+08:00","permalink":"/archives/180/","title":"Jenkins 与Maven集成"},{"content":"系统预配置 关闭防止跨站请求伪造 测试期间系统设置中临时关闭防止跨站请求伪造，否则容易出现登录超时无法新建任务 添加远程仓库配置 GitLab的用户邮箱、用户名、密码，用于拉取远程仓库代码 凭据 添加凭据，凭据为用户的标识 选择全局凭据 添加凭据 添加为Git用户 Jenkins job freestyle Job freestyle Job简介 需要在页面添加模块配置项与参数完成配置 每个Job仅能实现一个开发功能 无法将配置代码化，不利于Job配置迁移与版本控制 没有版本日志记录 逻辑简单 新建freestyle Job 新建任务 添加选项参数 添加文本参数 指定远程仓库地址 添加Shell脚本 保存后选择通过参数构建 构建报错 检查后发现Shell脚本开头的#!/bin/bash写成了#! /bin/bash，中间包含空格后系统无法识别\n修正错误后重新构建 报语法错误，12行缺少空格 检查发现if [ -s test.properties ]写成了if [ -s test.properties]，修正错误后再次构建 修正后的Shell脚本为\n#!/bin/bash # export PATH=\u0026#34;/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin\u0026#34; # 打印环境变量 echo \u0026#34;[INFO] 打印环境变量\u0026#34; echo \u0026#34;[INFO] 当前发布环境为 $deploy_env\u0026#34; \u0026gt;\u0026gt; test.properties echo \u0026#34;[INFO] 构建版本为 $version \u0026#34; \u0026gt;\u0026gt; test.properties echo \u0026#34;[INFO] 打印环境变量完成...\u0026#34; # 检查test.properties echo \u0026#34;[INFO] 检查test.properties\u0026#34; if [ -s test.properties ] then cat test.properties echo \u0026#34;[INFO] 检查完成...\u0026#34; else echo \u0026#34;[WARN] test.properties为空\u0026#34; fi echo \u0026#34;[INFO] 构建完成...\u0026#34; 构建成功，输出正常 查看构建历史 同时显示构建稳定性 pipeline job pipeline job简介 所有代码包裹在pipeline{}内 stages{}用来包含pipeline所有子层 stage(管道名称){}每个子层可以执行一个任务，并不互相影响 steps{}用来添加具体需要调用的模块语句 新建pipeline任务 编写pipeline 脚本 #!groovy pipeline { // agent 定义pipeline在何处运行，使用any、none或者Jenkins node主机名 agent {node {label \u0026#39;master\u0026#39;}} // 声明环境变量，environment 定义系统环境变量，可以应用于所有stages // 定义局部环境变量则于steps平级即可 environment { PATH=\u0026#34;/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin\u0026#34; } parameters { // 声明可选参数 choice( // 可选参数为两个，以空行分隔 choices: \u0026#39;dev\\nprod\u0026#39;, // 设置参数描述 description: \u0026#39;选择发布环境\u0026#39;, // 声明参数变量名 name: \u0026#39;deploy_env\u0026#39; ) // 声明文本参数，和默认值 string (name: \u0026#39;version\u0026#39;, defaultValue: \u0026#39;0.0.1\u0026#39;,description: \u0026#39;构建版本号\u0026#39;) } stages { // 执行一个任务 stage(\u0026#34;切换到测试版本仓库\u0026#34;) { // 添加具体需要调用的模块语句 steps { // 关闭Git ssl安全认证 sh \u0026#39;git config --system http.sslVerify false\u0026#39; // 指定工作空间 dir (\u0026#34;${env.WORKSPACE}\u0026#34;) { // 指定分支和Jenkins凭据ID git branch: \u0026#39;master\u0026#39;, credentialsId: \u0026#39;52f4c34f-544c-4830-828d-6691d603b8e4\u0026#39;, url: \u0026#39;http://home.fjy8018.top:8099/jenkins/test-repo.git\u0026#39; } } } stage(\u0026#34;打印环境变量\u0026#34;) { steps { dir (\u0026#34;${env.WORKSPACE}\u0026#34;) { sh \u0026#34;\u0026#34;\u0026#34; echo \u0026#34;[INFO] 打印环境变量\u0026#34; echo \u0026#34;[INFO] 当前发布环境为 $deploy_env\u0026#34; \u0026gt;\u0026gt; test.properties echo \u0026#34;[INFO] 构建版本为 $version \u0026#34; \u0026gt;\u0026gt; test.properties echo \u0026#34;[INFO] 打印环境变量完成...\u0026#34; \u0026#34;\u0026#34;\u0026#34; } } } stage(\u0026#34;检查test.properties\u0026#34;) { steps { dir (\u0026#34;${env.WORKSPACE}\u0026#34;) { sh \u0026#34;\u0026#34;\u0026#34; echo \u0026#34;[INFO] 检查test.properties\u0026#34; if [ -s test.properties ] then cat test.properties echo \u0026#34;[INFO] 检查完成...\u0026#34; else echo \u0026#34;[WARN] test.properties为空\u0026#34; fi \u0026#34;\u0026#34;\u0026#34; echo \u0026#34;[INFO] 构建完成...\u0026#34; } } } } } 其中credentialsId来自用户的凭据唯一标识，可在凭据页面查看 写入Shell脚本 执行报错，error: could not lock config file /etc/gitconfig: ???? 由于使用root用户执行同样命令不会报错，故猜测没有权限，修改语法为git config --global http.sslVerify false即可 重新构建 日志符合预期 freestyle shell Job 新建任务 写入shell脚本如下\n#!/bin/bash user=`whoami` if [ $user == \u0026#39;deploy\u0026#39; ]; then echo \u0026#34;成功，当前用户为 $user\u0026#34; else echo \u0026#34;失败，当前用户不是 $user\u0026#34; fi # 查看主机ip ip addr # 查看系统版本 cat /etc/system-release # 查看内存状态 free -m # 查看磁盘状态 df -h # 将python安装位置赋值给变量 py_cmd=`which python` # 查看python版本号 $py_cmd --version 运行并查看日志，符合预期 参数任务 新建参数化构建任务 添加选项参数等 添加shell构建参数 构建 输出符合预期 ","date":"2018-08-26T12:00:00+08:00","permalink":"/archives/178/","title":"Jenkins job编写测试"},{"content":"使用Docker安装 基础环境 docker环境 查看docker版本\n[root@localhost ~]# docker -v Docker version 18.06.0-ce, build 0ffa825 新建用户目录，用户挂载容器存储\nuseradd jenkins-docker 运行docker命令\ndocker run -p 8081:8080 -p 50001:50000 -v /home/jenkins-docker:/var/jenkins_home hub.c.163.com/library/jenkins:latest 命令中没有使用后台启动为了查看输出的管理员密码 若使用了挂载目录也可以在挂载的目录下查看 访问web界面，输入密码即可 使用原生系统安装 基础环境 CentOS7 1804最小安装\n安装基础wget工具yum install -y wget 导入仓库源 wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins.io/redhat-stable/jenkins.repo 下载并导入仓库的key，用于验证仓库安全性\nrpm --import http://pkg.jenkins.io/redhat-stable/jenkins.io.key 安装java8 yum -y install java 验证java版本\n[root@localhost ~]# java -version openjdk version \u0026#34;1.8.0_181\u0026#34; OpenJDK Runtime Environment (build 1.8.0_181-b13) OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode) 关闭防火墙 [root@localhost ~]# systemctl stop firewalld 关闭防火墙开机启动\n[root@localhost ~]# systemctl disable firewalld Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. 关闭强制访问控制安全策略 vi /etc/sysconfig/selinux # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=enforcing # SELINUXTYPE= can take one of three two values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted SELINUX=enforcing 改成\nSELINUX=disabled reboot使禁用操作生效\n查看操作是否生效\n[root@localhost ~]# getenforce Disabled 安装jenkins 直接安装 yum -y install jenkins 安装后会自动添加jenkins，此时可以自定义发布用户\n[root@localhost ~]# useradd jenkins useradd：用户“jenkins”已存在 [root@localhost ~]# useradd deploy 配置 编辑配置文件\n[root@localhost ~]# vi /etc/sysconfig/jenkins 将用户改为新建的用户JENKINS_USER=\u0026quot;deploy\u0026quot;，端口号默认为8080 修改log目录和权限\n[root@localhost ~]# chown -R deploy:deploy /var/lib/jenkins [root@localhost ~]# chown -R deploy:deploy /var/log/jenkins [root@localhost ~]# chown -R deploy:deploy /var/cache/jenkins 启动jenkins [root@localhost ~]# systemctl start jenkins 查看服务是否启动\n[root@localhost ~]# ps -ef | grep jenkins root 1445 1311 0 19:06 pts/0 00:00:00 grep --color=auto jenkins 或者使用lsof（需要额外安装）\n[root@localhost jenkins]# lsof -i:8080 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 1683 deploy 161u IPv6 24353 0t0 TCP *:webcache (LISTEN) 访问web界面 地址为ip:8080 去提示的目录下查找密码 安装推荐配置插件 新建管理员用户 安装成功 ","date":"2018-08-26T11:32:00+08:00","permalink":"/archives/179/","title":"Jenkins 安装配置"},{"content":"环境准备 两台CentOS 7虚拟机，主IP为192.168.163.128，并在主IP虚拟机上安装了ansible2.5版本，目标IP为192.168.163.132\nAnsible编写 切换用户\n[root@localhost ~]# su - ansible 上一次登录：六 8月 18 20:12:23 CST 2018pts/0 上 进入虚拟环境\n[ansible@localhost ~]$ source .py3-a2.5-env/bin/activate (.py3-a2.5-env) [ansible@localhost ~]$ 加载ansible到当前用户目录\nsource .py3-a2.5-env/ansible/hacking/env-setup -q 验证版本\n(.py3-a2.5-env) [ansible@localhost ~]$ ansible-playbook --version ansible-playbook 2.5.8 (stable-2.5 b23b1c2ee0) last updated 2018/08/18 20:35:45 (GMT +800) config file = None configured module search path = [\u0026#39;/home/ansible/.ansible/plugins/modules\u0026#39;, \u0026#39;/usr/share/ansible/plugins/modules\u0026#39;] ansible python module location = /home/ansible/.py3-a2.5-env/ansible/lib/ansible executable location = /home/ansible/.py3-a2.5-env/ansible/bin/ansible-playbook python version = 3.7.0 (default, Aug 18 2018, 20:09:08) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] (.py3-a2.5-env) [ansible@localhost ~]$ ll 总用量 0 (.py3-a2.5-env) [ansible@localhost ~]$ pwd /home/ansible 配置目标主机的秘钥认证 (.py3-a2.5-env) [ansible@localhost ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/ansible/.ssh/id_rsa): Created directory \u0026#39;/home/ansible/.ssh\u0026#39;. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/ansible/.ssh/id_rsa. Your public key has been saved in /home/ansible/.ssh/id_rsa.pub. The key fingerprint is: SHA256:pXMJqz9DPkjt6nKws35NYoMQeUdfu4CGATw76tr9lJY ansible@localhost.localdomain The key\u0026#39;s randomart image is: +---[RSA 2048]----+ | ..o... . | | = .o.o . . | | =..o + o | | + . * o | | . o . .S + | |. o =++o | |. =EX | | o .+o+.B | |o ..+O+..+ | +----[SHA256]-----+ 拷贝公钥\n(.py3-a2.5-env) [ansible@localhost ~]$ ssh-copy-id -i /home/ansible/.ssh/id_rsa.pub root@test.fjy8018.top /bin/ssh-copy-id: INFO: Source of key(s) to be installed: \u0026#34;/home/ansible/.ssh/id_rsa.pub\u0026#34; The authenticity of host \u0026#39;test.fjy8018.top (192.168.163.132)\u0026#39; can\u0026#39;t be established. ECDSA key fingerprint is SHA256:3UEr3lx2jznmN3FL3SflViR05IZe6AWweb3TeSsfX0M. ECDSA key fingerprint is MD5:0f:31:c1:cc:54:69:7f:d8:8d:7c:8c:22:95:2e:03:4e. Are you sure you want to continue connecting (yes/no)? yes /bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@test.fjy8018.top\u0026#39;s password: Number of key(s) added: 1 Now try logging into the machine, with: \u0026#34;ssh \u0026#39;root@test.fjy8018.top\u0026#39;\u0026#34; and check to make sure that only the key(s) you wanted were added. 测试免密登录\n(.py3-a2.5-env) [ansible@localhost ~]$ ssh root@test.fjy8018.top Last login: Sat Aug 25 11:27:35 2018 from 192.168.163.1 [root@localhost ~]# exit 登出 Connection to test.fjy8018.top closed. 编写环境变量 预期的文件结构为\n├── deploy.yml ├── inventory │ └── testenv └── roles └── testbox └── tasks └── main.yml deploy.yml为工程入口，testenv为环境变量文件，tasks为任务文件夹，其中的main.yml为要执行的任务 新建playbook工程目录，并根据ansible规范创建对应的文件夹\n(.py3-a2.5-env) [ansible@localhost ~]$ mkdir test_playbook (.py3-a2.5-env) [ansible@localhost ~]$ cd test_playbook/ (.py3-a2.5-env) [ansible@localhost test_playbook]$ mkdir inventory (.py3-a2.5-env) [ansible@localhost test_playbook]$ mkdir roles (.py3-a2.5-env) [ansible@localhost test_playbook]$ cd inventory/ (.py3-a2.5-env) [ansible@localhost inventory]$ vi testenv testenv文件内容如下\n[testservers] test.fjy8018.top [testservers:vars] server_name=test.fjy8018.top user=root output=/root/test.txt 远程执行命令并加载环境变量的值 (.py3-a2.5-env) [ansible@localhost test_playbook]$ ll 总用量 0 drwxrwxr-x 2 ansible ansible 21 8月 18 20:49 inventory drwxrwxr-x 2 ansible ansible 6 8月 18 20:47 roles (.py3-a2.5-env) [ansible@localhost test_playbook]$ cd roles/ (.py3-a2.5-env) [ansible@localhost roles]$ mkdir testbox (.py3-a2.5-env) [ansible@localhost roles]$ cd testbox/ (.py3-a2.5-env) [ansible@localhost testbox]$ mkdir tasks (.py3-a2.5-env) [ansible@localhost testbox]$ cd tasks/ (.py3-a2.5-env) [ansible@localhost tasks]$ vi main.yml main.yml文件内容如下\n- name: Print server name and user to remote testbox shell: \u0026#34;echo \u0026#39;Currently {{ user }} is loggining {{ server_name }} \u0026#39; \u0026gt; {{ output }} \u0026#34; 编写工程入口文件\n(.py3-a2.5-env) [ansible@localhost test_playbook]$ pwd /home/ansible/test_playbook (.py3-a2.5-env) [ansible@localhost test_playbook]$ ll 总用量 0 drwxrwxr-x 2 ansible ansible 21 8月 18 20:49 inventory drwxrwxr-x 3 ansible ansible 21 8月 18 20:51 roles (.py3-a2.5-env) [ansible@localhost test_playbook]$ vi deploy.yml deploy.yml文件内容如下\n# 调用testenv里的servers标签定义的目标主机 - hosts: \u0026#34;testservers\u0026#34; # 用于获取目标主机的信息 gather_facts: true # 使用root用户操作 remote_user: root # 指定进入roles目录下的testbox任务文件执行任务 roles: - testbox 查看当前目录结构，符合预期\n[ansible@localhost test_playbook]$ tree . . ├── deploy.yml ├── inventory │ └── testenv └── roles └── testbox └── tasks └── main.yml 4 directories, 3 files 执行任务\n(.py3-a2.5-env) [ansible@localhost ~]$ cd test_playbook/ (.py3-a2.5-env) [ansible@localhost test_playbook]$ ls deploy.yml inventory roles (.py3-a2.5-env) [ansible@localhost test_playbook]$ ansible-playbook -i inventory/testenv ./deploy.yml 部署完成 登录目标主机查看 远程文件创建 编辑任务文件\n(.py3-a2.5-env) [ansible@localhost test_playbook]$ vi roles/testbox/tasks/main.yml 修改为在远程创建文件，指定路径并赋予权限为755，所有者为foo用户，属于foo用户组，故添加如下内容\n- name: create a file file: \u0026#39;path=/root/foo.txt state=touch mode=0755 owner=foo group=foo\u0026#39; 执行文件 执行成功，登录目标主机查看是否创建 远程文件拷贝 新建存放文件目录\n(.py3-a2.5-env) [ansible@localhost test_playbook]$ mkdir roles/testbox/files (.py3-a2.5-env) [ansible@localhost test_playbook]$ tree . . ├── deploy.yml ├── inventory │ └── testenv └── roles └── testbox ├── files └── tasks └── main.yml 5 directories, 3 files 新建目标文件foo.sh\n(.py3-a2.5-env) [ansible@localhost test_playbook]$ vi roles/testbox/files/foo.sh 文件内容如下\necho \u0026#34;This is a test script for ansible\u0026#34; 编辑任务文件\n(.py3-a2.5-env) [ansible@localhost test_playbook]$ vi roles/testbox/tasks/main.yml remote_src=no定义为文件传输，并设置644权限，指定强制执行，故添加任务如下\n- name: copy a file copy: \u0026#39;remote_src=no src=roles/testbox/files/foo.sh dest=/root/foo.sh mode=0644 force=yes\u0026#39; 执行成功 判断文件是否存在 在roles/testbox/tasks/main.yml添加任务如下\n# 任务名称 - name: check if foo.sh exists stat: \u0026#39;path=/root/foo.sh\u0026#39; register: script_stat # 若when中返回信息，则输出msg后的值 - debug: msg=\u0026#34;foo.sh exists\u0026#34; # 调用模块任务 when: script_stat.stat.exists 执行，成功输出信息 远程执行命令 在roles/testbox/tasks/main.yml添加任务如下\n- name: run the script command: \u0026#39;sh /root/foo.sh\u0026#39; 执行成功 template模块任务 环境变量配置文件中inventory/testenv添加如下命令\nserver_name=test.fjy8018.top port=80 user=ansible worker_processes=4 max_open_file=65505 root=/www 新建template文件夹\n(.py3-a2.5-env) [ansible@localhost test_playbook]$ mkdir roles/testbox/templates 添加nginx j2文件\n(.py3-a2.5-env) [ansible@localhost test_playbook]$ vi roles/testbox/templates/nginx.conf.j2 文件内容如下，其中调用了环境变量中的配置\n# For more information on configuration, see: user {{ user }}; worker_processes {{ worker_processes }}; error_log /var/log/nginx/error.log; pid /var/run/nginx.pid; events { worker_connections {{ max_open_file }}; } ... 在roles/testbox/tasks/main.yml添加任务如下\n# 复制nginx配置文件 - name: write the nginx config file template: src=roles/testbox/templates/nginx.conf.j2 dest=/etc/nginx/nginx.conf # 远程安装最新版nginx - name: install nginx at latest version yum: pkg=nginx state=latest # 新增模块任务启动nginx服务 - name: start nginx service service: name=nginx state=started 所有任务列表 查看变量参数是否应用成功 ssh root@test.fjy8018.top ps -ef | grep nginx查看nginx是否启动成功 ","date":"2018-08-25T16:19:00+08:00","permalink":"/archives/177/","title":"Ansible Playbook编写测试"},{"content":"系统预配置 关闭防火墙\n[root@localhost ~]# systemctl stop firewalld 关闭防火墙开机启动\n[root@localhost ~]# systemctl disable firewalld Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. 关闭强制访问控制安全策略\nvi /etc/sysconfig/selinux # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=enforcing # SELINUXTYPE= can take one of three two values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted SELINUX=enforcing改成SELINUX=disabled\nreboot使禁用操作生效\n查看操作是否生效\n[root@localhost ~]# getenforce Disabled 安装Gitlab依赖 yum install curl policycoreutils openssh-server openssh-clients postfixs [root@localhost ~]# yum install curl policycoreutils openssh-server openssh-clients postfixs 已加载插件：fastestmirror Determining fastest mirrors * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com base | 3.6 kB 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 (1/4): base/7/x86_64/group_gz | 166 kB 00:00:00 (2/4): extras/7/x86_64/primary_db | 174 kB 00:00:00 (3/4): base/7/x86_64/primary_db | 5.9 MB 00:00:01 (4/4): updates/7/x86_64/primary_db | 5.0 MB 00:00:02 软件包 curl-7.29.0-46.el7.x86_64 已安装并且是最新版本 软件包 policycoreutils-2.5-22.el7.x86_64 已安装并且是最新版本 软件包 openssh-server-7.4p1-16.el7.x86_64 已安装并且是最新版本 软件包 openssh-clients-7.4p1-16.el7.x86_64 已安装并且是最新版本 没有可用软件包 postfixs。 无须任何处理 下载gitlab仓库源\ncurl -sS https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.rpm.sh | sudo bash 启动邮件服务 [root@localhost ~]# systemctl start postfix 配置开机启动\n[root@localhost ~]# systemctl enable postfix 安装gitlab yum -y install gitlab-ce 配置SSL证书 mkdir -p /etc/gitlab/ssl 创建本地私有秘钥 [root@localhost ~]# openssl genrsa -out \u0026quot;/etc/gitlab/ssl/gitlab.fjy8018.top.key\u0026quot; 2048 Generating RSA private key, 2048 bit long modulus ..................................................+++ ......................+++ e is 65537 (0x10001) 创建CSR证书 [root@localhost ~]# openssl req -new -key \u0026quot;/etc/gitlab/ssl/gitlab.fjy8018.top.key\u0026quot; -out \u0026quot;/etc/gitlab/ssl/gitlab.fjy8018.top.csr\u0026quot; You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:cn State or Province Name (full name) []:gd Locality Name (eg, city) [Default City]:gz Organization Name (eg, company) [Default Company Ltd]: Organizational Unit Name (eg, section) []: Common Name (eg, your name or your server's hostname) []:gitlab.fjy8018.top Email Address []:admin@example.com Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []:12345678 An optional company name []: 查看证书是否成功创建\n[root@localhost ~]# cd /etc/gitlab/ssl [root@localhost ssl]# ll 总用量 8 -rw-r--r-- 1 root root 1070 8月 18 11:11 gitlab.fjy8018.top.csr -rw-r--r-- 1 root root 1675 8月 18 11:08 gitlab.fjy8018.top.key 创建签署证书 利用秘钥和csr证书创建签署证书 X.509为证书格式 365为有效期\n[root@localhost ssl]# openssl x509 -req -days 365 -in \u0026quot;/etc/gitlab/ssl/gitlab.fjy8018.top.csr\u0026quot; -signkey \u0026quot;/etc/gitlab/ssl/gitlab.fjy8018.top.key\u0026quot; -out \u0026quot;/etc/gitlab/ssl/gitlab.fjy8018.top.crt\u0026quot; Signature ok subject=/C=cn/ST=gd/L=gz/O= /OU= /CN=gitlab.fjy8018.top/emailAddress=admin@example.com Getting Private key 查看证书目录\n[root@localhost ssl]# ll 总用量 12 -rw-r--r-- 1 root root 1265 8月 18 11:14 gitlab.fjy8018.top.crt -rw-r--r-- 1 root root 1070 8月 18 11:11 gitlab.fjy8018.top.csr -rw-r--r-- 1 root root 1675 8月 18 11:08 gitlab.fjy8018.top.key 输出pem证书 openssl dhparam -out /etc/gitlab/ssl/dhparams.pem 2048 [root@localhost ssl]# ll 总用量 16 -rw-r--r-- 1 root root 424 8月 18 11:17 dhparams.pem -rw-r--r-- 1 root root 1265 8月 18 11:14 gitlab.fjy8018.top.crt -rw-r--r-- 1 root root 1070 8月 18 11:11 gitlab.fjy8018.top.csr -rw-r--r-- 1 root root 1675 8月 18 11:08 gitlab.fjy8018.top.key 证书创建完成 修改证书权限\n[root@localhost ssl]# chmod 700 * 查看权限\n[root@localhost ssl]# ll 总用量 16 -rw------- 1 root root 424 8月 18 11:17 dhparams.pem -rw------- 1 root root 1265 8月 18 11:14 gitlab.fjy8018.top.crt -rw------- 1 root root 1070 8月 18 11:11 gitlab.fjy8018.top.csr -rw------- 1 root root 1675 8月 18 11:08 gitlab.fjy8018.top.key 配置gitlab证书文件 vi /etc/gitlab/gitlab.rb 修改external_url为https的域名\nexternal_url 'https://gitlab.fjy8018.top' 检索重定向配置，启用重定向 # nginx['client_max_body_size'] = '250m' nginx['redirect_http_to_https'] = true # nginx['redirect_http_to_https_port'] = 80 修改下方crt配置 原始为# nginx['ssl_certificate'] = \u0026quot;/etc/gitlab/ssl/#{node['fqdn']}.crt\u0026quot; 修改为nginx['ssl_certificate'] = \u0026quot;/etc/gitlab/ssl/gitlab.fjy8018.top.crt\u0026quot; key、ssl_dhparam同样配置\nnginx['ssl_certificate_key'] = \u0026quot;/etc/gitlab/ssl/gitlab.fjy8018.top.key\u0026quot; nginx['ssl_dhparam'] = \u0026quot;/etc/gitlab/ssl/dhparams.pem\u0026quot; # Path to dhparams.pem, eg. /etc/gitlab/ssl/dhparams.pem 初始化配置文件\ngitlab-ctl reconfigure 修改gitlab的nginx配置文件\nvi /var/opt/gitlab/nginx/conf/gitlab-http.conf 搜索server_name\nserver_name gitlab.fjy8018.top; 并在下方添加重定向HTTP请求\nrewrite ^(.*)$ https://$hosst1 permanent; 使nginx配置生效\n[root@localhost ssl]# gitlab-ctl restart 所有服务启动成功 访问成功 ","date":"2018-08-19T16:39:00+08:00","permalink":"/archives/176/","title":"gitlab-ce安装配置"},{"content":"ansible官方文档指南\n系统预配置 关闭防火墙\n[root@localhost ~]# systemctl stop firewalld 关闭防火墙开机启动\n[root@localhost ~]# systemctl disable firewalld Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. 关闭强制访问控制安全策略\nvi /etc/sysconfig/selinux # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=enforcing # SELINUXTYPE= can take one of three two values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted SELINUX=enforcing改成SELINUX=disabled\nreboot使禁用操作生效\n查看操作是否生效\n[root@localhost ~]# getenforce Disabled 安装Python3 wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz 解压编译包\ntar xf Python-3.7.0.tar.xz 查看文件\n[root@localhost ~]# ls anaconda-ks.cfg Python-3.7.0 Python-3.7.0.tar.xz [root@localhost ~]# cd Python-3.7.0 安装OpenSSL依赖，否则使用pip3.7会报找不到SSL模块错误\nyum install openssl-devel 设定编译参数，/usr/local为安装位置，\u0026ndash;with-ensurepip=install指定安装包管理工具，\u0026ndash;enable-shared配置Python匹配系统参数\n[root@localhost Python-3.7.0]# ./configure --prefix=/usr/local --with-ensurepip=install --enable-shared LDFLAGS=\u0026quot;-Wl,-rpath /usr/local/lib\u0026quot; --with-ssl checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-pc-linux-gnu checking for python3.7... no checking for python3... no checking for python... python checking for --enable-universalsdk... no checking for --with-universal-archs... no checking MACHDEP... checking for --without-gcc... no checking for --with-icc... no checking for gcc... no checking for cc... no checking for cl.exe... no configure: error: in `/root/Python-3.7.0': configure: error: no acceptable C compiler found in $PATH See `config.log' for more details 报错为找不到C编译器，由于系统为最小化安装，没有自带gcc编译器 安装gcc\nyum install gcc 再次编译，编译通过 安装编译文件\nmake \u0026amp;\u0026amp; make altinstall 日志如下\nTraceback (most recent call last): File \u0026quot;/root/Python-3.7.0/Lib/runpy.py\u0026quot;, line 193, in _run_module_as_main \u0026quot;__main__\u0026quot;, mod_spec) File \u0026quot;/root/Python-3.7.0/Lib/runpy.py\u0026quot;, line 85, in _run_code exec(code, run_globals) File \u0026quot;/root/Python-3.7.0/Lib/ensurepip/__main__.py\u0026quot;, line 5, in \u0026lt;module\u0026gt; sys.exit(ensurepip._main()) File \u0026quot;/root/Python-3.7.0/Lib/ensurepip/__init__.py\u0026quot;, line 204, in _main default_pip=args.default_pip, File \u0026quot;/root/Python-3.7.0/Lib/ensurepip/__init__.py\u0026quot;, line 117, in _bootstrap return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths) File \u0026quot;/root/Python-3.7.0/Lib/ensurepip/__init__.py\u0026quot;, line 27, in _run_pip import pip._internal zipimport.ZipImportError: can't decompress data; zlib not available make: *** [altinstall] 错误 1 编译报错，找不到zlib，同理，安装zlib\nyum -y install zlib* 另外，此时安装的是3.7版本，3.7版本需要一个新的包libffi-devel否则也会编译报错\nyum install libffi-devel -y 安装成功\nPython3.7对应的包管理工具pip版本也为3.7，查看其路径\n[root@localhost Python-3.7.0]# which pip3.7 /usr/local/bin/pip3.7 设置软链接\nln -s /usr/local/bin/pip3.7 /usr/local/bin/pip 安装virtualenv 安装virtualenv，用于隔离Python环境\npip install virtualenv 安装git工具及其依赖\nyum -y install git nss curl 创建ansible账户\nuseradd ansible 切换用户\n[root@localhost Python-3.7.0]# su - ansible [ansible@localhost ~]$ 在ansible用户下创建virtualenv实例，用于集成ansible2.5\nvirtualenv -p /usr/local/bin/python3.7 .py3-a2.5-env 源码安装git-ansible2.5\ncd /home/ansible/.py3-a2.5-env 下载源码\ngit clone https://github.com/ansible/ansible.git 加载virtualenv环境\n[ansible@localhost .py3-a2.5-env]$ source /home/ansible/.py3-a2.5-env/bin/activate (.py3-a2.5-env) [ansible@localhost .py3-a2.5-env]$ 安装ansible依赖\npip install paramiko PyYAML jinja2 查看文件\n(.py3-a2.5-env) [ansible@localhost .py3-a2.5-env]$ ll 总用量 8 drwxrwxr-x 14 ansible ansible 4096 8月 18 20:24 ansible drwxrwxr-x 2 ansible ansible 248 8月 18 20:16 bin drwxrwxr-x 2 ansible ansible 24 8月 18 20:13 include drwxrwxr-x 3 ansible ansible 23 8月 18 20:13 lib -rw-rw-r-- 1 ansible ansible 59 8月 18 20:16 pip-selfcheck.json 移动源码至虚拟环境\nmv ansible/ .py3-a2.5-env/ 进入虚拟环境源码目录\n(.py3-a2.5-env) [ansible@localhost .py3-a2.5-env]$ cd .py3-a2.5-env/ (.py3-a2.5-env) [ansible@localhost .py3-a2.5-env]$ ll 总用量 108 drwxrwxr-x 2 ansible ansible 223 8月 18 20:24 bin drwxrwxr-x 3 ansible ansible 63 8月 18 20:24 changelogs -rw-rw-r-- 1 ansible ansible 10591 8月 18 20:24 CODING_GUIDELINES.md drwxrwxr-x 4 ansible ansible 53 8月 18 20:24 contrib -rw-rw-r-- 1 ansible ansible 35148 8月 18 20:24 COPYING drwxrwxr-x 7 ansible ansible 71 8月 18 20:24 docs -rw-rw-r-- 1 ansible ansible 48 8月 18 20:24 docsite_requirements.txt drwxrwxr-x 4 ansible ansible 192 8月 18 20:24 examples drwxrwxr-x 5 ansible ansible 305 8月 18 20:24 hacking drwxrwxr-x 3 ansible ansible 21 8月 18 20:24 lib drwxrwxr-x 2 ansible ansible 78 8月 18 20:24 licenses -rw-rw-r-- 1 ansible ansible 13452 8月 18 20:24 Makefile -rw-rw-r-- 1 ansible ansible 557 8月 18 20:24 MANIFEST.in -rw-rw-r-- 1 ansible ansible 201 8月 18 20:24 MODULE_GUIDELINES.md drwxrwxr-x 10 ansible ansible 120 8月 18 20:24 packaging -rw-rw-r-- 1 ansible ansible 5265 8月 18 20:24 README.rst -rw-rw-r-- 1 ansible ansible 371 8月 18 20:24 requirements.txt -rw-rw-r-- 1 ansible ansible 10586 8月 18 20:24 setup.py -rw-rw-r-- 1 ansible ansible 3287 8月 18 20:24 shippable.yml drwxrwxr-x 10 ansible ansible 121 8月 18 20:24 test -rw-rw-r-- 1 ansible ansible 1103 8月 18 20:24 tox.ini 切换版本\n(.py3-a2.5-env) [ansible@localhost .py3-a2.5-env]$ git checkout stable-2.5 分支 stable-2.5 设置为跟踪来自 origin 的远程分支 stable-2.5。 切换到一个新分支 \u0026lsquo;stable-2.5\u0026rsquo;\n(.py3-a2.5-env) [ansible@localhost ansible]$ git branch devel * stable-2.5 (.py3-a2.5-env) [ansible@localhost ansible]$ pwd /home/ansible/.py3-a2.5-env/ansible 加载ansible2.5版本 source /home/ansible/.py3-a2.5-env/ansible/hacking/env-setup -q 验证版本\n(.py3-a2.5-env) [ansible@localhost ansible]$ ansible --version ansible 2.5.8 (stable-2.5 b23b1c2ee0) last updated 2018/08/18 20:35:45 (GMT +800) config file = None configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /home/ansible/.py3-a2.5-env/ansible/lib/ansible executable location = /home/ansible/.py3-a2.5-env/ansible/bin/ansible python version = 3.7.0 (default, Aug 18 2018, 20:09:08) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] ","date":"2018-08-19T16:38:00+08:00","permalink":"/archives/175/","title":"Ansible安装配置"},{"content":"基础环境部署 基础环境使用CentOS 7，版本号为1804，部署在本地虚拟机VM上\n安装CentOS+docker CentOS下载地址\n使用最小安装，ISO镜像大小为906MB，在VM中导入镜像，给4G内存（Rancher2.0最低要求），安装过程系统配置均为默认配置\n安装好后为纯净环境，使用ip addr即可查看ip，使用外部远程连接工具连接虚拟机ip 使用yum install docker安装docker环境 安装好后使用命令systemctl start docker启动docker 使用 docker info或者docker version查看版本\n[root@localhost ~]# docker version Client: Version: 1.13.1 API version: 1.26 Package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64 Go version: go1.9.4 Git commit: dded712/1.13.1 Built: Tue Jul 17 18:34:48 2018 OS/Arch: linux/amd64 Server: Version: 1.13.1 API version: 1.26 (minimum version 1.12) Package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64 Go version: go1.9.4 Git commit: dded712/1.13.1 Built: Tue Jul 17 18:34:48 2018 OS/Arch: linux/amd64 Experimental: false 安装Rancher 为了加速Rancher下载速度，需要配置docker镜像下载地址，此处提供一个阿里云的镜像地址\n{ \u0026quot;registry-mirrors\u0026quot;: [\u0026quot;https://fy707np5.mirror.aliyuncs.com\u0026quot;] } 使用命令编辑docker镜像地址 vi /etc/docker/daemon.json 保存后重载配置systemctl daemon-reload 重启docker服务systemctl restart docker 镜像地址配置完成\n访问Rancher官网查看docker命令 Rancher英文官网\nRancher中文官网\n访问quickstart查看docker命令\n目前Rancher官方开始推荐2.0版本，首次使用推荐使用1.6稳定版 在下方找到1.6文档，使用命令在虚拟机中安装Rancher1.6 sudo docker run -d --restart=unless-stopped -p 8080:8080 rancher/server:stable 配置防火墙 安装后若没有打开8080端口则会报错，配置CentOS防火墙 来源：[莫小安-CentOS7使用firewalld打开关闭防火墙与端口][10]\n1、firewalld的基本使用 启动： systemctl start firewalld 关闭： systemctl stop firewalld 查看状态： systemctl status firewalld 开机禁用 ： systemctl disable firewalld 开机启用 ： systemctl enable firewalld 2.systemctl是CentOS7的服务管理工具中主要的工具，它融合之前service和chkconfig的功能于一体。 启动一个服务：systemctl start firewalld.service 关闭一个服务：systemctl stop firewalld.service 重启一个服务：systemctl restart firewalld.service 显示一个服务的状态：systemctl status firewalld.service 在开机时启用一个服务：systemctl enable firewalld.service 在开机时禁用一个服务：systemctl disable firewalld.service 查看服务是否开机启动：systemctl is-enabled firewalld.service 查看已启动的服务列表：systemctl list-unit-files|grep enabled 查看启动失败的服务列表：systemctl \u0026ndash;failed\n3.配置firewalld-cmd\n查看版本： firewall-cmd \u0026ndash;version 查看帮助： firewall-cmd \u0026ndash;help 显示状态： firewall-cmd \u0026ndash;state 查看所有打开的端口： firewall-cmd \u0026ndash;zone=public \u0026ndash;list-ports 更新防火墙规则： firewall-cmd \u0026ndash;reload 查看区域信息: firewall-cmd \u0026ndash;get-active-zones 查看指定接口所属区域： firewall-cmd \u0026ndash;get-zone-of-interface=eth0 拒绝所有包：firewall-cmd \u0026ndash;panic-on 取消拒绝状态： firewall-cmd \u0026ndash;panic-off 查看是否拒绝： firewall-cmd \u0026ndash;query-panic\n那怎么开启一个端口呢 添加 firewall-cmd \u0026ndash;zone=public \u0026ndash;add-port=80/tcp \u0026ndash;permanent （\u0026ndash;permanent永久生效，没有此参数重启后失效） 重新载入 firewall-cmd \u0026ndash;reload 查看 firewall-cmd \u0026ndash;zone= public \u0026ndash;query-port=80/tcp 删除 firewall-cmd \u0026ndash;zone= public \u0026ndash;remove-port=80/tcp \u0026ndash;permanent\n通过命令放行8080端口\nfirewall-cmd --zone=public --add-port=8080/tcp --permanent 重载防火墙\nfirewall-cmd --reload 查看端口放行情况\nfirewall-cmd --zone=public --list-ports 设置Rancher 设置Rancher界面语言 通过虚拟机ip+端口号访问Rancher：http://192.168.79.142:8080 默认为英文界面，Rancher官方有中文支持 修改右下方语言选项即可 设置Rancher主机 首次访问需要设置主机 按照默认识别的IP即可，点击保存 选择主机类型，包含自定义主机、亚马逊主机等，此处选择自定义主机 依照指示操作，为了确保通信测试，暂时关闭防火墙 为了测试多台Rancher Server和agent服务，使用VM保存快照并复制一份虚拟机启动 目前测试：128为server，129为agent，注意此处填的是客户端的地址，非主机地址 设置Rancher客户端 复制Server的配置命令，在agent服务器上运行，等待下载 按照指示，运行后关闭Server主机配置 查看主机列表 也可以添加环境、测试环境、生产环境等 ","date":"2018-08-11T11:41:00+08:00","permalink":"/archives/174/","title":"从零开始部署Rancher"},{"content":"依赖模块报错：程序包不存在 问题描述 使用SprinCloud构建的多模块项目，各应用使用Springboot构建，工程编写完毕，在模块中添加构建配置 org.springframework.boot spring-boot-maven-plugin 使用maven命令构建应用mvn install package -DskipTests maven报错如下 问题分析 通过报错看出，只要有依赖外部模块的均报“程序包不存在”或者“找不到符号”的错误，而通过IDEA运行工程均正常，故排除代码错误，猜测为构建错误 查询到公共模块要先打包，故对公共包执行了一次install，再打包子模块，失败 百度查找问题解决办法，做了如下尝试 （1）配置maven中的jdk版本，确保小版本号也是一致的，无果 （2）更换构建配置如下，无效 org.apache.maven.plugins maven-compiler-plugin org.springframework.boot spring-boot-maven-plugin （3）使用不同的maven命令构建，包括clean、package、install等，报错信息相同 突然发现有人的博客说明了公共包不应该加入构建配置，否则会二次构建，导致找不到包 来源 注意！注意！这里有一个巨坑，我已经义无反顾的跳进去一次了，大家一定不要再往里面跳了：Common打包出来的应该是不可执行的jar包，所以不要在Common的pom中定义spring-boot-maven-plugin插件，因为这个SpringBoot插件会在Maven的package后进行二次打包，目的为了生成可执行jar包，如果C中定义了这个插件，会报错提示没有找到main函数。这时你就可以去打包front项目了，当然打包的时候可能还是不行，这里还有一个小坑，如果还是不能进行打包的话，那么就install一下root项目，也就是总目录下的pom文件对应的install操作，这样再打包front项目基本上就没有问题了，老铁，都是经验呀，希望对你们有帮助 根据博客说法去除了公共包的构建配置，依然报错\n解决方法 根据博客说法思考了一下，考虑到父工程也没有主类，无需运行，故去除父工程的构建配置，最后添加了构建模块只有带启动类的模块，再次对整个父工程执行clean-\u0026gt;install，构建成功 ","date":"2018-08-07T21:22:00+08:00","permalink":"/archives/173/","title":"maven构建多模块SpringBoot项目遇到的坑"},{"content":"SpringCloud Config 配置无法读取 问题描述 配置中心的配置文件如下\neureka: client: service-url: defaultZone: http://localhost:8761/eureka/ spring: application: name: config cloud: config: server: git: uri: http://XXX/fjy8018/config-repo username: 1186032234@qq.com password: XXX basedir: spring-cloud\\config\\basedir server: port: 8084 已经通过网页测试该账号和密码可以访问该仓库，并有权限，启动配置中心后通过页面访问提示Cannot clone or checkout repository 解决方法 SpringCloud正式版的配置中心路径必须是带.git结尾的全路径，将配置改为全路径即可\neureka: client: service-url: defaultZone: http://localhost:8761/eureka/ spring: application: name: config cloud: config: server: git: uri: http://XXX/fjy8018/config-repo.git username: 1186032234@qq.com password: XXX basedir: spring-cloud\\config\\basedir server: server: port: 8084 ","date":"2018-07-24T21:34:00+08:00","permalink":"/archives/172/","title":"SpringCloud中遇到的坑"},{"content":"bindingResult无法捕获校验错误 错误复现 原工程代码，RegisterForm对象设置了校验规则 预期的返回结果\n{ \u0026quot;code\u0026quot;: 4, \u0026quot;msg\u0026quot;: \u0026quot;表单错误\u0026quot;, \u0026quot;data\u0026quot;: null } 实际执行结果 并且控制台打出了异常堆栈，而且记录的log4j日志也没有显示 原因 经过查阅资料，得到原因\n@Valid 和 BindingResult 是一一对应的，如果有多个@Valid，那么每个@Valid后面跟着的BindingResult就是这个@Valid的验证结果，顺序不能乱\n解决 故将BindingResult跟在RegisterForm后面即可\npublic ResultVO doRegister(@Valid RegisterForm registerForm, BindingResult bindingResult, HttpServletRequest request) throws Exception{ 再次执行得到预期返回值 控制台也打出预期的日志，并没有显示异常 ","date":"2018-07-08T16:03:00+08:00","permalink":"/archives/170/","title":"SpringBoot的一些坑"},{"content":"镜像无法删除 问题复现 当前的镜像\nPS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker images REPOSITORY TAG IMAGE ID CREATED SIZE orderbook latest 6514a4df3367 7 seconds ago 300MB hub.c.163.com/fjy8018/mysql-orderbooks latest 37a82bac5f71 About an hour ago 407MB hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB 运行一个镜像\nPS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker run -d -p 8888:8080 orderbook:latest 678dcf28c1c3206cd78a824aab51cc5d1f6c333ed85f48d4c364a51526049f20 查看进程，发现进程没有出现（3306是已经启动的MySQL容器），由于容器制作过程出现问题导致进程无法成功启动\nPS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 43146ddfc771 hub.c.163.com/library/mysql:latest \u0026quot;docker-entrypoint.s…\u0026quot; 2 hours ago Up 2 hours 0.0.0.0:8006-\u0026gt;3306/tcp hopeful_euclid 检查镜像\nPS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker images REPOSITORY TAG IMAGE ID CREATED SIZE orderbook latest 6514a4df3367 44 seconds ago 300MB hub.c.163.com/fjy8018/mysql-orderbooks latest 37a82bac5f71 About an hour ago 407MB hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB 删除没有成功运行的镜像报错，报该容器已经被占用\nPS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker rmi 6514a4df3367 Error response from daemon: conflict: unable to delete 6514a4df3367 (must be forced) - image is being used by stopped container 678dcf28c1c3 删除报错中提示的容器ID\nPS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker rmi 678dcf28c1c3 Error: No such image: 678dcf28c1c3 报无该镜像\n解决 官方对rm与rmi的解释： rm Remove one or more containers rmi Remove one or more images 可知rm删除的是容器，rmi删除的是镜像，两者的区别如下：\n容器（container）的定义和镜像（image）几乎一模一样，也是一堆层的统一视角，唯一区别在于容器的最上面那一层是可读可写的。 镜像（Image）就是一堆只读层（read-only layer）的统一视角 要点：容器 = 镜像 + 可读层。并且容器的定义并没有提及是否要运行容器。\n故对于不在镜像列表的ID，要删除的是容器，使用rm命令 PS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker rm 678dcf28c1c3 678dcf28c1c3\n列出所有的镜像\nPS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker images REPOSITORY TAG IMAGE ID CREATED SIZE orderbook latest 6514a4df3367 2 minutes ago 300MB hub.c.163.com/fjy8018/mysql-orderbooks latest 37a82bac5f71 About an hour ago 407MB hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB 再删除目标镜像 PS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker rmi 6514a4df3367 Untagged: orderbook:latest Deleted: sha256:6514a4df33678bd98e32b5738150c88c5d72b9a3a4d86d1c370406f0f7f18788 Deleted: sha256:68a70a0ac20590c8412286b3c34ab2bc5c5b7c8845088536f03702185ac34105 Deleted: sha256:bf15e37a31e4266bc8a962a2b11fc3223c07de328ff85b17c70a15653c71061b Deleted: sha256:b447a606517609703f1efe4c560c42b9aeeaee669d22f12a65d2d2929d8ecc20\n删除成功 PS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker images REPOSITORY TAG IMAGE ID CREATED SIZE hub.c.163.com/fjy8018/mysql-orderbooks latest 37a82bac5f71 About an hour ago 407MB hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB\n停止并删除所有容器 批处理命令\ndocker stop $(docker ps -aq) docker rm $(docker ps -aq) ","date":"2018-07-05T14:21:00+08:00","permalink":"/archives/169/","title":"Docker部署中遇到的坑"},{"content":"准备docker环境 去官网下载docker环境，此处使用win10的docker环境，若早期版本的windows要安装额外的工具 检查docker版本 下载基础环境 本次决定搭建的docker为订书系统，其基于jsp编写，运行在tomcat上，数据库依赖MySQL，故需要一个MySQL和tomcat环境（tomcat环境自带Java） 去网易镜像中心下载镜像，网易镜像中心与官方镜像中心保持同步更新，而且对于国内来说速度更快\n部署MySQL镜像 下载 复制下载地址 mysql镜像下方也有使用说明 以及环境变量的描述，是否需要在创建时建立数据库等 启动 使用命令启动mysql的docker，指定本机8006端口映射到docker的3306端口，同时设置数据库root用户密码为root -d指定为后台运行\ndocker run -d -p 8006:3306 -e MYSQL_ROOT_PASSWORD=root hub.c.163.com/library/mysql:latest 使用命令查看当前运行的docker\ndocker ps 测试 使用Navicat连接docker的数据库 建立相关数据库，不使用环境变量创建是为了设置字符集 启用命令行界面创建相关用户并授权 导入数据库文件 部署Tomcat镜像 查看本机的ip 下载 从镜像中心下载tomcat 查看镜像中配置的tomcat目录 查看当前已经下载的镜像\nPS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker images REPOSITORY TAG IMAGE ID CREATED SIZE hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB 编写Dockerfile文件将工程移到tomcat的webapps目录下\nfrom hub.c.163.com/library/tomcat MAINTAINER fjy8018 fjy8018@gmail.com COPY /OrderBooks /usr/local/tomcat/webapps/OrderBooks 修改工程中的数据库地址，地址为ip+暴露的docker端口 使用docker build -t orderbook .命令将docker打包成镜像 启动 使用命令启动带工程的tomcat，指定访问端口为8888\ndocker run -d -p 8888:8080 orderbook:latest 配置 由于使用了jsp+jdbc作为数据库连接，故需要进入容器内部将jdbc的jar复制到tomcat的lib目录下，否则tomcat会报无法加载驱动，使用命令进入容器内部\ndocker exec -it {容器ID} /bin/sh 使用exec命令比attach优势在于，exec进入容器后再使用exit命令不会令容器停止 进入容器将工程的lib复制到tomcat中的lib\nPS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker exec -it 6fe288a25be0 /bin/sh # cp /usr/local/tomcat/webapps/OrderBooks/WebRoot/WEB-INF/lib/mysql-connector-java-5.1.45-bin.jar /usr/local/tomcat/lib # exit PS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker restart 6fe288a25be0 6fe288a25be0 测试 外部浏览器访问工程 工程正常访问 提交到网易镜像仓库 官方推送指南\n提交修改的镜像 docker commit提交修改的镜像\nPS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker commit 6fe288a25be0 orderbooks sha256:30c6a878adc80f54118f5c67c39c271baa5dc68006aa26b64ef5281832fb4b37 PS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker images REPOSITORY TAG IMAGE ID CREATED SIZE orderbooks latest 30c6a878adc8 5 seconds ago 301MB orderbook latest a57e26354272 10 minutes ago 300MB hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB 提交mysql容器的修改 PS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker commit 43146ddfc771 hub.c.163.com/fjy8018/mysql-orderbooks sha256:37a82bac5f71d54e3ff31b0ddd795604e46d4f2e4dde8a6a73d556cc5f26411a PS D:\\JAVA WORKSPACE\\dockerTemp\u0026gt; docker images REPOSITORY TAG IMAGE ID CREATED SIZE hub.c.163.com/fjy8018/mysql-orderbooks latest 37a82bac5f71 3 seconds ago 407MB orderbooks latest 30c6a878adc8 13 minutes ago 301MB hub.c.163.com/fjy8018/tomcat-orderbooks latest 30c6a878adc8 13 minutes ago 301MB orderbook latest a57e26354272 23 minutes ago 300MB hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB\n推送镜像 将镜像推送到网易仓库 登录到网易镜像仓库查看，默认推送会设置为private，推送完成 ","date":"2018-07-04T17:06:00+08:00","permalink":"/archives/168/","title":"自定义docker容器"},{"content":"准备文件 目前GitLab汉化由xhang维护，故直接克隆其工程到本地目录即可\ngit clone https://gitlab.com/xhang/gitlab.git 当前GitLab最新版为10.8.4，其他版本自行看release切换到对应的分支或者下载对应的发布版本即可 下载完毕为文件夹 修改系统名称别名 CentOS 7中出于安全考虑，设置了部分别名，防止误操作，但在此会导致复制文件需要反复输入y，故将其关闭 通过alias查看当前的别名设置，若有alias cp='cp -i'，则说明每次复制文件都会出现提示信息 使用vi ~/.bashrc修改配置文件 source ~/.bashrc使配置文件生效，但还需要重启会话，否则修改的命令可能依旧无效 再次查看，此时别名配置修改成功 配置汉化 此处为初次安装汉化，故与补丁方式汉化不同，直接覆盖文件即可 覆盖前备份一下\ncp -r /opt/gitlab/embedded/service/gitlab-rails{,.ori} 覆盖文件\ncp gitlab/* /opt/gitlab/embedded/service/gitlab-rails/ -rf 重启GitLab\ngitlab-ctl restart 成功 英文 中文 ","date":"2018-06-12T22:30:00+08:00","permalink":"/archives/160/","title":"GitLab汉化"},{"content":"当前版本 当前GitLab版本由宝塔自动安装的8.8.5,但截止至2018-06-12,CentOS的GitLab官方最新版本为10.8.4,显然版本相差较大,故打算跨版本升级至最新版 步骤 升级指南 GitLab社区版不支持直接跨版本升级，需按照官方指南一步步向上升级，若版本越旧，需要跨的版本就越多 下载相关安装包 安装包国内选择清华镜像站速度较快，教育网还有蜜汁加成\n若升级其他版本也可自行Ctrl+F wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-8.17.7-ce.0.el7.x86_64.rpm wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-9.5.8-ce.0.el7.x86_64.rpm wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-10.1.4-ce.0.el7.x86_64.rpm wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-10.8.4-ce.0.el7.x86_64.rpm 升级 关闭GitLab核心服务\ngitlab-ctl stop unicorn gitlab-ctl stop sidekiq gitlab-ctl stop nginx 安装8.17.7\nrpm -Uvh gitlab-ce-8.17.7-ce.0.el7.x86_64.rpm 重启服务 若之前从宝塔安装后未配置GitLab的Nginx启动命令则需要自行配置后才能正常启动，操作见文章GitLab修改配置后nginx无法启动 重启服务\n[root@localhost ~]# gitlab-ctl restart /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.3.6/lib/omnibus-ctl.rb:533: warning: Insecure world writable dir /www/server/nvm/versions/node/v8.11.2/bin in PATH, mode 040777 ok: run: gitlab-workhorse: (pid 39809) 1s ok: run: logrotate: (pid 39821) 0s ok: run: nginx: (pid 39828) 0s ok: run: postgresql: (pid 39835) 0s ok: run: redis: (pid 39844) 0s ok: run: sidekiq: (pid 39849) 1s ok: run: unicorn: (pid 39854) 0s 重启后稍等片刻即可访问，查看版本号，已经更新为8.17.7，后续步骤同上 可能的报错 部分CentOS版本较低未自带policycoreutils-python组件可能无法升级至10.x,需要手动\nyum -y install policycoreutils-python 完成 更新到10.x后明显启动项都变多了 [root@localhost ~]# gitlab-ctl restart /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.5.0/lib/omnibus-ctl.rb:684: warning: Insecure world writable dir /www/server/nvm/versions/node/v8.11.2/bin in PATH, mode 040777 ok: run: alertmanager: (pid 45330) 1s ok: run: gitaly: (pid 45336) 0s ok: run: gitlab-monitor: (pid 45346) 1s ok: run: gitlab-workhorse: (pid 45362) 0s ok: run: logrotate: (pid 45369) 1s ok: run: nginx: (pid 45378) 0s ok: run: node-exporter: (pid 45383) 0s ok: run: postgres-exporter: (pid 45387) 1s ok: run: postgresql: (pid 45394) 0s ok: run: prometheus: (pid 45476) 1s ok: run: redis: (pid 45484) 0s ok: run: redis-exporter: (pid 45488) 1s ok: run: sidekiq: (pid 45493) 0s ok: run: unicorn: (pid 45503) 1s\n查看版本号，成功 ","date":"2018-06-12T21:50:00+08:00","permalink":"/archives/159/","title":"GitLab跨版本升级"},{"content":"背景 原理 在学习Redis分布式锁时编写Service类，其中包含加锁和解锁两个方法，通过调用Redis的SETNX和GETSET方法加锁 源码 为便于查看，加了logback进行日志打印 需要加锁的方法，该方法为模拟抢购场景的实现 交由SpringBoot自动注入写好的RedisLock包（其实问题出在这里，本应该是10*1000，手误写成10*000结果为0，故所有超时为0） 访问Controller，一个为抢购地址，抢购地址访问一次则数量减一，一个为查询地址 压测工具 使用Apache AB模拟高并发访问\n并发测试 手动测试 手动访问，测试功能是否正常 查询 订购 压测 10个请求，单线程发起测试 数量正常 50个请求，2个线程发起测试 出错 数量显然不对 查看日志，发现后续请求都从判断锁是否过期的方法获得的锁，但执行上数值一致，逻辑也没问题 原因 后来发现在高并发情况下超时时间如果设置过小，多个线程可能获得的时间戳一致，导致加锁无效，即每个线程都在判断时间戳时可以获得锁，导致“超售” 再次压测 数量正常 提高并发量 数量正常，问题解决 ","date":"2018-06-11T12:40:00+08:00","permalink":"/archives/158/","title":"Redis分布式锁在高并发情况下偶然突然失效"},{"content":"起因 在服务器上通过宝塔搭建了GitLab之后一直没修改GitLab的clone域名和IP配置，今天通过命令自定义域名\nvim /etc/gitlab/gitlab.rb 重启配置\nsudo gitlab-ctl reconfigure 显示启动成功，但网页访问无显示\n重启整个GitLab服务\nsudo gitlab-ctl restart Nginx报错无法启动 查找原因 既然报错，那就只能爬日志看看问题在哪里\ngitlab-ctl tail 报文件存在\nchpst: fatal: unable to run: /opt/gitlab/embedded/sbin/nginx: file does not exist 网上查了查，似乎是宝塔的问题， 原因应该是为了防止nginx冲突, 把gitlab的nginx启动文件名改成了gitlab-web, 但有个配置文件没改造成的\n解决 修改nginx启动文件\nvi /opt/gitlab/sv/nginx/run 把\nexec chpst -P /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx 改为\nexec chpst -P /opt/gitlab/embedded/sbin/gitlab-web -p /var/opt/gitlab/nginx 重启nginx服务，启动成功，问题解决 ","date":"2018-06-11T12:30:00+08:00","permalink":"/archives/157/","title":"GitLab修改配置后nginx无法启动"},{"content":"建立结构 建People表 方法：PUT 发送地址：http://192.168.79.138:9200/people\n{ \u0026quot;setting\u0026quot;: { \u0026quot;number_of_shards\u0026quot;: 3, \u0026quot;number_of_replicas\u0026quot;: 1 }, \u0026quot;mappings\u0026quot;: { \u0026quot;man\u0026quot;: { \u0026quot;properties\u0026quot;: { \u0026quot;name\u0026quot;: { \u0026quot;type\u0026quot;: \u0026quot;text\u0026quot; }, \u0026quot;country\u0026quot;: { \u0026quot;type\u0026quot;: \u0026quot;keyword\u0026quot; }, \u0026quot;age\u0026quot;: { \u0026quot;type\u0026quot;: \u0026quot;integer\u0026quot; }, \u0026quot;data\u0026quot;:{ \u0026quot;type\u0026quot;: \u0026quot;date\u0026quot;, \u0026quot;format\u0026quot;: \u0026quot;yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis\u0026quot; } } } } } 建book表 方法：PUT 发送地址：http://192.168.79.138:9200/book\n{ \u0026quot;setting\u0026quot;: { \u0026quot;number_of_shards\u0026quot;: 3, \u0026quot;number_of_replicas\u0026quot;: 1 }, \u0026quot;mappings\u0026quot;: { \u0026quot;novel\u0026quot;: { \u0026quot;properties\u0026quot;: { \u0026quot;word_count\u0026quot;: { \u0026quot;type\u0026quot;: \u0026quot;integer\u0026quot; }, \u0026quot;author\u0026quot;: { \u0026quot;type\u0026quot;: \u0026quot;keyword\u0026quot; }, \u0026quot;title\u0026quot;: { \u0026quot;type\u0026quot;: \u0026quot;text\u0026quot; }, \u0026quot;publish_date\u0026quot;:{ \u0026quot;type\u0026quot;: \u0026quot;date\u0026quot;, \u0026quot;format\u0026quot;: \u0026quot;yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis\u0026quot; } } } } } 插入数据 指定ID插入数据到people.man 方法：PUT 发送地址http://192.168.79.138:9200/people/man/1\n{ \u0026quot;name\u0026quot;: \u0026quot;F嘉阳\u0026quot;, \u0026quot;country\u0026quot;: \u0026quot;China\u0026quot;, \u0026quot;age\u0026quot;: 21, \u0026quot;date\u0026quot;: \u0026quot;1990-05-18\u0026quot; } 图形化查询检验 不指定ID插入数据到people.man 方法：PUT 发送地址http://192.168.79.138:9200/people/man/\n{ \u0026quot;name\u0026quot;: \u0026quot;FJY\u0026quot;, \u0026quot;country\u0026quot;: \u0026quot;China\u0026quot;, \u0026quot;age\u0026quot;: 28, \u0026quot;date\u0026quot;: \u0026quot;1990-05-18\u0026quot; } 图形化查询检验 更新 更新字段数据 方法：POST 发送地址：http://192.168.79.138:9200/people/man/1/_update\n{ \u0026quot;doc\u0026quot;: { \u0026quot;name\u0026quot;: \u0026quot;符嘉阳\u0026quot; } } 图形化查询检验 使用脚本语言更新数据 方法：POST 发送地址：http://192.168.79.138:9200/people/man/1/_update\n{ \u0026quot;script\u0026quot;: { \u0026quot;lang\u0026quot;: \u0026quot;painless\u0026quot;, \u0026quot;inline\u0026quot;: \u0026quot;ctx._source.age += 5\u0026quot; } } 图形化查询检验 使用脚本语言分离参数更新数据 方法：POST 发送地址：http://192.168.79.138:9200/people/man/1/_update\n{ \u0026quot;script\u0026quot;: { \u0026quot;lang\u0026quot;: \u0026quot;painless\u0026quot;, \u0026quot;inline\u0026quot;: \u0026quot;ctx._source.age = params.age\u0026quot;, \u0026quot;params\u0026quot;: { \u0026quot;age\u0026quot;: 21 } } } 图形化查询检验 删除 指定ID删除一行数据 方法：DELETE 发送地址：http://192.168.79.138:9200/people/man/1 图形化查询检验 删除表 方法：DELETE 发送地址：http://192.168.79.138:9200/people 查询 指定ID查询 方法：GET 发送地址：http://192.168.79.138:9200/book/novel/H5LbbWMBzFxBPWV2GG9d 查询所有 方法：GET 发送地址：http://192.168.79.138:9200/book/_search { \u0026ldquo;query\u0026rdquo;: { \u0026ldquo;match_all\u0026rdquo;: {} } }\n指定查询结果大小 { \u0026quot;query\u0026quot;: { \u0026quot;match_all\u0026quot;: {} }, \u0026quot;from\u0026quot;: 1, \u0026quot;size\u0026quot;: 1 } 查询 { \u0026quot;query\u0026quot;: { \u0026quot;match\u0026quot;: { \u0026quot;title\u0026quot;: \u0026quot;Java\u0026quot; } } } 指定查询结果排序方法 { \u0026quot;query\u0026quot;: { \u0026quot;match\u0026quot;: { \u0026quot;title\u0026quot;: \u0026quot;Java\u0026quot; } }, \u0026quot;sort\u0026quot;: [ { \u0026quot;publish_date\u0026quot;: { \u0026quot;order\u0026quot;: \u0026quot;desc\u0026quot; } } ] } 聚合 { \u0026quot;aggs\u0026quot;: { \u0026quot;group_by_word_count\u0026quot;: { \u0026quot;terms\u0026quot;: { \u0026quot;field\u0026quot;: \u0026quot;word_count\u0026quot; } } } } 多字段聚合 { \u0026quot;aggs\u0026quot;: { \u0026quot;group_by_word_count\u0026quot;: { \u0026quot;terms\u0026quot;: { \u0026quot;field\u0026quot;: \u0026quot;word_count\u0026quot; } }, \u0026quot;group_by_author\u0026quot;: { \u0026quot;terms\u0026quot;: { \u0026quot;field\u0026quot;: \u0026quot;author\u0026quot; } } } } 聚合 { \u0026quot;aggs\u0026quot;: { \u0026quot;grades_by_word_count\u0026quot;: { \u0026quot;stats\u0026quot;: { \u0026quot;field\u0026quot;: \u0026quot;word_count\u0026quot; } } } } 模糊匹配(默认匹配法则) { \u0026quot;query\u0026quot;: { \u0026quot;match\u0026quot;: { \u0026quot;title\u0026quot;: \u0026quot;Java虚拟机\u0026quot; } } } 习语匹配 { \u0026quot;query\u0026quot;: { \u0026quot;match_phrase\u0026quot;: { \u0026quot;title\u0026quot;: \u0026quot;Java虚拟机\u0026quot; } } } 多字段模糊匹配查询 { \u0026quot;query\u0026quot;: { \u0026quot;multi_match\u0026quot;: { \u0026quot;query\u0026quot;: \u0026quot;F嘉阳\u0026quot;, \u0026quot;fields\u0026quot;: [\u0026quot;author\u0026quot;,\u0026quot;title\u0026quot;] } } } 语法查询 { \u0026quot;query\u0026quot;: { \u0026quot;query_string\u0026quot;: { \u0026quot;query\u0026quot;: \u0026quot;ElasticSearch AND 入门\u0026quot; } } } { \u0026quot;query\u0026quot;: { \u0026quot;query_string\u0026quot;: { \u0026quot;query\u0026quot;: \u0026quot;(ElasticSearch AND 入门) OR PS\u0026quot; } } } 语法查询指定字段 { \u0026quot;query\u0026quot;: { \u0026quot;query_string\u0026quot;: { \u0026quot;query\u0026quot;: \u0026quot;ElasticSearch OR 瓦\u0026quot;, \u0026quot;fields\u0026quot;: [\u0026quot;title\u0026quot;,\u0026quot;author\u0026quot;] } } } 结构化查询(指定字段) { \u0026quot;query\u0026quot;: { \u0026quot;term\u0026quot;: { \u0026quot;word_count\u0026quot;: 1000 } } } { \u0026quot;query\u0026quot;: { \u0026quot;term\u0026quot;: { \u0026quot;author\u0026quot;: \u0026quot;F嘉阳\u0026quot; } } } 结构化查询(指定范围) gte-\u0026gt;大于等于 lte-\u0026gt;小于等于 e-\u0026gt;equal(等于)\n{ \u0026quot;query\u0026quot;: { \u0026quot;range\u0026quot;: { \u0026quot;word_count\u0026quot;: { \u0026quot;gte\u0026quot;: 1000, \u0026quot;lte\u0026quot;: 2000 } } } } 结构化查询(指定日期范围) { \u0026quot;query\u0026quot;: { \u0026quot;range\u0026quot;: { \u0026quot;publish_date\u0026quot;: { \u0026quot;gte\u0026quot;: \u0026quot;2018-01-01\u0026quot;, \u0026quot;lt\u0026quot;: \u0026quot;2018-12-31\u0026quot; } } } } 结构化查询(指定当前时间) { \u0026quot;query\u0026quot;: { \u0026quot;range\u0026quot;: { \u0026quot;publish_date\u0026quot;: { \u0026quot;gte\u0026quot;: \u0026quot;2018-01-01\u0026quot;, \u0026quot;lte\u0026quot;: \u0026quot;now\u0026quot; } } } } 子条件查询，使用filter 使用filter过滤器后产生缓存，下次查询速度更快\n{ \u0026quot;query\u0026quot;: { \u0026quot;bool\u0026quot;: { \u0026quot;filter\u0026quot;: { \u0026quot;term\u0026quot;: { \u0026quot;word_count\u0026quot;: 1000 } } } } } 复合条件查询-固定分数查询 { \u0026quot;query\u0026quot;: { \u0026quot;constant_score\u0026quot;: { \u0026quot;filter\u0026quot;: { \u0026quot;match\u0026quot;: { \u0026quot;title\u0026quot;: \u0026quot;Java\u0026quot; } } } } } 复合条件查询-指定分数查询 { \u0026quot;query\u0026quot;: { \u0026quot;constant_score\u0026quot;: { \u0026quot;filter\u0026quot;: { \u0026quot;match\u0026quot;: { \u0026quot;title\u0026quot;: \u0026quot;Java\u0026quot; } }, \u0026quot;boost\u0026quot;: 2 } } } 复合条件查询-布尔查询-指定应当满足的条件(满足其一即可) { \u0026quot;query\u0026quot;: { \u0026quot;bool\u0026quot;: { \u0026quot;should\u0026quot;: [ { \u0026quot;match\u0026quot;: { \u0026quot;author\u0026quot;: \u0026quot;F嘉阳\u0026quot; } }, { \u0026quot;match\u0026quot;: { \u0026quot;title\u0026quot;: \u0026quot;Java\u0026quot; } } ] } } } 复合条件查询-布尔查询-指定必须满足的条件(两个都必须匹配) { \u0026quot;query\u0026quot;: { \u0026quot;bool\u0026quot;: { \u0026quot;must\u0026quot;: [ { \u0026quot;match\u0026quot;: { \u0026quot;author\u0026quot;: \u0026quot;F嘉阳\u0026quot; } }, { \u0026quot;match\u0026quot;: { \u0026quot;title\u0026quot;: \u0026quot;Java\u0026quot; } } ] } } } 复合条件查询-布尔查询-过滤器 { \u0026quot;query\u0026quot;: { \u0026quot;bool\u0026quot;: { \u0026quot;must\u0026quot;: [ { \u0026quot;match\u0026quot;: { \u0026quot;author\u0026quot;: \u0026quot;F嘉阳\u0026quot; } }, { \u0026quot;match\u0026quot;: { \u0026quot;title\u0026quot;: \u0026quot;PS\u0026quot; } } ], \u0026quot;filter\u0026quot;: [ { \u0026quot;term\u0026quot;: { \u0026quot;word_count\u0026quot;: 700 } } ] } } } 复合条件查询-must_not { \u0026quot;query\u0026quot;: { \u0026quot;bool\u0026quot;: { \u0026quot;must_not\u0026quot;: { \u0026quot;term\u0026quot;: { \u0026quot;author\u0026quot;: \u0026quot;F嘉阳\u0026quot; } } } } } ","date":"2018-05-21T07:28:00+08:00","permalink":"/archives/155/","title":"elasticsearch原生json操作笔记"},{"content":"环境搭建 安装包和插件下载 下载elasticsearch 通过官网下载elasticsearch 官网下载页面 此处下载的为elasticsearch-6.2.4.tar.gz\n下载elasticsearch-head插件 下载elasticsearch-head插件可以在浏览器中以图形化的方式显示下载elasticsearch中的数据 GitHub下载页面\n安装node环境 elasticsearch-head插件依赖node环境来启动，可以通过官方node.js安装包安装，也可以通过宝塔自带的PM2管理器安装，此处安装宝塔的PM2管理器 安装后检查当前环境中的node版本，若装好后node命令无法使用可以尝试重启服务器或者让配置环境变量生效\n[root@localhost elasticsearch]# node -v v8.11.2 将两个下载好的文件用宝塔上传到CentOS 7环境中并解压 单节点环境配置 elasticsearch单节点配置 配置文件目录为elasticsearch-6.2.4/config 修改elasticsearch.yml\n#虚拟机IP network.host: 192.168.79.138 #es端口 http.port: 9200 #防止跨源请求阻塞 http.cors.enabled: true http.cors.allow-origin: \u0026#34;*\u0026#34; #主节点名称 cluster.name: Fjiayang #节点名称 node.name: master #是否为主节点 node.master: true 若虚拟机或者服务器内存较小，则还要配置jvm.options，将默认1g内存设置为512m或者更小\n-Xms512m -Xmx512m elasticsearch-head插件安装 进入elasticsearch-head安装目录，执行node的安装命令\n[root@localhost elasticsearch]# cd elasticsearch-head-master [root@localhost elasticsearch-head-master]# npm install 集群环境配置 集群环境只需要解压多份es压缩包，并按如下对elasticsearch.yml配置文件进行简单配置即可，此处搭建了一个主节点，两个从节点 从节点1\n#子节点IP network.host: 192.168.79.138 #子节点端口 http.port: 8200 #主节点IP discovery.zen.ping.unicast.hosts: [\u0026#34;192.168.79.138\u0026#34;] #主节点名称 cluster.name: Fjiayang #子节点名称 node.name: slave1 从节点2\n#子节点IP network.host: 192.168.79.138 #子节点端口 http.port: 8300 #主节点IP discovery.zen.ping.unicast.hosts: [\u0026#34;192.168.79.138\u0026#34;] #主节点名称 cluster.name: Fjiayang #子节点名称 node.name: slave2 服务启动 elasticsearch启动 elasticsearch默认不支持root用户启动，故要新建用户并对运行目录授予权限才能启动es\n[root@localhost elasticsearch]# adduser es [root@localhost elasticsearch]# passwd es 更改用户 es 的密码 。 新的 密码： 无效的密码： 密码未通过字典检查 - 过于简单化/系统化 重新输入新的 密码： passwd：所有的身份验证令牌已经成功更新。 在root用户下对es目录所有权进行更改\n[root@localhost www]# chown -R es /www/elasticsearch [root@localhost www]# ll 总用量 4 drwxrwxrwx 5 es www 151 5月 16 21:10 elasticsearch drwxr-xr-x 6 root root 4096 5月 7 09:01 Recycle_bin drwxr-xr-x. 14 root root 179 5月 7 09:20 server drwxrwxrwx. 3 root root 116 5月 6 15:41 wwwlogs drwxr-xr-x. 4 root root 38 5月 6 15:41 wwwroot 切换到新建的用户es，用/bin/elasticsearch启动es服务，若加上-d则为后台启动，即/bin/elasticsearch -d\n[root@localhost www]# su es [es@localhost elasticsearch]$ cd elasticsearch/elasticsearch-6.2.4 [es@localhost elasticsearch-6.2.4]$ ls bin config data hs_err_pid4546.log lib LICENSE.txt logs modules NOTICE.txt plugins README.textile [es@localhost elasticsearch-6.2.4]$ ./bin/elasticsearch -d 如果配置了集群则继续对每个从节点逐一启动即可\nelasticsearch-head插件启动 进入elasticsearch-head目录，使用npm run start启动node服务即可 服务检验 首先使用宝塔开放相关防火墙端口，\n直接访问es服务，浏览器默认显示为原生json格式，此处使用chrome浏览器的JSON viewer进行了美化 访问elasticsearch-head插件，此处截图为配置好集群后的截图 elasticsearch环境搭建完成！\n","date":"2018-05-17T18:01:00+08:00","permalink":"/archives/154/","title":"elasticsearch分布式环境搭建和插件安装"},{"content":"目标 当前工作簿内容，其标题为万科A(000002)近一个月历史交易(1525501238 符嘉阳) 现希望其根据当前证券名称结合下方的日期自动变更标题为万科A(000002)从20180309至20180410近一个月历史交易(15251101238 符嘉阳) 分析 其中应该涉及一下几点\n保留当前的标题，并对证券名称和证券代码进行截取 将开始日期和结束日期获取并转为目标格式的字符串 字符串拼接 所用函数 函数名称：MID 主要功能：从一个文本字符串的指定位置开始，截取指定数目的字符。 使用格式：MID(text,start_num,num_chars) 参数说明：text代表一个文本字符串；start_num表示指定的起始位置；num_chars表示要截取的数目。\n函数名称：TEXT 主要功能：将数值转换为按指定数字格式表示的文本。 使用格式：TEXT(value,format_text) 参数说明：Value为数值、计算结果为数字值的公式，或对包含数字值的单元格的引用。\n函数名称：CONCATENATE 主要功能：将两个或多个文本字符串联接为一个字符串。 使用格式：CONCATENATE(text1, [text2], \u0026hellip;) 参数说明：text1 （必需）要联接的第一个项目。项目可以是文本值、数字或单元格引用。Text2, \u0026hellip; （可选）要联接的其他文本项目。最多可以有 255 个项目，总共最多支持 8,192 个字符。\n设计结果 =CONCATENATE(MID(A89,1,FIND(\u0026quot;)\u0026quot;,A89)),\u0026quot;从\u0026quot;,TEXT(A25,\u0026quot;yyyymmdd\u0026quot;),\u0026quot;至\u0026quot;,TEXT(A5,\u0026quot;yyyymmdd\u0026quot;),\u0026quot;近一个月历史交易(15251101238 符嘉阳)\u0026quot;) 效果如图 通用性测试成功 时间也会自动随内容变化 ","date":"2018-05-12T13:10:00+08:00","permalink":"/archives/153/","title":"Excel字符串根据指定内容进行变化的嵌套函数"},{"content":"简单宏操作 Excel的宏实质是VB语句，现考虑制作一个宏用于将当前表中的数据复制成4个工作簿\n录制 打开目标工作簿，当前只有Sheet1中有数据，预期结果是将当期数据复制到Sheet2、Sheet3、Sheet4中 点击视图-\u0026gt;录制宏 指定名称和快捷键即可开始录制操作 查看代码 录制完成后店家查看宏，选择编辑即可查看VB代码 在此提供代码，注意，该代码起点是Sheet2,即切换到Sheet2中才能开始执行，否则会报错 Sub 复制() ' \u0026rsquo; 复制 宏 ' \u0026rsquo; 快捷键: Ctrl+o ' Cells.Select Selection.Copy Sheets(\u0026ldquo;Sheet2\u0026rdquo;).Select ActiveSheet.Paste Sheets(\u0026ldquo;Sheet3\u0026rdquo;).Select ActiveSheet.Paste Sheets.Add After:=ActiveSheet ActiveSheet.Paste Range(\u0026ldquo;E29\u0026rdquo;).Select Application.CutCopyMode = False ActiveWorkbook.Save End Sub\n复杂宏 VB自然支持函数（宏）调用，例如以下代码，先录制了三个预测函数操作，通过一个预测()宏对三个函数操作进行调用\nSub 预测线性() ' ' 预测线性 宏 ' ' 快捷键: Ctrl+a ' Range(\u0026quot;A31\u0026quot;).Select ActiveCell.FormulaR1C1 = \u0026quot;=R[-26]C+1\u0026quot; Range(\u0026quot;A32\u0026quot;).Select ActiveWorkbook.Save Range(\u0026quot;B31\u0026quot;).Select Application.CutCopyMode = False ActiveCell.FormulaR1C1 = _ \u0026quot;=FORECAST(RC[-1],R[-26]C:R[-1]C,R[-26]C[-1]:R[-1]C[-1])\u0026quot; Range(\u0026quot;A31:B31\u0026quot;).Select Selection.Font.Bold = True ActiveWorkbook.Save End Sub Sub 指数预测() ' ' 指数预测 宏 ' ' 快捷键: Ctrl+e ' Range(\u0026quot;A31\u0026quot;).Select ActiveCell.FormulaR1C1 = \u0026quot;=R[-26]C+1\u0026quot; Range(\u0026quot;B31\u0026quot;).Select Application.CutCopyMode = False Application.CutCopyMode = False Application.CutCopyMode = False ActiveCell.FormulaR1C1 = _ \u0026quot;=GROWTH(R[-26]C:R[-8]C,R[-26]C[-1]:R[-8]C[-1],RC[-1])\u0026quot; Range(\u0026quot;A31:B31\u0026quot;).Select Selection.Font.Bold = True ActiveWorkbook.Save End Sub Sub 对数预测() ' ' 对数预测 宏 ' ' 快捷键: Ctrl+w ' Application.Run \u0026quot;PERSONAL.XLSB!预测线性\u0026quot; Range(\u0026quot;B31\u0026quot;).Select Selection.ClearContents Range(\u0026quot;B31\u0026quot;).Select ActiveCell.FormulaR1C1 = \u0026quot;=(Sheet1!RC+Sheet3!RC)/2\u0026quot; Range(\u0026quot;B31\u0026quot;).Select ActiveWorkbook.Save End Sub Sub 预测() ' ' 预测 宏 ' ' 快捷键: Ctrl+q ' Application.Run \u0026quot;PERSONAL.XLSB!预测线性\u0026quot; Sheets(\u0026quot;Sheet3\u0026quot;).Select Application.Run \u0026quot;PERSONAL.XLSB!指数预测\u0026quot; Sheets(\u0026quot;Sheet2\u0026quot;).Select Application.Run \u0026quot;PERSONAL.XLSB!对数预测\u0026quot; ActiveWorkbook.Save End Sub ","date":"2018-05-12T12:42:00+08:00","permalink":"/archives/152/","title":"Excel使用宏将复杂操作自动化"},{"content":"DataNode没启动 这种情况多出现于该机器之前配置过hadoop，没清理干净的情况，即仍有文件目录，导致新hadoop的clusterID与旧的不一致，导致HDFS无法启动，报JavaIO异常\n2018-04-22 00:21:15,868 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /www/hadoop/app/tmp/dfs/data: namenode clusterID = CID-6c7766c6-b260-42b6-8c03-07dda802c0cf; datanode clusterID = CID-a53df7ad-e593-4a29-ad15-a93486c54776 2018-04-22 00:21:15,868 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool \u0026lt;registering\u0026gt; (Datanode Uuid unassigned) service to hadoop0/192.168.79.129:8020. Exiting. java.io.IOException: All specified directories are failed to load. at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1394) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1355) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:228) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829) at java.lang.Thread.run(Thread.java:745) 2018-04-22 00:21:15,869 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool \u0026lt;registering\u0026gt; (Datanode Uuid unassigned) service to hadoop0/192.168.79.129:8020 解决方案 清理所有节点的旧文件存储目录，重新再根节点格式化存储空间\nrm -rf app/ hdfs namenode -format 重新启动HDFS\n[hadoop@hadoop0 bin]$ cd ../sbin [hadoop@hadoop0 sbin]$ ./start-all.sh 查看进程，DataNode已启动\n[hadoop@hadoop0 sbin]$ jps 37618 DataNode 37783 SecondaryNameNode 37932 ResourceManager 37485 NameNode 38045 NodeManager 38365 Jps NodeManager没启动/启动后进程消失 该情况由于yarn-site.xml文件配置有误，导致从节点找不到主节点，报JavaNet异常\norg.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException: Problem binding to [hadoop0:0] java.net.BindException: 无法指定被请求的地址; For more details see: http://wiki.apache.org/hadoop/BindException at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139) at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65) at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:406) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:266) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:475) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:521) 解决方案 重新配置yarn-site.xml，确保其有下面这一段，并且使用SSH 主机名能够访问该节点\n\u0026lt;!--ResourceManager主机名--\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;yarn.resourcemanager.hostname\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;hadoop0\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; YARN运算IO错误 该错误由于机器内存过小，却执行复杂的运算，导致内存不足，报JavaIO异常，例如在2G内存的机器上执行 hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 4 5运算\nJob Finished in 34.889 seconds java.io.FileNotFoundException: File does not exist: hdfs://192.168.79.129:8020/user/hadoop/QuasiMonteCarlo_1524323663509_1454922894/out/reduce-out at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1219) at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211) at org.apache.hadoop.io.SequenceFile$Reader.\u0026lt;init\u0026gt;(SequenceFile.java:1750) at org.apache.hadoop.io.SequenceFile$Reader.\u0026lt;init\u0026gt;(SequenceFile.java:1774) at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314) at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) YARN也显示工作失败 解决方案 调整虚拟内存，或者减小运算精度，例如设置精度为hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 2 3则一般不会有问题\n机器时间不同步 该问题出现在MapReduce测试时，若主从节点机器时间不一致则会报同步错误\n18/04/21 23:55:20 INFO mapreduce.Job: Task Id : attempt_1524320196806_0003_m_000000_2, Status : FAILED Container launch failed for container_1524320196806_0003_01_000004 : org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container. This token is expired. current time is 1524333912535 found 1524326717165 Note: System times on machines may be out of sync. Check system time and time zones. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168) at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:159) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 解决方案 首先检查各节点的时间 主节点\n[hadoop@hadoop0 testFile]$ timedatectl status Local time: 六 2018-04-21 23:57:21 CST Universal time: 六 2018-04-21 15:57:21 UTC RTC time: 六 2018-04-21 15:44:44 Time zone: Asia/Shanghai (CST, +0800) NTP enabled: yes NTP synchronized: no RTC in local TZ: no DST active: n/a 从节点1\n[hadoop@hadoop1 hadoop]$ timedatectl status Local time: 日 2018-04-22 02:06:49 CST Universal time: 六 2018-04-21 18:06:49 UTC RTC time: 六 2018-04-21 17:54:14 Time zone: Asia/Shanghai (CST, +0800) NTP enabled: yes NTP synchronized: no RTC in local TZ: no DST active: n/a 从节点2\n[hadoop@hadoop2 hadoop]$ timedatectl status Local time: 日 2018-04-22 02:07:00 CST Universal time: 六 2018-04-21 18:07:00 UTC RTC time: 日 2018-04-22 04:00:48 Time zone: Asia/Shanghai (CST, +0800) NTP enabled: yes NTP synchronized: no RTC in local TZ: no DST active: n/a 三个节点显然时间不一致，使用ntpdate工具同步各机器时间\n[root@hadoop2 hadoop]# yum -y install ntp ntpdate 已加载插件：fastestmirror base | 3.6 kB 00:00:00 epel/x86_64/metalink | 7.7 kB 00:00:00 epel | 4.7 kB 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 (1/4): epel/x86_64/updateinfo | 916 kB 00:00:01 (2/4): extras/7/x86_64/primary_db | 185 kB 00:00:01 (3/4): epel/x86_64/primary_db | 6.3 MB 00:00:01 (4/4): updates/7/x86_64/primary_db | 6.9 MB 00:00:03 Determining fastest mirrors * base: centos.ustc.edu.cn * epel: mirror01.idc.hinet.net * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com 软件包 ntp-4.2.6p5-25.el7.centos.2.x86_64 已安装并且是最新版本 软件包 ntpdate-4.2.6p5-25.el7.centos.2.x86_64 已安装并且是最新版本 无须任何处理 [root@hadoop2 hadoop]# ntpdate cn.pool.ntp.org 22 Apr 12:02:55 ntpdate[36396]: step time server 108.59.2.24 offset 35618.521668 sec 将时间偏差写入机器时间\n[root@hadoop2 hadoop]# hwclock --systohc [root@hadoop2 hadoop]# date 2018年 04月 22日 星期日 12:03:33 CST 问题解决\n","date":"2018-04-22T10:39:00+08:00","permalink":"/archives/151/","title":"hadoop分布式搭建的一些坑"},{"content":"搭建环境 Windows VM虚拟机\n虚拟机配置 CPU:7700 单核 RAM:2G SSD:20G OS:CentOS 7.4\n环境配置 修改主机名 因为三台虚拟机均为一台虚拟机克隆而成，且源主机名为localhost，先计划将三台主机分别设置为hadoop0，hadoop1，hadoop2 IP、主机名、职能分别为： 192.168.79.129 hadoop0 NameNode DataNode ResourceManager NodeManager 192.168.79.130 hadoop1 DataNode NodeManager 192.168.79.131 hadoop2 DataNode NodeManager\nCentOS 7使用hostnamectl set-hostname hadoop0设置主机名，其他几台同样操作，输入后可通过hostnamectl status查看是否修改成功\n[root@hadoop0 ~]# hostnamectl status Static hostname: hadoop0 Icon name: computer-vm Chassis: vm Machine ID: 4eddbba909c84e159cfa474604412c3c Boot ID: df37e835fde94d039153028c9479b68e Virtualization: vmware Operating System: CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel: Linux 3.10.0-693.el7.x86_64 Architecture: x86-64 由于hadoop0是根节点，用于资源调度，故其需要具有所有DataNode的访问地址和权限，进入/etc/hosts文件，添加如下内容将主机名解析到对应的IP\n192.168.79.129 hadoop0 192.168.79.130 hadoop1 192.168.79.131 hadoop2 SSH免密码登录 进入当前用户的根目录/home/hadoop/.ssh/，在该目录下执行ssh-keygen -t rsa生成主机公钥，过程中弹出的除了 Overwrite (y/n)? y选择y确认覆盖之外，其余全为空值，则可生成免密码登录的公钥 【注意】Ubuntu系统操作略有特殊，建议使用CentOS系统进行操作\n[hadoop@hadoop0 root]$ cd /home/hadoop/.ssh/ [hadoop@hadoop0 .ssh]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): /home/hadoop/.ssh/id_rsa already exists. Overwrite (y/n)? y Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: SHA256:RjyGCGedx9xzxXG97KGgjXMIZ/pg7Q+tLU1nJoxIhm8 hadoop@hadoop0 The key\u0026#39;s randomart image is: +---[RSA 2048]----+ | . o. + . oo.o| | + .oo+ o . ...| | . o.= o . .| | . * + . + | | + S B . o .| | E B.* = . | | o +.=.= | | ++. | | .oo | +----[SHA256]-----+ 使用ssh-copy-id将本地主机的公钥复制到远程主机的authorized_keys文件上，此时先对本地主机hadoop0授权\n[hadoop@hadoop0 .ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop0 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: \u0026#34;/home/hadoop/.ssh/id_rsa.pub\u0026#34; The authenticity of host \u0026#39;hadoop0 (fe80::e99d:ad75:28e9:9276%ens33)\u0026#39; can\u0026#39;t be established. ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew. ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys hadoop@hadoop0\u0026#39;s password: Number of key(s) added: 1 Now try logging into the machine, with: \u0026#34;ssh \u0026#39;hadoop0\u0026#39;\u0026#34; and check to make sure that only the key(s) you wanted were added. 测试SSH免密码登录成功\n[hadoop@hadoop0 /]$ ssh hadoop0 Last login: Sat Apr 21 17:18:37 2018 [hadoop@hadoop0 ~]$ exit 登出 Connection to hadoop0 closed. 对另外两台主机访问授权 hadoop1\n[hadoop@hadoop0 root]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop1 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: \u0026#34;/home/hadoop/.ssh/id_rsa.pub\u0026#34; The authenticity of host \u0026#39;hadoop1 (192.168.79.130)\u0026#39; can\u0026#39;t be established. ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew. ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys hadoop@hadoop1\u0026#39;s password: Number of key(s) added: 1 Now try logging into the machine, with: \u0026#34;ssh \u0026#39;hadoop1\u0026#39;\u0026#34; and check to make sure that only the key(s) you wanted were added. hadoop2\n[hadoop@hadoop0 root]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop2 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: \u0026#34;/home/hadoop/.ssh/id_rsa.pub\u0026#34; The authenticity of host \u0026#39;hadoop2 (192.168.79.131)\u0026#39; can\u0026#39;t be established. ECDSA key fingerprint is SHA256:GEEH9kBVcDuaYpJZLNsaWZA3O47+HdqIbd91fGAWcew. ECDSA key fingerprint is MD5:cd:df:9e:8c:92:3a:9f:ae:b0:10:9b:b6:25:9c:4d:c6. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys hadoop@hadoop2\u0026#39;s password: Number of key(s) added: 1 Now try logging into the machine, with: \u0026#34;ssh \u0026#39;hadoop2\u0026#39;\u0026#34; and check to make sure that only the key(s) you wanted were added. 对另外两台主机做类似操作，即仅对本地主机授权，无需对hadoop0主机授权，操作后可查看authorized_keys文件内容，当前主机名即可 环境变量 三台主机均由之前的单节点主机克隆而来，故宝塔、java、hadoop环境是自带的 Java8\n[hadoop@hadoop0 .ssh]$ java -version java version \u0026#34;1.8.0_121\u0026#34; Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) [hadoop@hadoop0 .ssh]$ javac -version javac 1.8.0_121 hadoop\n[hadoop@hadoop0 .ssh]$ cd /www/hadoop [hadoop@hadoop0 hadoop]$ [hadoop@hadoop0 hadoop]$ ls hadoop-2.6.0-cdh5.7.0 hadoop-2.6.0-cdh5.7.0.tar.gz testFile tmp [hadoop@hadoop0 hadoop]$ source ~/.bash_profile $HADOOP_HOME\n[hadoop@hadoop0 hadoop]$ echo $HADOOP_HOME /www/hadoop/hadoop-2.6.0-cdh5.7.0 hadoop配置 若之前配置过hadoop，建议将原目录删除后重新解压，主节点和从节点的目录都要删除干净，否则会导致大量问题 配置主节点core-site.xml\n\u0026lt;configuration\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;fs.defaultFS\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;hdfs://192.168.79.129:8020\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;/configuration\u0026gt; 配置HDFS文件存放目录和节点数hdfs-site.xml\n\u0026lt;property\u0026gt; \u0026lt;name\u0026gt;dfs.namenode.name.dir\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;/www/hadoop/app/tmp/dfs/name\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;dfs.datanode.data.dir\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;/www/hadoop/app/tmp/dfs/data\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; 配置YARN的yarn.xml，删除单节点的配置，添加如下字段\n\u0026lt;configuration\u0026gt; \u0026lt;!-- Site specific YARN configuration properties --\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;yarn.nodemanager.aux-services\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;mapreduce_shuffle\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;yarn.log-aggregation-enable\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;true\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;!--ResourceManager主机名--\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;yarn.resourcemanager.hostname\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;hadoop0\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;/configuration\u0026gt; 配置从节点slave，默认slave为localhost\n[hadoop@hadoop0 hadoop]$ cat slaves localhost 加入host配置\n[hadoop@hadoop0 hadoop]$ cat slaves hadoop0 hadoop1 hadoop2 额外配置，启用jobhistory\n\u0026lt;configuration\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;mapreduce.framework.name\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;yarn\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;mapreduce.jobhistory.address\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;192.168.79.129:10020\u0026lt;/value\u0026gt; \u0026lt;description\u0026gt;MapReduce JobHistory Server IPC host:port\u0026lt;/description\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;mapreduce.jobhistory.webapp.address\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;192.168.79.129:19888\u0026lt;/value\u0026gt; \u0026lt;description\u0026gt;MapReduce JobHistory Server Web UI host:port\u0026lt;/description\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;mapreduce.jobhistory.done-dir\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;/history/done\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;mapreduce.jobhistory.intermediate-done-dir\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;/history/done_intermediate\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;/configuration\u0026gt; 【注意】配置好后将单节点时的/www/hadoop/hadoop-2.6.0-cdh5.7.0/logs文件夹删除或者直接重新解压hadoop的压缩包防止出现文件夹权限导致的错误\n资源分发 将主节点的所有配置分发到从节点，避免重复配置。在主节点执行如下命令即可\nscp -r /www/hadoop hadoop@hadoop1:/www scp -r /www/hadoop hadoop@hadoop2:/www 分发完成 去根节点查看文件是否已经传到，分别执行source ~/.bash_profile以启用hadoop命令\n注意分发前执行一次hadoop命令是报未找到命令，说明从节点的hadoop旧根目录是删除干净的，分发完成后再次执行命令就有效了 NameNode格式化 格式化只对主节点执行即可\nhdfs namenode -format 格式化成功 hadoop分布式启动 主节点启动，在/sbin目录下，执行./start-all.sh\n[hadoop@hadoop0 bin]$ cd ../sbin [hadoop@hadoop0 sbin]$ ./start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh 18/04/21 22:16:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Starting namenodes on [hadoop0] 从节点的datanode和namenode逐一启动\nhadoop0: starting namenode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop0.out hadoop1: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop1.out hadoop0: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop0.out hadoop2: starting datanode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop2.out Starting secondary namenodes [0.0.0.0] 主节点secondarynamenode启动\n0.0.0.0: starting secondarynamenode, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop0.out 18/04/21 22:16:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 启动yarn和resourcemanager、nodemanager\nstarting yarn daemons starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop0.out hadoop1: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop1.out hadoop2: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop2.out hadoop0: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop0.out hadoop分布式启动验证 命令行方式 使用jps查看进程，以下进程少一个都不行，若缺少，查看另一篇文章寻找解决办法hadoop分布式搭建的一些坑 [[hadoop分布式搭建的一些坑]]\n启动主节点后从节点的HDFS会自动启动，进入从节点，使用jps查看，有两个进程即成功\n[hadoop@hadoop2 hadoop]$ jps 29876 Jps 29575 DataNode 29692 NodeManager webUI方式 使用webUI确认所有进程成功启动，查看HDFS节点 查看YARN 已经有三个从节点注册到YARN 分布式集群测试 分布式文件测试 上传测试文件再查看节点情况 初始节点为空\n[hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ ls bin cloudera examples include libexec logs README.txt share bin-mapreduce1 etc examples-mapreduce1 lib LICENSE.txt NOTICE.txt sbin src [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls / 18/04/21 22:33:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 新建数据目录\n[hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -mkdir /data 18/04/21 22:33:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls / 18/04/21 22:33:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 1 items drwxr-xr-x - hadoop supergroup 0 2018-04-21 22:33 /data 上传小文件\n[hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -put README.txt /data 18/04/21 22:34:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [hadoop@hadoop0 hadoop-2.6.0-cdh5.7.0]$ hadoop fs -ls /data 18/04/21 22:34:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 1 items -rw-r--r-- 3 hadoop supergroup 1366 2018-04-21 22:34 /data/README.txt 上传大文件\n[hadoop@hadoop0 hadoop]$ ll 总用量 304284 drwxrwxr-x 3 hadoop hadoop 17 4月 21 22:16 app drwxrwxrwx 15 www www 253 4月 21 22:08 hadoop-2.6.0-cdh5.7.0 -rwxrwxrwx 1 www www 311585484 3月 31 23:06 hadoop-2.6.0-cdh5.7.0.tar.gz drwxr-xr-x 2 www www 301 4月 11 06:31 testFile [hadoop@hadoop0 hadoop]$ hadoop fs -put hadoop-2.6.0-cdh5.7.0.tar.gz /data 18/04/21 22:35:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [hadoop@hadoop0 hadoop]$ hadoop fs -ls /data 18/04/21 22:36:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 2 items -rw-r--r-- 3 hadoop supergroup 1366 2018-04-21 22:34 /data/README.txt -rw-r--r-- 3 hadoop supergroup 311585484 2018-04-21 22:36 /data/hadoop-2.6.0-cdh5.7.0.tar.gz 通过webUI查看文件情况 分布式MapReduce测试 测试MapReduce，进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce目录，执行pi运算hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 2 3 【注意】由于此处是虚拟机，不建议将后面的精度设置过大，否则会报IO错误 进入YARN的web界面查看 运行成功 分布式日志离线处理测试 上传日志数据 [hadoop@hadoop0 testFile]$ hadoop fs -put access.log /data 18/04/21 23:42:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform\u0026hellip; using builtin-java classes where applicable 执行\nhadoop jar hadoopstudy-0.6-jar-with-dependencies.jar com.fjy.hadoop.hadoop.project.LogApp /data/access.log /logaccess/out Web查看工作详情 查看结果 题外话 其实同时运行三台虚拟机是很吃配置的，特别是内存，对于hadoop来说内存太小很多都会报异常，因此本次的测试机器配置还可以的，三台虚拟机均运行在480G SATA3的固态上 CPU:7700 4core 8thread RAM:16G DDR4 Dual Channal SSD:128+480 nvme+sata3 HDD:9T sata3 Eth:1000M I219V 但在实际运行中依然顶着内存瓶颈，多次使用了系统盘的交换分区 ","date":"2018-04-21T18:10:00+08:00","permalink":"/archives/150/","title":"Hadoop分布式搭建测试"},{"content":"代码 GitHub链接 开源工具类UserAgentParser地址\n日志来源 根据之前的作业提交系统项目的后台获取，为了本地测试，故先从数据库中备份下来并做了适当的处理，仅保留UserAgent的相关信息 本地文件测试 使用开源工具类解析UserAgent中的信息\n@Test public void testReadFile() throws Exception { String path = \u0026quot;F:\\\\JAVA Workspace\\\\hadoopstudy\\\\access.log\u0026quot;; BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(new File(path)))); String line = \u0026quot;\u0026quot;; int count = 0; //模拟MapReduce存储 Map\u0026lt;String, Integer\u0026gt; browerMap = new HashMap\u0026lt;String, Integer\u0026gt;(); UserAgentParser userAgentParser = new UserAgentParser(); while (line != null) { line = reader.readLine(); count++; if (StringUtils.isNotBlank(line)) { String source = line; UserAgent agent = userAgentParser.parse(source); //测试 String browser = agent.getBrowser(); String engine = agent.getEngine(); String engineVersion = agent.getEngineVersion(); String os = agent.getOs(); String platform = agent.getPlatform(); boolean mobile = agent.isMobile(); Integer browserValue = browerMap.get(browser); if (browserValue != null) { browerMap.put(browser, browerMap.get(browser) + 1); } else { browerMap.put(browser, 1); } //输出解析的信息 System.out.println(browser + \u0026quot;,\u0026quot; + engine + \u0026quot;,\u0026quot; + engineVersion + \u0026quot;,\u0026quot; + os + \u0026quot;,\u0026quot; + platform + \u0026quot;,\u0026quot; + mobile); } } System.out.println(\u0026quot;总记录数：\u0026quot; + count); System.out.println(\u0026quot;====================================\u0026quot;); for (Map.Entry\u0026lt;String,Integer\u0026gt;entry :browerMap.entrySet()){ System.out.println(entry.getKey()+\u0026quot;:\u0026quot;+entry.getValue()); } } 执行单元测试 使用MapReduce统计 代码以WordCount为原型，结合本地测试的代码编写 \u0026hellip; boolean mobile = agent.isMobile();\nif (mobile){ mobileuser = \u0026quot;手机用户\u0026quot;; }else { mobileuser = \u0026quot;非手机用户\u0026quot;; } //通过上下文把map的处理结果输出 context.write(new Text(browser), one); ... 打包运行 将工程用mvn assembly:assembly命令把插件一起打包，上传至虚拟机，同时将log上传至HDFS的根目录 [hadoop@localhost testFile]$ hadoop fs -put access.log / 18/04/11 06:12:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform\u0026hellip; using builtin-java classes where applicable [hadoop@localhost testFile]$ hadoop fs -ls / 18/04/11 06:12:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform\u0026hellip; using builtin-java classes where applicable Found 9 items -rw-r\u0026ndash;r\u0026ndash; 1 hadoop supergroup 73 2018-04-11 00:30 /PartitionerTest.txt -rw-r\u0026ndash;r\u0026ndash; 1 hadoop supergroup 26888 2018-04-11 06:12 /access.log drwxr-xr-x - hadoop supergroup 0 2018-04-08 10:56 /hdfsapi -rw-r\u0026ndash;r\u0026ndash; 1 hadoop supergroup 60 2018-04-10 23:03 /hello.txt drwxrwx\u0026mdash; - hadoop supergroup 0 2018-04-11 01:28 /history drwxr-xr-x - hadoop supergroup 0 2018-04-11 00:35 /output drwxr-xr-x - hadoop supergroup 0 2018-04-08 10:35 /test drwx\u0026mdash;\u0026mdash; - hadoop supergroup 0 2018-04-11 01:36 /tmp drwxr-xr-x - hadoop supergroup 0 2018-04-09 20:11 /user 运行 结果与本地测试一致 [hadoop@localhost testFile]$ hadoop fs -ls /logaccess/browserout 18/04/11 06:15:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform\u0026hellip; using builtin-java classes where applicable Found 2 items -rw-r\u0026ndash;r\u0026ndash; 1 hadoop supergroup 0 2018-04-11 06:14 /logaccess/browserout/_SUCCESS -rw-r\u0026ndash;r\u0026ndash; 1 hadoop supergroup 32 2018-04-11 06:14 /logaccess/browserout/part-r-00000 [hadoop@localhost testFile]$ hadoop fs -text /logaccess/browserout/part-r-00000 18/04/11 06:15:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform\u0026hellip; using builtin-java classes where applicable Chrome 191 Firefox 10 Unknown 1 增加其他属性重新运行MapReduce，结果如下 [hadoop@localhost testFile]$ hadoop fs -ls /logaccess 18/04/11 06:33:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform\u0026hellip; using builtin-java classes where applicable Found 2 items -rw-r\u0026ndash;r\u0026ndash; 1 hadoop supergroup 0 2018-04-11 06:32 /logaccess/_SUCCESS -rw-r\u0026ndash;r\u0026ndash; 1 hadoop supergroup 192 2018-04-11 06:32 /logaccess/part-r-00000 [hadoop@localhost testFile]$ hadoop fs -text /logaccess/part-r-00000 18/04/11 06:33:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform\u0026hellip; using builtin-java classes where applicable 20100101 10 537.36 191 604.3.5 1 Android 48 Chrome 191 Firefox 10 Gecko 10 Linux 48 Unknown 1 Webkit 192 Windows 296 Windows 7 10 iPhone 1 iPhone OS 11.1 1 手机用户 49 非手机用户 153 日志离线统计完成！\n","date":"2018-04-20T22:58:00+08:00","permalink":"/archives/149/","title":"MapReduce对网站日志进行分析"},{"content":"官方原理解析 MapReduce演进 MapReduce2.x的架构 MapReduce1.x的架构，一旦JobTracker挂了，整个分布式节点就崩溃了，因此被淘汰 开发WordCount WordCount原理 数据输入到MapReduce，每个MapReduce任务都被初始化为一个Job，每个Job又可以分为两种阶段：map阶段和reduce阶段。map方法接收一个\u0026lt;key,value\u0026gt;形式的输入，key表示词组偏移量，value表示该偏移量对应的值，然后同样产生一个\u0026lt;key,value\u0026gt;形式的中间输出，Hadoop函数接收一个如\u0026lt;key,(list of values)\u0026gt;形式的输入，list of values即Java中的集合，用于表示相同偏移量下的字符统计数量，然后对这个value集合进行处理，每个Reduce方法产生0或1个输出，reduce的输出也是\u0026lt;key,value\u0026gt;形式的。 如果文本为\nHadoop Hello Hadoop 实际过程大概为：\nMapping：\n\u0026lt;Hadoop ,1\u0026gt;，\u0026lt;Hello,1\u0026gt;，\u0026lt;Hadoop ,1\u0026gt; Shuffling：\n\u0026lt;Hadoop ,1\u0026gt;，\u0026lt;Hadoop ,1\u0026gt; \u0026lt;Hello,1\u0026gt; Reducing：\n\u0026lt;Hadoop ,2\u0026gt; \u0026lt;Hello,1\u0026gt; 多节点下的并行计算原理示意图 开发代码 查看源码 通过查看Mapper类的源码，发现官方已经定义的run方法，包含了所有基础的Mapper操作（模板方法设计模式），主要操作在map中，因此，只要重写map方法即可。 同理Reduce类中也有类似的定义 重写方法 这里只列举部分代码，其余所有代码可去本人的GitHub中查看：GitHub链接 /** * Map：读取输入文件 * Text:类似字符串 */ public static class MyMapper extends Mapper\u0026lt;LongWritable, Text, Text, LongWritable\u0026gt; {\nLongWritable one = new LongWritable(1); /** * @param key 偏移量 * @param value 每行的字符串 * @param context 上下文 * @throws IOException * @throws InterruptedException */ @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { /*super.map(key, value, context);*/ //接收到每一行数据 String line = value.toString(); //按照指定分隔符进行拆分 /*line.split(\u0026quot;\\t\u0026quot;);//以Tab分隔*/ String[] words = line.split(\u0026quot; \u0026quot;);//以空格分隔 for (String word : words) { //通过上下文把map的处理结果输出 context.write(new Text(word), one); } } } 运行 通过宝塔面板上传到服务器，确保此时Hadoop所有服务均为启动状态 在HDFS中预先创建好要测试的文件，通过绝对路径的方法hadoop fs -ls hdfs://192.168.79.129:8020/确认文件是可以访问的 输入运行命令即可运行\nhadoop jar hadoopstudy-1.0-SNAPSHOT.jar com.fjy.hadoop.mapreduce.WordCountApp hdfs://192.168.79.129:8020/hello.txt hdfs://192.168.79.129:8020/output/wc 启动至FileInputFormat\n[hadoop@localhost testFile]$ hadoop jar hadoopstudy-1.0-SNAPSHOT.jar com.fjy.hadoop.mapreduce.WordCountA2.168.79.129:8020/hello.txt hdfs://192.168.79.129:8020/output/wc 18/04/10 09:57:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 18/04/10 09:57:44 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 18/04/10 09:57:45 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 18/04/10 09:57:45 INFO input.**FileInputFormat**: Total input paths to process : 1 执行Job\n18/04/10 09:57:46 INFO mapreduce.JobSubmitter: number of splits:1 18/04/10 09:57:46 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1523322221448_0001 18/04/10 09:57:47 INFO impl.YarnClientImpl: Submitted application application_1523322221448_0001 18/04/10 09:57:47 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1523322221448_0001/ 18/04/10 09:57:47 INFO mapreduce.Job: Running job: job_1523322221448_0001 18/04/10 09:57:56 INFO mapreduce.Job: Job job_1523322221448_0001 running in uber mode : false 18/04/10 09:57:56 INFO mapreduce.Job: map 0% reduce 0% 18/04/10 09:58:04 INFO mapreduce.Job: map 100% reduce 0% 18/04/10 09:58:10 INFO mapreduce.Job: map 100% reduce 100% 18/04/10 09:58:12 INFO mapreduce.Job: Job job_1523322221448_0001 completed successfully 18/04/10 09:58:12 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=39 FILE: Number of bytes written=222915 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=113 HDFS: Number of bytes written=17 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Map方法执行\nLaunched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=5920 Total time spent by all reduces in occupied slots (ms)=3321 Total time spent by all map tasks (ms)=5920 Total time spent by all reduce tasks (ms)=3321 Total vcore-seconds taken by all map tasks=5920 Total vcore-seconds taken by all reduce tasks=3321 Total megabyte-seconds taken by all map tasks=6062080 Total megabyte-seconds taken by all reduce tasks=3400704 Map-Reduce Framework Map input records=1 Map output records=2 Map output bytes=29 Map output materialized bytes=39 Reduce 方法执行\nInput split bytes=101 Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=39 Reduce input records=2 Reduce output records=2 Spilled Records=4 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=115 CPU time spent (ms)=1080 Physical memory (bytes) snapshot=300331008 Virtual memory (bytes) snapshot=5493092352 Total committed heap usage (bytes)=165810176 Shuffle 方法执行\nShuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 `File Input`和`File Output` 方法执行 File Input Format Counters Bytes Read=12 File Output Format Counters Bytes Written=17 运行结果如图 进一步优化 使用Combiner 使用Combiner组件可以在Mapping时就先进行一次聚合，再发送到Shuffling继续运算，从而提高运算效率 在主方法中进行如下配置即可\n//通过job设置combiner处理类，逻辑上与reduce一致，注意，如果要计算平均数等不能使用Combiner! job.setCombinerClass(MyReducer.class); 编写完后使用Maven重新打包\nmvn clean package -DskipTests 上传到服务器后再次执行计算命令\nhadoop jar hadoopstudy-0.3.jar com.fjy.hadoop.mapreduce.WordCountApp hdfs://192.168.79.129:8020/hello.txt hdfs://192.168.79.129:8020/output/wc 查看图形界面可观察是否完成 通过查看代码可发现Combine input records已经不为0(之前未使用Combiner 时该值为0)，说明使用成功\nMap-Reduce Framework Map input records=4 ... Combine input records=9 Combine output records=5 ... Physical memory (bytes) snapshot=298844160 Virtual memory (bytes) snapshot=5493088256 Total committed heap usage (bytes)=165810176 Shuffle Errors BAD_ID=0 ... 查看结果\n[hadoop@localhost testFile]$ hadoop fs -cat /output/wc/part-r-00000 18/04/10 23:06:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ! 1 Hadoop 3 Hello 2 MapReduce 2 YARN 1 使用Partitioner组件 作用 Partitioner决定MapTask输出的数据交由哪个ReduceTask处理，其默认实现为分发的key的hash值对ReduceTask个数取模\n文件准备 在宝塔测试目录/www/hadoop/testFile下准备好PartitionerTest.txt文件，文件内容如图，将其上传到HDFS根目录中 [hadoop@localhost root]$ cd /www/hadoop/testFile [hadoop@localhost testFile]$ ls PartitionerTest.txt hello.txt [hadoop@localhost testFile]$ hadoop fs -put PartitionerTest.txt / 18/04/11 00:30:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [hadoop@localhost testFile]$ hadoop fs -ls / 18/04/11 00:30:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 7 items -rw-r--r-- 1 hadoop supergroup 73 2018-04-11 00:30 /PartitionerTest.txt drwxr-xr-x - hadoop supergroup 0 2018-04-08 10:56 /hdfsapi -rw-r--r-- 1 hadoop supergroup 60 2018-04-10 23:03 /hello.txt drwxr-xr-x - hadoop supergroup 0 2018-04-10 23:05 /output drwxr-xr-x - hadoop supergroup 0 2018-04-08 10:35 /test drwx------ - hadoop supergroup 0 2018-04-09 20:11 /tmp drwxr-xr-x - hadoop supergroup 0 2018-04-09 20:11 /user [hadoop@localhost testFile]$ hadoop fs -text /PartitionerTest.txt 18/04/11 00:30:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable xiaomi 200 huawei 300 xiaomi 100 huawei 200 iphone 300 iphone 200 sony 50 代码编写 主要对分隔符和上下文内容进行修改，map类中修改为\n//每一个空格是一个手机品牌，另一个是销售数量 context.write(new Text(words[0]), new LongWritable(Long.parseLong(words[1]))); 新增Partitioner处理类\n/** * Partitioner处理类 */ public static class MyPartitioner extends Partitioner\u0026lt;Text,LongWritable\u0026gt;{ @Override public int getPartition(Text key, LongWritable value, int i) { if (\u0026#34;xiaomi\u0026#34;.equals(key.toString())) { return 0;//若为xiaomi则交由0 ReduceTask处理 } if (\u0026#34;huawei\u0026#34;.equals(key.toString())) { return 1;//若为huawei则交由1 ReduceTask处理 } if (\u0026#34;iphone\u0026#34;.equals(key.toString())) { return 2;//若为iphone则交由2 ReduceTask处理 } return 3;//若为其他，则交由3 ReduceTask处理 } } 最后在驱动中配置Partitioner\n//设置job的Partition job.setPartitionerClass(MyPartitioner.class); //设置四个reducer，每个分区一个，否则Partitioner配置不生效 job.setNumReduceTasks(4); 通过复制类名全路径（以IDEA为例），修改运行命令 此时，运行命令为\nhadoop jar hadoopstudy-0.4.jar com.fjy.hadoop.mapreduce.WordCountPartitionerApp hdfs://192.168.79.129:8020/PartitionerTest.txt hdfs://192.168.79.129:8020/output/partitioner 执行 再次用Maven打包并上传到测试目录运行 运行前： 运行中： 运行结束 查看结果内容：\n[hadoop@localhost testFile]$ hadoop fs -ls /output/partitioner 18/04/11 00:36:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 5 items -rw-r--r-- 1 hadoop supergroup 0 2018-04-11 00:35 /output/partitioner/_SUCCESS -rw-r--r-- 1 hadoop supergroup 11 2018-04-11 00:35 /output/partitioner/part-r-00000 -rw-r--r-- 1 hadoop supergroup 11 2018-04-11 00:35 /output/partitioner/part-r-00001 -rw-r--r-- 1 hadoop supergroup 11 2018-04-11 00:35 /output/partitioner/part-r-00002 -rw-r--r-- 1 hadoop supergroup 8 2018-04-11 00:35 /output/partitioner/part-r-00003 [hadoop@localhost testFile]$ hadoop fs -text /output/partitioner/part-r-00000 18/04/11 00:36:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable xiaomi 300 [hadoop@localhost testFile]$ hadoop fs -text /output/partitioner/part-r-00001 18/04/11 00:36:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable huawei 500 [hadoop@localhost testFile]$ hadoop fs -text /output/partitioner/part-r-00002 18/04/11 00:37:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable iphone 500 [hadoop@localhost testFile]$ hadoop fs -text /output/partitioner/part-r-00003 18/04/11 00:37:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable sony 50 结果正确！\n开启JobHistory JobHistory作用 记录已经运行完 的MapReduce信息到指定的HDFS目录下，但默认不开启 因此此时不配置直接访问如下\n![jobhistory][33]\n配置JobHistory 在mapred-site.xml加入配置\n\u0026lt;!-- jobhistory地址--\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;mapreduce.jobhistory.address\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;192.168.79.129:10020\u0026lt;/value\u0026gt; \u0026lt;description\u0026gt;MapReduce JobHistory Server IPC host:port\u0026lt;/description\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;!-- jobhistory web的地址--\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;mapreduce.jobhistory.webapp.address\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;192.168.79.129:19888\u0026lt;/value\u0026gt; \u0026lt;description\u0026gt;MapReduce JobHistory Server Web UI host:port\u0026lt;/description\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;!-- 作业运行完存放地址--\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;mapreduce.jobhistory.done-dir\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;/history/done\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;!-- 作业运行中存放地址--\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;mapreduce.jobhistory.intermediate-done-dir\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;/history/done_intermediate\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; 配置完后重启YARN\n[hadoop@localhost mapreduce]$ cd /www/hadoop/hadoop-2.6.0-cdh5.7.0/sbin [hadoop@localhost sbin]$ ./stop-yarn.sh stopping yarn daemons stopping resourcemanager localhost: stopping nodemanager no proxyserver to stop [hadoop@localhost sbin]$ jps 2353 DataNode 80224 Jps 2228 NameNode 2522 SecondaryNameNode [hadoop@localhost sbin]$ ./start-yarn.sh starting yarn daemons starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out localhost: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-localhost.localdomain.out [hadoop@localhost mapreduce]$ jps 2353 DataNode 80336 ResourceManager 2228 NameNode 81510 Jps 2522 SecondaryNameNode 63582 JobHistoryServer 80447 NodeManager 启动jobhistory服务\n[hadoop@localhost sbin]$ ./mr-jobhistory-daemon.sh start historyserver starting historyserver, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/mapred-hadoop-historyserver-localhost.localdomain.out 运行PI测试\n[hadoop@localhost sbin]$ cd /www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce [hadoop@localhost mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 1 2 访问报错 进入日志查看，发现仍然无法访问 报未开启聚合的错误，同时发现指向跳转链接为localhost 增加配置 因此在yarn-site.xml加入配置\n\u0026lt;!-- 开启YARN log的聚合功能--\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;yarn.log-aggregation-enable\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;true\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; 同时，为了让虚拟机的跳转链接不为localhost，此处再加入一段配置，制定IP\n\u0026lt;property\u0026gt; \u0026lt;name\u0026gt;yarn.nodemanager.hostname\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;192.168.79.129\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; 重启服务 再次重启YARN，并同时重启jobhistory服务\n[hadoop@localhost sbin]$ ./mr-jobhistory-daemon.sh stop historyserver stopping historyserver [hadoop@localhost sbin]$ ./stop-yarn.sh stopping yarn daemons stopping resourcemanager localhost: stopping nodemanager no proxyserver to stop [hadoop@localhost sbin]$ ./start-yarn.sh starting yarn daemons starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out localhost: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-localhost.localdomain.out [hadoop@localhost sbin]$ ./mr-jobhistory-daemon.sh start historyserver starting historyserver, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/mapred-hadoop-historyserver-localhost.localdomain.out 【注意】此时若只重启YARN就算配置正确，仍然可能出现Aggregation is not enabled的错误\n验证 再次执行PI运算\n[hadoop@localhost sbin]$ cd /www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce [hadoop@localhost mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 1 2 查看Logs记录 已经有日志记录，配置成功！ ","date":"2018-04-17T13:03:00+08:00","permalink":"/archives/146/","title":"单节点MapReduce开发"},{"content":"问题产生 最近有一个新的业务系统上线运行，源站使用8085端口，通过Nginx反向代理到80和443端口，为了保证全国的延迟在50ms以内，故启用CDN加速，仅对js和css等静态文件加速，其余文件自动回源，然而一旦开启了静态文件CDN后，网页便无法渲染（Vue），通过控制台查看，报重定向（301）过多的错误 一旦关闭CDN对js和css的加速后再清除浏览器缓存就立刻可以访问 寻找解决办法 经过网上查阅资料无果后向阿里云提交工单咨询 过程十分曲折，在阿里云工程师的指引下反复修改CDN各种配置，从上午一直修改到晚上10点，更换了3个工程师（从工号判断），其中还包括阿里工程师直接打电话询问问题情况（好评） 测试途径包括但不限于：跟随回源协议、抓包等 目前折衷方案 根据阿里的建议，现在只能暂停对静态文件的加速，全部回源到源站 造成的影响 2018年4月9日全天整个业务系统通过域名访问处于崩溃状态，对所有需要访问的客户重定向到IP+端口号的地址，并且该地址还一直被微信拦截（微信拦截非80端口的网址）\n","date":"2018-04-09T22:54:00+08:00","permalink":"/archives/145/","title":"阿里云CDN加速的一些经历"},{"content":"配置YARN 首先启动之前配置好的HDFS服务，然后进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/etc/hadoop目录，按照官方指导配置YARN单节点环境 使用cp mapred-site.xml.template mapred-site.xml命令把mapred-site.xml.template复制一份，加入官方的配置 mapreduce.framework.name yarn 同时对yarn-site.xml也进行配置 yarn.nodemanager.aux-services mapreduce_shuffle 启动YARN 进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/sbin目录，使用./start-yarn.sh命令启动YARN服务\n[hadoop@localhost sbin]$ ./start-yarn.sh starting yarn daemons starting resourcemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-localhost.localdomain.out localhost: starting nodemanager, logging to /www/hadoop/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-localhost.localdomain.out 用jps命令查看是否启动成功，看到如下显示即为启动成功，主要是NodeManager和ResourceManager两个服务\n[hadoop@localhost sbin]$ jps 7056 SecondaryNameNode 7475 NodeManager 6775 NameNode 6872 DataNode 7374 ResourceManager 7679 Jps 开放端口 YARN的浏览器客户端占用8088端口，故使用宝塔面板开放8088端口 访问YARN图形化界面 使用浏览器打开http://ip:8088/可见如下界面，则YARN启动成功 测试 YARN官方提供了示例java程序来测试资源调度，进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce目录，可见\n[hadoop@localhost mapreduce]$ ls hadoop-mapreduce-client-app-2.6.0-cdh5.7.0.jar hadoop-mapreduce-client-nativetask-2.6.0-cdh5.7.0.jar hadoop-mapreduce-client-common-2.6.0-cdh5.7.0.jar hadoop-mapreduce-client-shuffle-2.6.0-cdh5.7.0.jar hadoop-mapreduce-client-core-2.6.0-cdh5.7.0.jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar hadoop-mapreduce-client-hs-2.6.0-cdh5.7.0.jar lib hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.7.0.jar lib-examples hadoop-mapreduce-client-jobclient-2.6.0-cdh5.7.0.jar sources hadoop-mapreduce-client-jobclient-2.6.0-cdh5.7.0-tests.jar 其中hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar就是MapReduce的测试程序，通过hadoop jar的方式来运行 用命令查看可配置的参数hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar\n[hadoop@localhost mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar An example program must be given as the first argument. Valid program names are: aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files. aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files. bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi. dbcount: An example job that count the pageview counts from a database. distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi. grep: A map/reduce program that counts the matches of a regex in the input. join: A job that effects a join over sorted, equally partitioned datasets multifilewc: A job that counts words from several files. pentomino: A map/reduce tile laying program to find solutions to pentomino problems. pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method. randomtextwriter: A map/reduce program that writes 10GB of random textual data per node. randomwriter: A map/reduce program that writes 10GB of random data per node. secondarysort: An example defining a secondary sort to the reduce. sort: A map/reduce program that sorts the data written by the random writer. sudoku: A sudoku solver. teragen: Generate data for the terasort terasort: Run the terasort teravalidate: Checking results of terasort wordcount: A map/reduce program that counts the words in the input files. wordmean: A map/reduce program that counts the average length of the words in the input files. wordmedian: A map/reduce program that counts the median length of the words in the input files. wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files. [hadoop@localhost mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi Usage: org.apache.hadoop.examples.QuasiMonteCarlo \u0026lt;nMaps\u0026gt; \u0026lt;nSamples\u0026gt; Generic options supported are -conf \u0026lt;configuration file\u0026gt; specify an application configuration file -D \u0026lt;property=value\u0026gt; use value for given property -fs \u0026lt;local|namenode:port\u0026gt; specify a namenode -jt \u0026lt;local|resourcemanager:port\u0026gt; specify a ResourceManager -files \u0026lt;comma separated list of files\u0026gt; specify comma separated files to be copied to the map reduce cluster -libjars \u0026lt;comma separated list of jars\u0026gt; specify comma separated jar files to include in the classpath. -archives \u0026lt;comma separated list of archives\u0026gt; specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] 从上可知测试程序可以做诸如数学π、随机文件写入等操作，这里做数学π的运算 使用命令hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 2 3做两个map计算精度为3的数学π运算,运算结束后会显示结果如下\nJob Finished in 26.996 seconds Estimated value of Pi is 4.00000000000000000000 同时浏览器也可看到作业情况 ","date":"2018-04-09T20:14:00+08:00","permalink":"/archives/144/","title":"YARN单节点伪分布式搭建"},{"content":"无法创建、上传文件 问题描述 当前hdfs文件结构如下，现通过JavaAPI远程创建目录和文件 创建目录 执行单元测试成功 通过API写文件 发现迟迟没有完成（可能有问题了，因为是虚拟机，延迟不应这么大） 测试不通过 报错为 org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /hdfsapi/test/api.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1595) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3287) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:677) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:213) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:485) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)\nat org.apache.hadoop.ipc.Client.call(Client.java:1471) at org.apache.hadoop.ipc.Client.call(Client.java:1408) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) at com.sun.proxy.$Proxy12.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:404) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy13.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1704) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1500) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:668) Process finished with exit code -1 查看文件目录，发现有该文件，但大小为0，显然是文件流无法写入 解决方法 经过爬贴寻找，发现是因为没有开放端口导致文件流无法写入，经过查找，发现Hadoop相关的端口很多，为了便于学习，直接关闭虚拟机防火墙即可，CentOS7 的命令为sudo systemctl stop firewalld.service\n关闭防火墙后重新测试，测试通过 再次查看文件系统，发现文件大小已经不为0，说明文件内容已写入，用hadoop fs -cat /hdfsapi/test/api.txt查看文件内容，发现内容一致，问题解决 ","date":"2018-04-08T11:10:00+08:00","permalink":"/archives/143/","title":"Hadoop伪分布式的一些坑"},{"content":"下载相关插件 文章计数：下载地址 评论微信提醒：下载地址\n上传插件到服务器 将下载好的插件上传到typecho下的usr/plugins目录并解压，将解压后的评论微信提醒文件夹目录名改为Comment2Wechat 使用admin账号登录typecho控制台，启用刚刚添加的插件 文章计数 设置需要显示计数的地方 进入外观设置，配置当前外观代码 根据官方文档说明，post.php\t为日志页面文件，故配置post.php 在文件中搜索Published on ，找到该行代码，在\u0026lt;/div\u0026gt;前加入\u0026amp;nbsp;\u0026lt;?php Views_Plugin::theViews(); ?\u0026gt;用于显示浏览计数，该文件共用4处地方需要增加代码，自行搜索即可 查看效果 进入文章页面，可见在日志末尾已加入浏览计数 评论微信提示 官方指引地址 根据提示登录GitHub账号即可得到一个KEY，此时还不能接收消息 继续按指引绑定微信账号，此时在KEY页面发送消息微信已经可以接收了 进入typecho插件设置，将获得的KEY绑定到插件 此时评论后马上就可以收到消息 ","date":"2018-04-07T23:04:00+08:00","permalink":"/archives/142/","title":"typecho添加文章计数和评论微信提醒"},{"content":"环境 系统：CentOS Linux 7.4.1708 (Core) 客户端：putty 服务器：虚拟机1core、2G RAM 同时搭建了宝塔面板（自动配置了java8）\nHadoop下载 CDH版本Hadoop下载\n为了统一版本，全部使用hadoop-2.6.0-cdh5.7.0 点击可进入官方推荐配置页 解压并配置 将下载好的文件通过宝塔上传到服务器，这里选择上传到/www/hadoop目录 解压tar包 配置 新建用户 新建hadoop用户，忽略密码强度提示\nadduser hadoop passwd hadoop 为方便远程连接和学习，同时授予root权限，输入whereis sudoers查看权限控制文件位置\n[root@fjy ~]# whereis sudoers sudoers: /etc/sudoers /etc/sudoers.d /usr/libexec/sudoers.so /usr/share/man/man5/sudoers.5.gz 使用宝塔找到并修改/etc/sudoers文件 配置新用户免密码远程登录 切换至hadoop用户，使用 ssh-keygen -t rsa生成私钥信息，位置在/home/hadoop/，并且忽略所有提示输入的信息 查看生成的文件，该SSH文件夹为隐藏文件夹，故用ls -a查看，可见自动生成了.ssh目录和id_rsa,id_rsa.pub两个文件 复制公钥到.ssh目录\ncp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys 查看JAVA目录 因为是宝塔自动配置的java8，所以要首先查看当前java环境变量位置，使用如下命令\nupdate-alternatives --config java 此时宝塔自动将java配置到/usr/java/jdk1.8.0_121/jre/bin/java目录 即/usr/java/jdk1.8.0_121为java环境变量的目录\n配置hadoop的java目录 使用宝塔进入hadoop目录下的etc/hadoop/hadoop-env.sh，对应此处的绝对路径应为/www/hadoop/hadoop-2.6.0-cdh5.7.0/etc/hadoop 使用宝塔编辑文件，加入java环境变量位置 配置分布式文件参数 打开core-site.xml，配置如下\n\u0026lt;configuration\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;fs.defaultFS\u0026lt;/name\u0026gt; \u0026lt;!-- 配置主机IP--\u0026gt; \u0026lt;value\u0026gt;hdfs://192.168.79.129:8020\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;!-- 配置临时文件位置--\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;hadoop.tmp.dir\u0026lt;/name\u0026gt; \u0026lt;value\u0026gt;/www/hadoop/tmp\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;/configuration\u0026gt; 打开hdfs-site.xml，配置如下\n\u0026lt;configuration\u0026gt; \u0026lt;property\u0026gt; \u0026lt;name\u0026gt;dfs.replication\u0026lt;/name\u0026gt; \u0026lt;!-- 配置副本数量--\u0026gt; \u0026lt;value\u0026gt;1\u0026lt;/value\u0026gt; \u0026lt;/property\u0026gt; \u0026lt;/configuration\u0026gt; 启动 初始化 进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/bin目录，使用如下语句初始化Hadoop，该操作仅执行一次即可，以后使用无需重复执行，否则执行后会删除当前hadoop上的所有文件\nhadoop namenode -format 滚动均为info级别的日志 可看到已初始化成功 启动HDFS 切换到hadoop用户，进入/www/hadoop/hadoop-2.6.0-cdh5.7.0/sbin目录 输入./start-dfs.sh启动HDFS服务 【注意】，要提前在root用户下对/www/hadoop授予hadoop用户读写权限，否则无法启动 使用jps查看当前的java进程，若出现如下四个进程表示HDFS服务启动成功\n外部访问浏览器客户端 首先使用宝塔开放50070端口 使用外部浏览器访问ip:50070可查看如下页面，则表示HDFS已成功启动 hdfs命令 配置hdfs命令 使用vi ~/.bash_profile进入配置文件，按如下配置\nexport HADOOP_HOME=/www/hadoop/hadoop-2.6.0-cdh5.7.0 export PATH=$HADOOP_HOME/bin:$PATH 使配置文件生效source ~/.bash_profile 输入hdfs回车测试命令是否生效，若出现如下画面则表示命令已经生效 命令测试 先在本地创建hello.txt文件，使用 hadoop fs -put hello.txt /命令将文件推送到hdfs根目录下 查看服务器根目录hadoop fs -ls /，已经有该文件，hadoop伪分布式配置完成 ","date":"2018-04-07T13:01:00+08:00","permalink":"/archives/141/","title":"HDFS伪分布式单节点搭建"},{"content":"修改前 一般安装好typecho和主题后默认的底部信息如下，此时发现没有显示ICP备案信息 修改目的 根据国家法律法规要求备案后，是需要在页面底部增加ICP备案信息和链接的，否则将可能会面临相应的处罚 修改教程 首先以admin账号登录控制台，选择修改外观 选择编辑当前外观，打开footer.php 找到大约位于32行的info-textdiv标签，在\u0026amp;copy; 后加入如下代码，备案号换成自己的备案号即可\n\u0026lt;p\u0026gt;ICP\u0026lt;a href=\u0026quot;http://www.miitbeian.gov.cn/\u0026quot; target=\u0026quot;_blank\u0026quot;\u0026gt; 粤ICP备18026XXX号\u0026lt;/a\u0026gt;\u0026lt;/p\u0026gt; 最后点击保存即可 最终效果 访问有ICP备案号的博客链接 ","date":"2018-03-17T00:01:00+08:00","permalink":"/archives/136/","title":"Typecho  Pinghsu 主题加入备案信息"},{"content":"前提条件 使用国内CDN加速服务前提条件几乎都是：网站必须已经备案\nCDN作用 CDN（云分发）可以结合优质的网络节点资源、传输路由技术优势，有效降低用户访问延迟，提升可用性。具有按需计费，资源弹性扩容等特点。 说白了就是能加快访问速度\n教程 开启阿里云CDN服务 在阿里云处购买CDN服务，考虑到仅仅为了学习和测试，因此本人的CDN服务购买流量为20G，开通后会有邮件提醒 配置阿里云CDN 在阿里云首页登录控制台，进入CDN控制台 选择域名管理，添加需要加速的域名 逐步配置 配置HTTPS，上传SSL证书 配置好后等待1-2分钟会生成CDN加速的域名 复制加速域名，登录自己的域名控制台，设置加速域名的解析 配置加速域名解析 解析为CNAME，即解析到另一域名 配置好后等待20-30分钟才会生效\n检查加速域名解析是否生效 这里使用Windows powershell命令行工具ping目标地址，若解析到**.kunluncan.com则已经生效 监控 在多次访问后查看监控才会有流量显示，包含命中率和流量信息 多地访问测试 使用第三方网站进行全国多地的ping测试，可看到原本只有一个固定ip的博客网站已经被加速为34+个CDN加速IP地址，国内访问速度均在50ms以内 ","date":"2018-03-16T23:52:00+08:00","permalink":"/archives/135/","title":"阿里云CDN加速使用方法"},{"content":"起因 恰逢学校组织了一次志愿者活动，刚好周末没什么事，便带着出去活动下的目的报了名 过程 活动 3月10日上午7点集合，于8点准时到达目标学校，到达后主要是陪跑和陪伴各种不同性格的特殊儿童进行玩耍，许多特殊儿童的年龄其实都不小，基本上都在15-20岁左右，从外表上看他们与普通儿童并无显著差异，但由于天生缺陷，运动能力和特别是语言能力都比常人差了很多，因此极大限制了他们的沟通交流和思想的表达 时间短暂，但却能由此发现他们的内心世界并不消极，他们在天生不足的条件下依然努力生活，尽量学习自理能力，以及还要面对以后的融入社会和找工作的问题。 告别 在最后离去的时候，依然有些许不舍 再次回到题目，我们生来便比他们的条件更好，却不懂得珍惜\n","date":"2018-03-11T13:00:00+08:00","permalink":"/archives/134/","title":"我们一直过得更好，却不知珍惜"},{"content":"订书系统介绍 订书系统（点击访问） 访问后使用学号+姓名即可登录 登录后能够查看全班订书情况，总价为服务器自动计算 点击添加/修改即可为自己增加或修改订书信息 同时每位同学可查看后台提供的书籍价格和对应的课程名 也可查看实时公告信息 后台人员登录后可以修改价格，查看各类订书总数，对订书时间进行控制，当时间超过所设定日期后则不允许订书 后台也可以查看所有IP登录日志和设备信息，用于判断异地登录和系统审计，顺带一提，后面真有一次有人盗用账号进行登录修改，通过IP审计的方式找出登录地点和操作人，成功解决非法修改信息的问题 经过两个版本的更新后，系统增加了支付信息，后台可以根据支付宝记录更改支付状态 用户通过点击价格表，扫描其中的支付宝二维码即可支付书费，支付一段时间后经过确认会更新支付状态 开发进度 经过最后两次更新后，因多种原因，开发人员打算放弃开发和维护，愿有意接手维护和开发的人员与作者联系\n已知BUG 价格计算页面因二进制移位问题，导致其中一位同学的价格产生多位小数的错误 ","date":"2018-03-02T10:00:00+08:00","permalink":"/archives/129/","title":"班级订书系统"},{"content":"起因 在2017年下半学年收集作业的过程中，遇到经常需要统一命名格式的情况，十分繁琐，也有出现许多深夜、凌晨提交作业的情况，同时也有不少要求找回以往提交的作业的情况，故考虑开发作业提交系统，满足作业收集需求\n系统成品 作业提交系统正式版（点击访问正式版网站本体）\n作业提交系统测试服务器版本（点击访问测试服务器网站本体）\n系统使用说明（点击访问）\n系统优点 系统主要解决以下几大核心问题\n全天候接受作业提交并及时作出响应 作业自动按学号+姓名命名 作业按列表进行展示 用户可下载以往提交的作业和模板 后台自动获取所有未提交人员名单 系统延迟和访问速度 在购买了阿里云CDN加速后，通过在线网络测试工具得到如下结果 全球的平均延迟在50ms以内 同时进行国内多地访问时的下载速度能稳定在100K以上，若遇到打开缓慢请稍等片刻或者刷新页面 开发准备 确定立项后首先对所需架构进行选型，考虑范围包括：\n传统Java MVC、Struct、SpringMVC、SpringBoot、Mybatis、Hibernate bootstrap4.0、Vue、ElementUI、jQuery、原生ajax、axios、vue-resource thymeleaf、JSP\n最后根据各方优缺点决定使用Java开发 Java版本（使用了java8 的新特性，故不对低版本的jdk兼容）\nJava 8或以上\n后台使用\nSpring Boot2.0RC2\n数据库层使用\nSpringJpa\n前端界面组件使用\nElementUI\n前端底层数据响应式框架\nVue ajax使用Vue官方推荐的库 axios\n开发过程 开发系统生命周期选型采用螺旋模型，通过反复迭代的方式进行持续改进 由于切换新的架构（以前订书系统是纯jsp处理所有业务，Java MVC对于注解使用十分少），同时注重对数据库的范式设计（3NF），因此在开发准备花了不少时间，以及在后续开发遇到不少问题，耗时极大，根据私人Git仓库GitLab的记录，2018年1月29号开始代码编写V0.1，直到2018年2月27号 版本号V2.2 RC1网站正式上线公测持续接近1个月时间，目前（2018年2月28日）仍在对bug进行测试和修复 GitLab截图 后台部分代码截图 所有工程文件使用Git进行版本配置管理，并上传到私人代码仓库 所有jar包使用maven进行统一管理 下载的Controller 所有Controller 通过使用枚举对异常统一处理，并编写自定义异常类 对日志进行科学管理，使用SLF4j、Logback 日志框架对所有info和error日志进行每天一个文件的日志输出 单元测试 前端部分代码截图 使用Spring Boot官方推荐的thymeleaf模板引擎，其性能比JSP更优，而且也有与JSTL类似的语法，可以轻松实现组件化和动态更新数据，并能够保证前端所有页面统一为纯HTML axios异步获取后台传来的json数据，并使用Vue做数据绑定 前端使用饿了么的ElementUI组件进行快速开发，遗憾的是未使用ES6和Webpack 访问权限控制 通过使用Spring Boot拦截器，实现访问控制列表（Access Control List，ACL）与强制访问控制(Mandatory Access Control——MAC)结合的访问控制机制，对所有非注册用户和非管理员用户进行分辨和拦截 开发文档截图 后台提供的API文档 Restful风格的API由后台提供，前端可直接使用，实现前后端分离开发 后台提供的错误码文档 数据字典 数据库ER图 按照3NF规范进行开发，充分使用数据库视图作为外模式进行访问 服务器配置 硬件服务器 服务器采用由阿里云购买的轻量应用服务器，配置如下\n1核CPU Intel Xeon E5-2682 v4 2G DDR4 内存 1M上行带宽 VPC专有网络 40GB系统盘 SSD云盘 硬件服务器评价 整体配置基本满足系统运行需求，根据Beta测试阶段的数据，高并发时单核CPU瓶颈会显现，内存维持50%左右的水平，磁盘读写也令人满意，上行带宽略有不足，但由于为了保证系统高并发下的稳定性优先，以及系统主要用于用户上传，服务器下载速度没有瓶颈即可 服务器的安全策略 服务器的安全策略为 服务器每天凌晨自动备份网站存储的文件和数据库到阿里云OSS和又拍云FTP存储空间，确保收到攻击后能以最快速度恢复所有用户文件和数据\n★[2018/03/14 04:31:01] 数据库[db_imis2]已成功备份到阿里云OSS,用时[0.21]秒 |---保留最新的[3]份备份 |---文件名:Db_db_imis2_20180314_043101.sql.gz |---已清理过期备份文件：Db_db_imis2_20180311_043102.sql.gz ★[2018/03/15 04:31:02] 数据库[db_imis2]已成功备份到阿里云OSS,用时[0.21]秒 |---保留最新的[3]份备份 |---文件名:Db_db_imis2_20180315_043101.sql.gz |---已清理过期备份文件：Db_db_imis2_20180312_043101.sql.gz ★[2018/03/12 04:30:01] 网站[cms.fjy8018.xin]已成功备份到阿里云OSS,用时[0.23]秒 |---保留最新的[2]份备份 |---文件名:Web_cms.fjy8018.xin_20180312_043001.tar.gz Web服务器 考虑到业务需要轻量级，占有内存少，并发能力强的软件平台，故服务器使用nginx1.12作为Web服务器，并通过配置实现反向代理和负载均衡 upstream cms-host{ server cms.fjy8018.xin:8085; #负载均衡 } server { listen 80; listen 443 ssl; server_name cms.fjy8018.xin; #域名 index index.html; #指定的server的root的访问页面 root /index; #指定的server的root目录 if ($server_port !~ 443){ rewrite ^(/.*)$ https://$host$1 permanent; } location /cms { proxy_pass http://cms.fjy8018.xin:8085; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Port $server_port; # root html; # index index.html index.htm; } } 但由于没有使用分布式文件系统，故实际负载仍然为一台阿里云服务器\n域名备案 为了保证微信能够直接访问，故通过购买万网域名fjy8018.xin，在通信管理局进行ICP备案，经过6天的等待，终于获得ICP备案和备案号 启用HTTPS 通过免费获取TrustAsia证书，并在nginx中加入pem文件路径配置，让网址成功启用HTTPS安全协议 #HTTP_TO_HTTPS_END ssl_certificate XXX/cms/full_chain.pem; ssl_certificate_key XXX/cms/private.key; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:HIGH:!aNULL:!MD5:!RC4:!DHE; ssl_prefer_server_ciphers on; ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m; error_page 497 https://$host$request_uri;\n#SSL-END 开源想法 暂无对外公开源码的想法，若本班有感兴趣的可以一同交流学习，如有兴趣接手维护或者开发新的网页端、安卓端和IOS端的也可以和本人联系，本人将提供后台API文档。\n","date":"2018-02-27T19:50:00+08:00","permalink":"/archives/100/","title":"作业提交系统介绍"},{"content":"地址 不推荐使用手机访问或者使用微信内置浏览器访问 作业提交系统（点击访问）\n系统介绍（点击查看）\n注册 点击注册选项卡进行注册，若用户名已存在会返回相应的提示，若注册用户非15信管2班学生则不允许注册\n注册时务必牢记密码，任何人无法通过后台查询用户的密码，若容易忘记密码，建议登录后立即进入个人中心设置密保问题\n登录 点击登录选项卡进行登录，若用户和密码不匹配则会提示消息 作业上传 点击图中箭头位置的下拉按钮，展开作业列表可见作业详情 点击模板下载下方的链接可下载模板文件（edge浏览器下载文件名会乱码，文件内容不受影响） 点击选取文件按钮可选择上传文件 点击上传到服务器即可上传，若要取消上传则点击选中文件右侧的“×”即可 上传成功后会自动提示成功消息，并于1秒后自动刷新已提交文件列表 若文件已存在，则自动将源文件后缀加上.日期.bak，新文件自动按照学号+姓名命名 注意：操作按钮中的“删除”操作不会有“是否确认删除的提示”，请谨慎操作 修改个人信息和密保问题 点击个人中心，可在里面重置个人信息和密码 个人中心里可设置或修改密保问题，用于找回密码，问题答案可放心设置，网站对所有问题答案均进行SHA安全加密 关于页面 点击关于按钮和查看网站相关信息 点击“查看更新日志”可看到网站历史版本和所有内部技术更新信息 意见反馈 进入意见反馈和提交对网站的意见和建议以及遇到的bug，欢迎多多反馈，当然也可以给出对该网站的评价~ 登出 建议所有用户使用完后主动点击登出按钮销毁会话，登出后若不主动进行登录操作，任何人无法看到主页内容 已知兼容性问题 已对chrome进行完美测试，测试通过 已对QQ浏览器、edge浏览器、火狐浏览器进行登录测试，测试通过 已对360极速浏览器进行登录测试，必须在地址栏右侧选择“极速模式”才能正常访问 已对微信内置浏览器进行登录测试，微信会对非80端口的网站进行拦截，严重影响使用体验，不推荐使用微信内置浏览器进行访问\n网站未对手机显示进行优化，请使用电脑访问以获得最佳体验\n已知bug edge浏览器下载文件标题乱码\n版权 该网站完全由符嘉阳自主开发，运行于国内家庭服务器中\n","date":"2018-02-27T15:58:00+08:00","permalink":"/archives/99/","title":"作业提交系统使用说明"},{"content":"预告 2018-2-4 作业提交系统（班级管理系统）正式立项 预告图 ","date":"2018-02-04T12:48:00+08:00","permalink":"/archives/79/","title":"作业提交系统"},{"content":"环境准备 之前的文章讲到使用宝塔搭建树莓派运维环境，搭建nextcloud应用，现在在原有基础上搭建小型Java web服务\n工程准备 以下是在Windows环境中用myeclipse编写好的java web工程，将其打包为zip文件 上传工程文件 进入宝塔面板，选择文件，进入Tomcat文件目录/www/server/tomcat/webapps 上传压缩好的工程文件 点击命令行按钮，输入解压命令unzip OrderBooks.zip 解压完成后修改权限为777，所有者为www 修改Tomcat工程配置文件 进入Tomcat下的conf目录，找到``server.xml文件，配置工程的实际路径和地址栏显示路径，path后为显示路径，docBase```为实际路径，修改后保存 开放防火墙端口 进入安全面板，在防火墙中放行8080端口 启动Tomcat服务 进入软件管理面板，找到Tomcat 8 ，点击设置，选择重启 完成 浏览器中打开树莓派IP+8080端口号+server.xml配置的路径名即可看到Java web服务已正常运行 ","date":"2018-01-24T20:30:00+08:00","permalink":"/archives/73/","title":"使用宝塔部署Java Web服务"},{"content":"最终效果 步骤 之前的文章讲到使用宝塔运维树莓派Ubuntu16.04系统，现在在原有基础上搭建个人私有云服务nextcloud，与owncloud搭建方法几乎一致\n下载 百度搜索nextcloud即可找到官方下载地址，选择download，下载zip版本文件\n准备站点 进入宝塔面板，选择网站，点击添加站点 输入树莓派的局域网地址或域名，根目录为wwwroot 点击文件 默认打开的根目录即wwwroot,此处可见自动生成的站点文件 在浏览器中打开树莓派地址加/index.html可见站点创建成功 上传nextcloud 回到宝塔，点击上传，选择下载好的nextcloud文件，点击上传，直到显示已上传成功 点击命令行按钮 输入解压命令unzip nextcloud-12.0.4.zip，等待其自动解压 刷新根目录可见解压好的文件夹，右边显示所有者为root，显然是错误的 点击权限，对nextcloud文件夹授予775权限 准备数据库 点击数据库，查看数据库密码，若不是通过宝塔安装MySQL服务者无法在此查看密码 使用命令mysql -u root -p，输入密码进入数据库\nroot@fjy-desktop:/home/fjy# mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \\g. Your MySQL connection id is 4 Server version: 5.7.21-0ubuntu0.16.04.1 (Ubuntu)\nCopyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.\nOracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.\nType \u0026lsquo;help;\u0026rsquo; or \u0026lsquo;\\h\u0026rsquo; for help. Type \u0026lsquo;\\c\u0026rsquo; to clear the current input statement.\nmysql\u0026gt;\n输入如下命令创建nextcloud数据库和数据库用户 CREATE DATABASE nextclouddb; GRANT ALL ON nextclouddb.* TO nextclouddbuser@localhost IDENTIFIED BY 'nextcloud'; flush privileges;\nmysql\u0026gt; CREATE DATABASE nextclouddb; Query OK, 1 row affected (0.00 sec)\nmysql\u0026gt; GRANT ALL ON nextclouddb.* TO nextclouddbuser@localhost IDENTIFIED BY \u0026gt;\u0026lsquo;nextcloud\u0026rsquo;; Query OK, 0 rows affected, 1 warning (0.00 sec)\nmysql\u0026gt; flush privileges; Query OK, 0 rows affected (0.00 sec)\n准备php扩展环境 nextcloud还需其他的PHP扩展环境，运行命令apt-get install php-mysql php-json php-xml php-mbstring php-zip php-gd curl php-curl \u0026gt;php-pdo自动安装扩展\nroot@fjy-desktop:/home/fjy# apt-get install php-mysql php-json php-xml php-mbstring php-zip php-gd curl php-curl \u0026gt;php-pdo 正在读取软件包列表\u0026hellip; 完成 正在分析软件包的依赖关系树 正在读取状态信息\u0026hellip; 完成 注意，选中 \u0026lsquo;php7.0-common\u0026rsquo; 而非 \u0026lsquo;php-pdo\u0026rsquo; curl 已经是最新版 (7.47.0-1ubuntu2.5)。 将会同时安装下列软件： php-common php7.0-curl php7.0-gd php7.0-json php7.0-mbstring php7.0-mysql php7.0-xml php7.0-zip 下列【新】软件包将被安装： php-common php-curl php-gd php-json php-mbstring php-mysql php-xml php-zip php7.0-common php7.0-curl php7.0-gd php7.0-json php7.0-mbstring php7.0-mysql php7.0-xml php7.0-zip 升级了 0 个软件包，新安装了 16 个软件包，要卸载 0 个软件包，有 249 个软件包未被升级。 需要下载 1,508 kB 的归档。 解压缩后会消耗 6,490 kB 的额外空间。 选择Y 您希望继续执行吗？ [Y/n] Y\n安装nextcloud 地址栏输入树莓派IP+文件地址即可打开nextcloud页面，此处打开http://192.168.1.187/nextcloud 按照之前设置的信息进行配置，注意，若报SQLSTATE[HY000] [2002]错误，则将localhost改为127.0.0.1 配置后若不会自动跳转则自行输入http://192.168.1.187/nextcloud进行登录 登陆后nextcloud安装完成！ ","date":"2018-01-24T16:49:00+08:00","permalink":"/archives/53/","title":"使用宝塔安装nextcloud私有云服务"},{"content":"系统环境 硬件：树莓派3 CPU：四核1.2GHz Broadcom BCM2837 x64 RAM：1GB TF容量：32G U1速度 操作系统：ubuntu-mate-16.04.2-desktop-armhf-raspberry-pi 通过简单的方法可烧录TF卡，系统自动安装配置 终端输入sudo raspi-config启动树莓派配置工具，启动ssh服务并开机启动 操作步骤参考：玩转树莓派之Ubuntu MATE for the Raspberry 系统安装\n安装宝塔 官方安装要求 安装要求： 内存：128M以上，推荐512M以上（纯面板约占系统10M内存） 硬盘：100M以上可用硬盘空间（纯面板约占20M磁盘空间） 系统：暂时只对 CentOS 6.x / 7.x /Ubuntu/Debian 确保是干净的操作系统，没有安装过其它环境带的Apache/Nginx/php/MySQL\n下面使用putty登录PI，安装宝塔\n自动运行如下\nroot@fjy-desktop:/# wget -O install.sh http://download.bt.cn/install/install-ubuntu.sh \u0026amp;\u0026amp; sudo bash install.sh \u0026ndash;2018-01-23 13:13:41\u0026ndash; http://download.bt.cn/install/install-ubuntu.sh Resolving download.bt.cn (download.bt.cn)\u0026hellip; 103.224.251.67 Connecting to download.bt.cn (download.bt.cn)|103.224.251.67|:80\u0026hellip; connected. HTTP request sent, awaiting response\u0026hellip; 200 OK Length: 13572 (13K) [application/octet-stream] Saving to: ‘install.sh’\ninstall.sh 100%[===================\u0026gt;] 13.25K \u0026ndash;.-KB/s in 0s\n2018-01-23 13:13:41 (52.5 MB/s) - ‘install.sh’ saved [13572/13572]\n+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;- | Bt-WebPanel 5.x FOR Ubuntu/Debian +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;- | Copyright © 2015-2017 BT-SOFT(http://www.bt.cn) All rights reserved. +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;- | The WebPanel URL will be http://SERVER_IP:8888 when installed. +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;- 下面选择确定，输入小写y Do you want to install Bt-Panel to the /www directory now?(y/n): y 然后等待系统自动安装依赖\n安装完成后会显示如下信息，提供了出事登录用户名和IP地址 ================================================================== Congratulations! Install succeeded! ================================================================== Bt-Panel: http://113.83.XX.XXX:8888 username: admin password: 0df18173 Warning: If you cannot access the panel, release the following port (8888|888|80|443|20|21) in the security group ================================================================== Time consumed: 12 Minute!\n但ip地址一般都是不对的，请自行输入ip addr查询主机的ip地址\nroot@fjy-desktop:/# ip addr 1: lo: \u0026lt;LOOPBACK,UP,LOWER_UP\u0026gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enxb827eb9e1f8b: \u0026lt;NO-CARRIER,BROADCAST,MULTICAST,UP\u0026gt; mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000 link/ether b8:27:eb:9e:1f:8b brd ff:ff:ff:ff:ff:ff 3: wlan0: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether b8:27:eb:cb:4a:de brd ff:ff:ff:ff:ff:ff inet 192.168.31.9/24 brd 192.168.31.255 scope global dynamic wlan0 valid_lft 30658sec preferred_lft 30658sec inet6 fe80::ab44:81:1f49:ce98/64 scope link valid_lft forever preferred_lft forever 由上可知192.168.31.9就是我树莓派的局域网地址，因此在浏览器登录的地址为http://192.168.31.9:8888 输入默认账号密码即可登录，登录后进入如下页面，提示安装全套web服务组件\n这里选择安装LAMP组件套装，因为后面打算安装私有云服务（nextcloud），nextcloud对nginx支持并不友好，需要负载均衡的可以安装LNMP 点击一键安装即可 安装状态可以点击左上角的标号查看 点击监软件管理，可以安装其他软件，包括Tomcat、gitlab等，安装Tomcat还会自动配置java和javac环境，可以说是非常的方便 以后要搭建java web服务，因此这里选择安装Tomcat，版本选择8\n最后点击监控面板，打开服务器状态监控 一段时间后可查看服务器状态曲线图 有时候安装完成后发现MySQL服务无法被宝塔安装，那么可以到putty中使用命令行自行安装，输入sudo apt-get install mysql-server\nroot@fjy-desktop:/home/fjy# apt-get install mysql-server 正在读取软件包列表\u0026hellip; 完成 正在分析软件包的依赖关系树 正在读取状态信息\u0026hellip; 完成 将会同时安装下列软件： libaio1 libcgi-fast-perl libcgi-pm-perl libevent-core-2.0-5 libfcgi-perl libhtml-template-perl mysql-client-5.7 mysql-client-core-5.7 mysql-common mysql-server-5.7 mysql-server-core-5.7 建议安装： libipc-sharedcache-perl mailx tinyca 下列【新】软件包将被安装： libaio1 libcgi-fast-perl libcgi-pm-perl libevent-core-2.0-5 libfcgi-perl libhtml-template-perl mysql-client-5.7 mysql-client-core-5.7 mysql-common mysql-server mysql-server-5.7 mysql-server-core-5.7 升级了 0 个软件包，新安装了 12 个软件包，要卸载 0 个软件包，有 249 个软件包未被升级。 需要下载 17.7 MB 的归档。 解压缩后会消耗 140 MB 的额外空间。 您希望继续执行吗？ [Y/n] Y 之后会弹出如下对话框，输入默认数据库管理员密码即可 有时候安装后会发现Tomcat无法正常启动，进入putty用命令javac，发现未安装jdk\nroot@fjy-desktop:/var/lib/mysql# javac The program \u0026lsquo;javac\u0026rsquo; can be found in the following packages:\ndefault-jdk ecj gcj-5-jdk openjdk-8-jdk-headless gcj-4.8-jdk gcj-4.9-jdk Try: apt install root@fjy-desktop:/var/lib/mysql# java -version openjdk version \u0026ldquo;1.8.0_151\u0026rdquo; OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12) OpenJDK Client VM (build 25.151-b12, mixed mode) 则输入sudo apt-get install openjdk-8-jdk安装jdk环境\nroot@fjy-desktop:/var/lib/mysql# apt-get install openjdk-8-jdk 正在读取软件包列表\u0026hellip; 完成 正在分析软件包的依赖关系树 正在读取状态信息\u0026hellip; 完成 将会同时安装下列软件： fonts-dejavu-extra libgif7 libice-dev libpthread-stubs0-dev libsm-dev libx11-dev libx11-doc libxau-dev libxcb1-dev libxdmcp-dev libxt-dev openjdk-8-jdk-headless openjdk-8-jre x11proto-core-dev x11proto-input-dev x11proto-kb-dev xorg-sgml-doctools xtrans-dev 建议安装： libice-doc libsm-doc libxcb-doc libxt-doc openjdk-8-demo openjdk-8-source visualvm icedtea-8-plugin 下列【新】软件包将被安装： fonts-dejavu-extra libgif7 libice-dev libpthread-stubs0-dev libsm-dev libx11-dev libx11-doc libxau-dev libxcb1-dev libxdmcp-dev libxt-dev openjdk-8-jdk openjdk-8-jdk-headless openjdk-8-jre x11proto-core-dev x11proto-input-dev x11proto-kb-dev xorg-sgml-doctools xtrans-dev 升级了 0 个软件包，新安装了 19 个软件包，要卸载 0 个软件包，有 249 个软件包未被升级。 需要下载 11.8 MB 的归档。 解压缩后会消耗 61.9 MB 的额外空间。 您希望继续执行吗？ [Y/n] Y\n手动安装JDK后应去Tomcat配置文件中更改JAVA_HOME的路径以确保Tomcat正常运行 查看当前系统jre和jdk路径，输入update-alternatives --config java和update-alternatives --config javac\nroot@fjy-desktop:/etc# update-alternatives \u0026ndash;config java 链接组 java (提供 /usr/bin/java)中只有一个候选项：/usr/lib/jvm/java-8-openjdk-ar mhf/jre/bin/java 无需配置。 root@fjy-desktop:/etc# update-alternatives \u0026ndash;config javac 链接组 javac (提供 /usr/bin/javac)中只有一个候选项：/usr/lib/jvm/java-8-openjdk- armhf/bin/javac 无需配置。 输入root@fjy-desktop:/etc# vi /etc/init.d/tomcat进入Tomcat配置文件 找到JAVA_HOME，修改为系统配置的jdk路径 默认为 JAVA_HOME=/usr/java/jdk CATALINA_HOME=/www/server/tomcat 改成 JAVA_HOME=JAVA_HOME=/usr/lib/jvm/java-8-openjdk-armhf/jre CATALINA_HOME=/www/server/tomcat 保存退出即可\n","date":"2018-01-23T13:11:00+08:00","permalink":"/archives/19/","title":"使用宝塔运维Ubuntu 16.04"},{"content":"介绍 虎课网第一期原创合成作品，同时是原定2016年参加深蓝电脑协会的参赛作品，最终因记错时间没有提交参赛 教程链接（免费）： https://huke88.com/course/5642.html\n其他细节 界面截图 ","date":"2018-01-22T15:49:00+08:00","permalink":"/archives/16/","title":"合成案例教学作品1——恐龙入侵"},{"content":"1.域名 博客域名为阿里万网域名，二级域名为fjy8018.top，一次性买了5年 博客使用的是三级域名：web.fjy8018.top\n2.公网IP 博客ip位于美国洛杉矶，为购买vps主机赠送，试用期一年，因为在国外，所以就免备案~ 3.VPS主机配置 CPU 至强E5 单核 内存 ECC 512M 硬盘 SSD 10G 带宽 1GMbps 流量 1T/月 4.服务主机 服务主机位于中国广东电信内网，通过frp映射到美国vps 5.主机配置 CPU I5-2500s 内存 DDR3-1600 4G 硬盘 2.5inch 1T 5400PRM 其他 NEC二手机箱，主板，电源，风扇 网络 千兆网卡 操作系统 CentOS 7 x64\n至于如何运维CentOS ，Putty大法好！ ","date":"2018-01-22T14:33:00+08:00","permalink":"/archives/4/","title":"第一篇就用来介绍服务器吧"},{"content":"如果您看到这篇文章,表示您的 blog 已经安装成功.\n","date":"2018-01-22T11:06:00+08:00","permalink":"/archives/1/","title":"欢迎使用 Typecho"}]