Docker on F嘉阳

Kubesphere落地DevOps

Sat, 19 Jun 2021 22:16:00 +0800

创建企业空间

创建一个企业空间和一个帐户 (project-admin)，必须邀请该帐户至该企业空间并赋予 workspace-self-provisioner 角色。

进入企业空间，切换到DevOps工程，点击创建

创建工程

安装helm

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

找不到环境变量报错

安装helm遇到报错

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
 % Total % Received % Xferd Average Speed Time Time Time Current
unset -f pathmunge
 Dload Upload Total Spent Left Speed
100 11248 100 11248 0 0 61543 0 --:--:-- --:--:-- --:--:-- 61802
Helm v3.6.0 is available. Changing from version v3.2.1.
Downloading https://get.helm.sh/helm-v3.6.0-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
helm not found. Is /usr/local/bin on your $PATH?
Failed to install helm
 For support, go to https://github.com/helm/helm.

则需要配置环境变量

$ vi /etc/profile
# 添加配置
export PATH=/usr/local/bin:$PATH

$ source /etc/profile

验证

$ helm version
version.BuildInfo{Version:"v3.6.0", GitCommit:"7f2df6467771a75f5646b7f12afb408590ed1755", GitTreeState:"clean", GoVersion:"go1.16.3"}

安装SonarQube

执行以下命令安装 SonarQube 服务器

$ helm upgrade --install sonarqube sonarqube --repo https://charts.kubesphere.io/main -n kubesphere-devops-system --create-namespace --set service.type=NodePort

输出

执行以下命令以获取 SonarQube NodePort

$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
$ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo http://$NODE_IP:$NODE_PORT

输出

此时可在kubesphere控制台中看到sonar服务

配置 SonarQube 服务器

访问地址，点击右上角的 Log in，然后使用默认帐户 admin/admin 登录。

创建 SonarQube 管理员令牌 (Token)

点击右上角字母 A，然后从菜单中选择 My Account 以转到 Profile 页面。

生成token

创建 Webhook 服务器

执行以下命令获取 SonarQube Webhook 的地址。

$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services ks-jenkins)
$ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo http://$NODE_IP:$NODE_PORT/sonarqube-webhook/

输出

依次点击 Administration、Configuration 和 Webhooks 创建一个 Webhook。

点击 Create

由于webhook都通过内部网络调用，故可以使用内部网络地址

将 SonarQube 配置添加到 ks-installer

执行以下命令编辑 ks-installer

$ kubectl edit cc -n kubesphere-system ks-installer

搜寻至 devops。添加字段 sonarqube 并在其下方指定 externalSonarUrl 和 externalSonarToken。

 devops:
 enabled: true
 jenkinsJavaOpts_MaxRAM: 2g
 jenkinsJavaOpts_Xms: 512m
 jenkinsJavaOpts_Xmx: 512m
 jenkinsMemoryLim: 2Gi
 jenkinsMemoryReq: 1500Mi
 jenkinsVolumeSize: 8Gi
 sonarqube:
 externalSonarUrl: http://10.170.0.5:32108
 externalSonarToken: c6a6e2e15a9d52e30b744df0eef4c639bbea88a1

完成操作后保存此文件。

将 SonarQube 服务器添加至 Jenkins

执行以下命令获取 Jenkins 的地址。

$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services ks-jenkins)
$ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo http://$NODE_IP:$NODE_PORT

输出

使用地址 http://{$Public IP}:30180 访问 Jenkins。安装 KubeSphere 时，默认情况下也会安装 Jenkins 仪表板。此外，Jenkins 还配置有 KubeSphere LDAP，这意味着您可以直接使用 KubeSphere 帐户（例如 admin/P@88w0rd）登录 Jenkins。

使用kubesphere admin账户登录

前往系统管理下的 Manage Credentials 并点击 Stores scoped to Jenkins 下的 Jenkins，再点击全局凭据 (unrestricted)

点击左侧导航栏的添加凭据，用 SonarQube 管理员令牌添加凭证

点击左侧的系统管理，向下翻页找到并点击系统配置。

搜寻到 SonarQube servers，然后点击 Add SonarQube。

添加sonar服务器地址和名称，点击应用完成操作。

将 sonarqubeURL 添加到 KubeSphere 控制台

指定 sonarqubeURL，以便可以直接从 KubeSphere 控制台访问 SonarQube。

执行以下命令：

$ kubectl edit cm -n kubesphere-system ks-console-config

搜寻到 client，添加 devops 字段并指定 sonarqubeURL。

 client:
 version:
 kubesphere: v3.1.0
 kubernetes: v1.20.4
 openpitrix: v0.3.5
 enableKubeConfig: true
 devops:
 sonarqubeURL: http://34.92.90.181:32108

结果

重启服务

执行以下命令。

$ kubectl -n kubesphere-system rollout restart deploy ks-apiserver
$ kubectl -n kubesphere-system rollout restart deploy ks-console

为新工程创建 SonarQube Token

创建一个 SonarQube 令牌，以便您的流水线可以在运行时与 SonarQube 通信。

在 SonarQube 控制台上，点击 Create new project。

设置工程名称

生成token

创建令牌后，点击 Continue。

分别选择 Java 和 Maven。复制下图所示绿色框中的序列号，如果要在流水线中使用，则需要在凭证中添加此序列号。

创建流水线工程

创建凭证

凭证 ID	类型	用途
dockerhub-id	帐户凭证	Docker Hub
demo-kubeconfig	kubeconfig	Kubernetes

demo-kubeconfig

dockerhub-id

创建一个凭证 ID (sonar-token)

列表中看到已创建的三个凭证。

创建流水线

在流水线页面点击创建，为了使用图形化流水线搭建，不在此处引用代码仓库，直接点击下一步

保持默认配置，点击创建

创建成功

进入流水线，点击编辑

此时选择CICD模板

可视化编辑流水线

编辑后运行

流水线支持交互式操作

在 KubeSphere 控制台查看结果

流水线执行成功后，即可查看到代码质量标签

sonar控制台也可以查看对应结果

jenkins视图

blue ocean

Kubesphere运维Kubernetes集群

Sun, 23 May 2021 22:58:00 +0800

背景

使用原生Kubernetes-dashboard和Grafana+Prometheus管理较大规模集群比较困难，有如下缺陷

每次执行命令都要登陆服务器
容器日志采集方案、监控方案搭建比较繁琐
添加节点不方便
管理存储插件、秘钥、配置不方便

解决方案

目前国内流行的k8s管理平台主要有rancher、kubesphere、kubeboard

rancher主打原生集群管理，侧重运维人员
kubesphere主打开发运维工作流，侧重研发人员
kubeboard与kubesphere类似，侧重微服务管理，但功能上较少，对多节点管理需要收费

经过评估，最终决定采用kubesphere进行集群管理

Rancher — 多云混合云多集群Kubernetes管理平台

[post url=“https://kubesphere.io/zh/” title=“Kubesphere” intro=“面向云原生应用的容器混合云” cover=“https://kubesphere.io/images/logo.svg” /]

Kuboard — 快速在 Kubernetes 上落地微服务

安装步骤

目标是面对小规模集群管理，节点数最多为5个，故不考虑高可用集群配置，采用单主节点+4个worker节点配置

采用纯净CentOS 7 Linux系统，需要做以下前置准备

依赖包安装

安装官方要求的依赖包，由于计划安装NFS存储插件，还需要安装nfs工具包

$ yum install -y socat conntrack ebtables ipset nfs-utils

配置DNS解析

将各节点DNS配置为私有DNS服务器

$ tee /etc/resolv.conf <<-'EOF'
# Generated by NetworkManager
nameserver 192.168.1.13
EOF

私有DNS搭建可以参考文章

[[利用Docker自建Dns服务器]]

按照官方教程安装多节点kubernetes

强烈建议先用KubeKey安装纯净k8s集群+控制面板，关闭所有插件。等安装完毕再一个个插件打开，防止因为内存不够而无法打开kubesphere

安装步骤参考官方文档一键安装方式

多节点安装 — Kubesphere

启动成功

启动成功后按照官方指南打开插件

启用可插拔组件 — Kubesphere

所有插件打开后如下

使用

节点状态

存储卷管理

监控告警

日志采集

日志查询

操作审计

工具箱

企业空间权限控制

内置helm应用商店，支持上线审批，在线编辑等

支持关联外部仓库

应用流量治理

链路追踪

灰度发布

费用统计

更多的使用场景包括：DevOps、边缘节点、多集群管理还在探索中

利用Docker自建DNS服务器

Sun, 09 May 2021 18:14:12 +0800

背景

当Linux集群中机器不断增多，集群内通过配置/hosts实现域名内网访问。

目标

相同域名在集群之外通过公有DNS访问外网地址
相同域名在集群内通过DNS解析访问内网地址

步骤

搭建DNS服务器

DNS服务器可以通过jpillora/dnsmasq容器实现

jpillora/dnsmasq — DockerHub

新增配置文件 dnsmasq.conf

# 完整配置文件参考以下地址
# http://oss.segetech.com/intra/srv/dnsmasq.conf
# 记录所有DNS查询请求
log-queries
# 不使用本地配置的域名解析器
no-resolv
# 不使用本地hosts
no-hosts
# 配置上游DNS服务器地址，用于解析国内域名
server=114.114.114.114
# 配置国际DNS服务器地址
server=8.8.8.8
strict-order
# 配置内网解析
address=/gitlab.fjy8018.top/192.168.1.11

配置docker-compose文件，DNS服务器默认通过53 udp端口解析，不建议修改53端口的外部地址

version: "3.5"

services:

 dnsmasq:
 image: jpillora/dnsmasq:1.1.0
 networks:
 - dns-net
 ports:
 - 53:53/udp
 - 5380:8080
 volumes:
 - ./dnsmasq.conf:/etc/dnsmasq.conf
 environment:
 - "HTTP_USER=admin"
 - "HTTP_PASS=admin"
 restart: always

networks:
 dns-net:

启动服务即可

后续需要修改DNS解析只需要访问5380的web UI界面即可动态调整

服务器指向新DNS服务器

将所有服务器/etc/resolv.conf文件指向新DNS服务器地址

$ tee /etc/resolv.conf <<-'EOF'
# Generated by NetworkManager
nameserver 192.168.1.13
EOF

补充

对于kubernetes集群，由于集群内使用了coredns和nodelocaldns缓存服务，还需要重启两个dns服务组件

$ kubectl -n kube-system rollout restart daemonset nodelocaldns
$ kubectl -n kube-system rollout restart deploy coredns

Sentinel持久化存储和镜像构建

Sun, 07 Feb 2021 12:06:58 +0800

背景

Sentinel默认只能对限流规则存储到内存中

在生产环境中使用 Sentinel — Sentinel中文文档

官方文档对于数据持久化描述如下

我们提供了 ZooKeeper, Apollo, Nacos 等的动态数据源实现。以 ZooKeeper 为例子，如果要使用第三方的配置中心作为配置管理，您需要做下面的几件事情:

实现一个公共的 ZooKeeper 客户端用于推送规则，在 Sentinel 控制台配置项中需要指定 ZooKeeper 的地址，启动时即创建 ZooKeeper Client。

我们需要针对每个应用（appName），每种规则设置不同的 path（可随时修改）；或者约定大于配置（如 path 的模式统一为 /sentinel_rules/{appName}/{ruleType}，e.g. sentinel_rules/appA/flowRule）。

规则配置页需要进行相应的改造，直接针对应用维度进行规则配置；修改同个应用多个资源的规则时可以批量进行推送，也可以分别推送。Sentinel 控制台将规则缓存在内存中（如 InMemFlowRuleStore），可以对其进行改造使其支持应用维度的规则缓存（key 为 appName），每次添加/修改/删除规则都先更新内存中的规则缓存，然后需要推送的时候从规则缓存中获取全量规则，然后通过上面实现的 Client 将规则推送到 ZooKeeper 即可。

应用客户端需要注册对应的读数据源以监听变更，可以参考相关文档。

由于alibaba/Sentinel官方并没有实现对具体数据源的写入支持，只提供的实现的接口，故无法做到开箱即用

解决方法

已经有贡献者提供了对三种动态数据源的实现PR，但还没合并进主版本中

添加持久化支持，可选持久化选项 — Fixes Issues #1696 添加持久化支持，可选持久化选项：nacos、apollo、zookeeper ; 修改 README.md 说明文档

进一步完善

本次基于jiajiangnan/Sentinelfork出自己的工程FJiayang/Sentinel，实现docker镜像打包，并基于GitHub Actions完成自动化构建和上传制品。

jiajiangnan/Sentinel — jiajiangnan的Sentinel仓库源码

FJiayang/Sentinel — FJiayang的Sentinel仓库源码

添加Dockerfile

基于最小jdk镜像制作

FROM openjdk:8-jdk-alpine3.7

MAINTAINER fjy8018@gmail.com

ENV VERSION 1.8.0

RUN mkdir -p /sentinel

WORKDIR /sentinel

ADD ./sentinel-dashboard/target/sentinel-dashboard.jar ./app-${VERSION}.jar

# 设置时区，默认为UTC
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN echo 'Asia/Shanghai' > /etc/timezone

CMD java ${PARAM} -jar app-${VERSION}.jar

添加GitHub流水线配置文件

name: Publish Docker image

on: [push]

jobs:
 build:

 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v2
 - name: Set up Maven Central Repository
 uses: actions/setup-java@v1
 with:
 java-version: 1.8

 - name: Publish package
 run: mvn --batch-mode clean package -Dmaven.test.skip=true

 - name: Push to Docker Hub
 uses: docker/build-push-action@v1
 with:
 username: ${{ secrets.DOCKER_USERNAME }}
 password: ${{ secrets.DOCKER_PASSWORD }}
 repository: fjy8018/sentinel
 tag_with_ref: true

具体的流水线编写可参考文章

使用方法

具体参数文档

项	类型	默认值	最小值	描述
sentinel.dashboard.auth.username	String	sentinel	无	登录控制台的用户名，默认为 `sentinel`
sentinel.dashboard.auth.password	String	sentinel	无	登录控制台的密码，默认为 `sentinel`
sentinel.dashboard.app.hideAppNoMachineMillis	Integer	0	60000	是否隐藏无健康节点的应用，距离最近一次主机心跳时间的毫秒数，默认关闭
sentinel.dashboard.removeAppNoMachineMillis	Integer	0	120000	是否自动删除无健康节点的应用，距离最近一次其下节点的心跳时间毫秒数，默认关闭
sentinel.dashboard.unhealthyMachineMillis	Integer	60000	30000	主机失联判定，不可关闭
sentinel.dashboard.autoRemoveMachineMillis	Integer	0	300000	距离最近心跳时间超过指定时间是否自动删除失联节点，默认关闭
datasource.provider	String	memory	无	默认为 `memory`, 可选持久化配置 `nacos`、`apollo`、`zookeeper`
datasource.provider.nacos.server-addr	String	localhost:8848	无	nacos 注册中心地址
datasource.provider.nacos.username	String		无	nacos 用户名，默认为空
datasource.provider.nacos.password	String		无	nacos 密码，默认为空
datasource.provider.nacos.namespace	String		无	nacos 名臣空间，默认为空
datasource.provider.nacos.group-id	String	SENTINEL_GROUP	无	nacos 分组，默认为 `SENTINEL_GROUP`
datasource.provider.apollo.server-addr	String	http://localhost:10034	无	apollo 注册中心地址，必须有前缀 `http://` 或 `https://`
datasource.provider.apollo.token	String	token	无	apollo 登录 token，默认为 `token`
datasource.provider.zookeeper.server-addr	String	localhost:2181	无	zookeeper 注册中心地址
datasource.provider.zookeeper.session-timeout	Integer	60000	0	zookeeper session超时时间，默认 `60000`
datasource.provider.zookeeper.connection-timeout	Integer	15000	0	zookeeper connection超时时间，默认 `15000`
datasource.provider.zookeeper.retry.max-retries	Integer	3	0	zookeeper 最大重试次数，默认 `3`
datasource.provider.zookeeper.retry.base-sleep-time	Integer	1000	1000	zookeeper 重试间隔最小时长，默认 `1000`
datasource.provider.zookeeper.retry.max-sleep-time	Integer	2147483647	0	zookeeper 重试间隔最大时长，默认 `2147483647`

对于单条Docker命令启动

$ docker run --rm -e PARAM="-Dserver.port=8858 -Dcsp.sentinel.dashboard.server=localhost:8858 -Dproject.name=sentinel-dashboard -Ddatasource.provider=nacos -Ddatasource.provider.nacos.server-addr=192.168.133.128:8848 -Ddatasource.provider.nacos.username=nacos -Ddatasource.provider.nacos.password=nacos -Ddatasource.provider.nacos.group-id=SENTINEL_GROUP" fjy8018/sentinel:1.8.0

对于docker-compose启动，此处结合nacos与mysql基础设施给出示例，mysql需要提前写入nacos相关初始化表结构

version: '3.7'
services:

 mysql57:
 image: mysql:5.7.29
 restart: always
 command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
 ports:
 - 3306:3306
 networks:
 - dev
 volumes:
 - ./mysql57/data:/var/lib/mysql
 - ./mysql57/custom-master.cnf:/etc/mysql/conf.d/custom.cnf
 - ./mysql57/log:/var/log/mysql
 environment:
 MYSQL_ROOT_PASSWORD: root

 nacos-standalone:
 image: nacos/nacos-server:1.1.4
 networks:
 - dev
 environment:
 - PREFER_HOST_MODE=hostname
 - MODE=standalone
 - SPRING_DATASOURCE_PLATFORM=mysql
 - MYSQL_MASTER_SERVICE_HOST=mysql57
 - MYSQL_MASTER_SERVICE_DB_NAME=nacos
 - MYSQL_MASTER_SERVICE_PORT=3306
 - MYSQL_SLAVE_SERVICE_HOST=mysql57
 - MYSQL_SLAVE_SERVICE_PORT=3306
 - MYSQL_MASTER_SERVICE_USER=root
 - MYSQL_MASTER_SERVICE_PASSWORD=root
 ports:
 - 8848:8848
 depends_on:
 - mysql57
 volumes:
 - ./standalone-logs/:/home/nacos/logs

 sentinel:
 image: fjy8018/sentinel:1.8.0
 environment:
 - PARAM=-Dserver.port=8858 -Dcsp.sentinel.dashboard.server=localhost:8858 -Dproject.name=sentinel-dashboard -Ddatasource.provider=nacos -Ddatasource.provider.nacos.server-addr=nacos-standalone:8848 -Ddatasource.provider.nacos.username=nacos -Ddatasource.provider.nacos.password=nacos -Ddatasource.provider.nacos.group-id=SENTINEL_GROUP
 networks:
 - dev
 ports:
 - 8858:8858
 restart: on-failure

networks:
 dev:

启动成功后可以看到nacos上成功注册了sentinel-dashboard

GitHub DevOps工作流构建Java项目并上传到DockerHub

Sun, 07 Feb 2021 11:58:43 +0800

工程准备

此处使用fork自alibaba/Sentinel的FJiayang/Sentinel项目实践

工程地址

Sentinel — Sentinel仓库源码

准备Dockerfile

该工程为标准Java项目，故准备好的Dockerfile文件如下

FROM anapsix/alpine-java:8_server-jre_unlimited

MAINTAINER fjy8018@gmail.com

ENV PARAM ""
ENV VERSION 1.8.0

RUN mkdir -p /sentinel

WORKDIR /sentinel

ADD ./sentinel-dashboard/target/sentinel-dashboard.jar ./app-${VERSION}.jar

# 设置时区，默认为UTC
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN echo 'Asia/Shanghai' > /etc/timezone

ENTRYPOINT ["java", "${PARAM}", "-jar", "app-${VERSION}.jar"]

编写GitHub工作流

GitHub DevOps工作流又叫GitHub Actions 是2019年末推出的自动化CICD持续集成工作流，有详细的样例文档

此处参考以下文档编写

使用 Maven 构建和测试 Java — GitHub Actions 指南

安全配置

由于需要建构建后的制品推送到DockerHub中，但又不能在工作流中直接使用用户名和密码登陆，故还需要配置环境变量

在DockerHub中新增秘钥

在构建工程的设置中新增用户名和密码变量

编写后的工作流如下

name: Publish Docker image

on: [push]

jobs:
 build:

 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v2
 - name: Set up Maven Central Repository
 uses: actions/setup-java@v1
 with:
 java-version: 1.8

 - name: Publish package
 run: mvn --batch-mode clean package -Dmaven.test.skip=true

 - name: Push to Docker Hub
 uses: docker/build-push-action@v1
 with:
 username: ${{ secrets.DOCKER_USERNAME }}
 password: ${{ secrets.DOCKER_PASSWORD }}
 repository: fjy8018/sentinel
 tag_with_ref: true

触发构建

当前的触发方式配置为PUSH，修改好文件后会自动触发流水线工作

Maven打包，速度很快

打包完成

检查结果

登录到DockerHub中检查推送结果，可见成功推送

总结

整体来看，GitHub工作流效率高，文档齐全，编码上手速度快，相比GitLab CICD和DockerHub CICD来说，GitHub CICD配置更加简单、友好和高效。

Docker MySQL主从复制方案

Fri, 17 Apr 2020 15:37:02 +0800

背景

现有网上很多教程都是修改了源MySQL镜像，因此会存在安全隐患，也不容易提升版本。此处给出不修改源镜像的主从复制配置方案。

现状

现有一台生产上运行的MySQL5.7服务器，由docker镜像提供支持

原docker-compose文件

version: '3.7'
services:

 mysql57:
 image: mysql:5.7
 restart: always
 command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
 ports:
 - 3306:3306
 networks:
 - prod
 volumes:
 - ./mysql57/data:/var/lib/mysql
 environment:
 MYSQL_ROOT_PASSWORD: root
networks:
 prod:

改造过程

配置主服务器

新增同步用户并授权

##创建slave1用户
CREATE USER 'slave1'@'%' IDENTIFIED BY 'slavepass';

##为slave1赋予REPLICATION SLAVE权限。
GRANT REPLICATION SLAVE ON *.* TO 'slave1'@'%';

开启binlog日志

由于原主服务器没有开启binlog日志，故需要挂载配置文件并开启binlog日志

新增配置文件custom-master.cnf

[mysqld]
log-bin = mysql-bin
log-bin-index = mysql-bin.index
binlog_format = mixed
server-id = 1
sync-binlog = 1
character-set-server = utf8

修改docker-compose文件为

version: '3.7'
services:

 mysql57:
 image: mysql:5.7.29
 restart: always
 command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
 ports:
 - 3306:3306
 networks:
 - prod
 volumes:
 - ./mysql57/data:/var/lib/mysql
 - ./mysql57/custom-master.cnf:/etc/mysql/conf.d/custom.cnf
 - ./mysql57/log:/var/log/mysql
 environment:
 MYSQL_ROOT_PASSWORD: root
networks:
 prod:

更新服务（会短暂停机）

$ docker-compose up -d

配置内容解释

log-bin：设置二进制日志文件的基本名；
log-bin-index：设置二进制日志索引文件名；
binlog_format：控制二进制日志格式，进而控制了复制类型，三个可选值
 -STATEMENT：语句复制
 -ROW：行复制
 -MIXED：混和复制，默认选项
server-id：服务器设置唯一ID，默认为1，推荐取IP最后部分；
sync-binlog：默认为0，为保证不会丢失数据，需设置为1，用于强制每次提交事务时，同步二进制日志到磁盘上。

备份主数据库数据

该步骤可以等到两台从服务器的docker环境都配置完毕后操作，以减少停机时间

获取一个一致性的快照，需对所有表设置读锁：

flush tables with read lock;

获取二进制日志的坐标：

show master status;

File	Position	Binlog_Do_DB	Binlog_Ignore_DB	Executed_Gtid_Set
mysql-bin.000003	6165

备份数据

# 针对事务性引擎
mysqldump -u root -p --all-databases -e --single-transaction --flush-logs --max_allowed_packet=1048576 --net_buffer_length=16384 > /var/log/mysql/all_db.sql
# 针对 MyISAM 引擎，或多引擎混合的数据库
mysqldump -uroot --all-databases -e -l --flush-logs --max_allowed_packet=1048576 --net_buffer_length=16384 > /var/log/mysql/all_db.sql

由于挂载了日志目录，故在容器外部可以看到该备份的数据文件

配置从服务器

新增两台CentOS 7服务器，并安装docker和docker-compose服务
由于主从复制要求所有机器版本号包括小版本号都必须一致，故此处拉取mysql:5.7.29镜像
从主库拷贝数据文件all_db.sql到两台从库上

$ scp /root/mysql/log/all_db.sql root@192.168.1.33:/root/mysql/log/

新增配置文件custom-slave1.cnf

[mysqld]
log-bin = mysql-bin
binlog_format = mixed
log-slave-updates = 0
server-id = 2
relay-log = mysql-relay-bin
relay-log-index = mysql-relay-bin.index
read-only = 1
slave_net_timeout = 10

新增docker-compose文件

version: '3'
services:

 mysql57-master:
 image: mysql:5.7.29
 restart: always
 command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
 ports:
 - 3306:3306
 networks:
 - mysql-net
 volumes:
 - ./data:/var/lib/mysql
 - ./custom-slave1.cnf:/etc/mysql/conf.d/custom.cnf
 - ./log:/var/log/mysql
 environment:
 MYSQL_ROOT_PASSWORD: root

networks:
 mysql-net:

由于也挂载了log目录，启动后容器内就已经有从主库同步过来的数据文件
数据恢复

mysql -u root -p < /var/log/mysql/all_db.sql

使用工具连接数据库，配置主从连接

CHANGE MASTER TO
MASTER_HOST='10.168.1.30',
MASTER_USER='slave1',
MASTER_PASSWORD='slavepass',
MASTER_LOG_FILE='mysql-bin.000003',
MASTER_LOG_POS=6165;

启动slave同步进程

start slave;

关闭主服务器全局读锁

UNLOCK TABLES;

查看从服务器同步状态，主要是Slave_IO_Running， Slave_SQL_Running状态必须为Yes

Slave_IO_State	Master_Host	Master_User	Master_Port	Connect_Retry	Master_Log_File	Read_Master_Log_Pos	Relay_Log_File	Relay_Log_Pos	Relay_Master_Log_File	Slave_IO_Running	Slave_SQL_Running	Replicate_Do_DB	Replicate_Ignore_DB	Replicate_Do_Table	Replicate_Ignore_Table	Replicate_Wild_Do_Table	Replicate_Wild_Ignore_Table	Last_Errno	Last_Error	Skip_Counter	Exec_Master_Log_Pos	Relay_Log_Space	Until_Condition	Until_Log_File	Until_Log_Pos	Master_SSL_Allowed	Master_SSL_CA_File	Master_SSL_CA_Path	Master_SSL_Cert	Master_SSL_Cipher	Master_SSL_Key	Seconds_Behind_Master	Master_SSL_Verify_Server_Cert	Last_IO_Errno	Last_IO_Error	Last_SQL_Errno	Last_SQL_Error	Replicate_Ignore_Server_Ids	Master_Server_Id	Master_UUID	Master_Info_File	SQL_Delay	SQL_Remaining_Delay	Slave_SQL_Running_State	Master_Retry_Count	Master_Bind	Last_IO_Error_Timestamp	Last_SQL_Error_Timestamp	Master_SSL_Crl	Master_SSL_Crlpath	Retrieved_Gtid_Set	Executed_Gtid_Set	Auto_Position	Replicate_Rewrite_DB	Channel_Name	Master_TLS_Version
Waiting for master to send event	192.168.1.30	slave1	3306	60	mysql-bin.000003	3722724	mysql-relay-bin.000002	3717604	mysql-bin.000003	Yes	Yes							0		0	3722724	3717811	None		0	No						0	No	0		0			1	4c99130f-e664-11e9-987b-0242ac120003	/var/lib/mysql/master.info	0		Slave has read all relay log; waiting for more updates	86400								0

可能出现的问题

死锁——waiting for global read lock

原因在于flush tables with read lock语句和事务执行的过程都涉及到连个锁，而且不是原子的，再加上并行复制以及设置了从库binlog的顺序提交，最终导致多个线程形成死锁

解决方法：

重启主数据库
查看所有连接线程，分析死锁的原因，杀掉执行flush tables with read lock的线程

参考资料：

k8s环境下GitLab+Helm+GitLab Runner Java项目CICD落地实践

Wed, 27 Nov 2019 10:24:00 +0800

背景

目前使用5台服务器搭建了Kubernetes集群环境，监控、日志采集均已落地，业务也手工迁移到集群中顺利运行，故需要将原本基于原生docker环境的CICD流程迁移到Kubernetes集群中

优势

Kubernetes集群实现CICD有几个显著优势

Deployment 天然支持滚动部署、结合其他Kubernetes特性还能实现蓝绿部署、金丝雀部署等
新版本的GitLab与GitLab Runner天然支持Kubernetes集群，支持runner自动伸缩，减小资源占用

环境

Kubernetes版本：1.14

GitLab版本：12.2.5

GitLab-Runner版本：12.1.0

Docker环境版本：17.03.1

GitLab-Runner部署

配置介绍

原始环境的gitlab runner通过手动执行官网提供的注册命令和启动命令，分成两部部署，需要较多的手工操作，而在Kubernetes中，其支持使用Helm一键部署，官方文档如下

GitLab Runner Helm Chart

其实官方文档的指引并不清晰，许多配置在文档中没有介绍用法，推荐去其源码仓库查看详细的参数使用文档

The Kubernetes executor

其中介绍了几个关键配置，在后面修改工程的ci配置文件时会用到

使用DinD方式构建已经不再推荐

官方文档介绍

Use docker-in-docker workflow with Docker executor
The second approach is to use the special docker-in-docker (dind) Docker image with all tools installed (docker) and run the job script in context of that image in privileged mode.

Note: docker-compose is not part of docker-in-docker (dind). To use docker-compose in your CI builds, follow the docker-compose installation instructions.

Danger: By enabling --docker-privileged, you are effectively disabling all of the security mechanisms of containers and exposing your host to privilege escalation which can lead to container breakout. For more information, check out the official Docker documentation on Runtime privilege and Linux capabilities.

Docker-in-Docker works well, and is the recommended configuration, but it is not without its own challenges:

When using docker-in-docker, each job is in a clean environment without the past history. Concurrent jobs work fine because every build gets it’s own instance of Docker engine so they won’t conflict with each other. But this also means jobs can be slower because there’s no caching of layers.

By default, Docker 17.09 and higher uses --storage-driver overlay2 which is the recommended storage driver. See Using the overlayfs driver for details.

Since the docker:19.03.1-dind container and the Runner container don’t share their root filesystem, the job’s working directory can be used as a mount point for child containers. For example, if you have files you want to share with a child container, you may create a subdirectory under /builds/$CI_PROJECT_PATH and use it as your mount point (for a more thorough explanation, check issue #41227):

总之使用DinD进行容器构建并非不可行，但面临许多问题，例如使用overlay2网络需要Docker版本高于 17.09

Using docker:dind
Running the docker:dind also known as the docker-in-docker image is also possible but sadly needs the containers to be run in privileged mode. If you’re willing to take that risk other problems will arise that might not seem as straight forward at first glance. Because the docker daemon is started as a service usually in your .gitlab-ci.yaml it will be run as a separate container in your Pod. Basically containers in Pods only share volumes assigned to them and an IP address by which they can reach each other using localhost. /var/run/docker.sock is not shared by the docker:dind container and the docker binary tries to use it by default.

To overwrite this and make the client use TCP to contact the Docker daemon, in the other container, be sure to include the environment variables of the build container:

DOCKER_HOST=tcp://localhost:2375 for no TLS connection.

DOCKER_HOST=tcp://localhost:2376 for TLS connection.

Make sure to configure those properly. As of Docker 19.03, TLS is enabled by default but it requires mapping certificates to your client. You can enable non-TLS connection for DIND or mount certificates as described in Use Docker In Docker Workflow wiht Docker executor

在Docker 19.03.1版本之后默认开启了TLS配置，在构建的环境变量中需要声明，否则报连接不上docker的错误，并且使用DinD构建需要runner开启特权模式，以访问主机的资源，并且由于使用了特权模式，在Pod中对runner需要使用的资源限制将失效

使用Kaniko构建Docker镜像

目前官方提供另一种方式在docker容器中构建并推送镜像，实现更加优雅，可以实现无缝迁移，那就是kaniko

Building a Docker image with kaniko

其优势官网描述如下

在Kubernetes集群中构建Docker映像的另一种方法是使用kaniko。iko子

允许您构建没有特权访问权限的映像。

无需Docker守护程序即可工作。

在后面的实践中会使用两种方式构建Docker镜像，可根据实际情况选择

使用Helm部署

拉取Helm Gitlab-Runner仓库到本地，修改配置

GitLab Runner

将原有的gitlab-runner配置迁移到Helm中，迁移后如下

image: alpine-v12.1.0
imagePullPolicy: IfNotPresent
gitlabUrl: https://gitlab.fjy8018.top/
runnerRegistrationToken: "***"
unregisterRunners: true
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
rbac:
 create: true
 clusterWideAccess: false
metrics:
 enabled: true
 listenPort: 9090
runners:
 image: ubuntu:16.04
 imagePullSecrets:
 - name: registry-secret
 locked: false
 tags: "k8s"
 runUntagged: true
 privileged: true
 pollTimeout: 180
 outputLimit: 4096
 cache: {}
 builds: {}
 services: {}
 helpers: {}
resources:
 limits:
 memory: 2048Mi
 cpu: 1500m
 requests:
 memory: 128Mi
 cpu: 200m
affinity: {}
nodeSelector: {}
tolerations: []
hostAliases:
 - ip: "192.168.1.13"
 hostnames:
 - "gitlab.fjy8018.top"
 - ip: "192.168.1.30"
 hostnames:
 - "harbor.fjy8018.top"
podAnnotations: {}

其中配置了私钥、内网harbor地址、harbor拉取资源私钥，资源限制策略

GitLab-Runner选择可能导致的坑

选择runner镜像为alpine-v12.1.0，这一点单独说一下，目前最新的runner版本为12.5.0，但其有许多问题，alpine新版镜像在Kubernetes中间断发生无法解析DNS的问题，反映到GitLab-Runner中就是Could not resolve host和server misbehaving

查阅解决方法

通过查询发现，其官方仓库还有多个相关issue没有关闭

官方gitlab：Kubernetes runner: Could not resolve host

stackoverflow：Gitlab Runner is not able to resolve DNS of Gitlab Server

给出的解决方案无一例外都是降级到alpine-v12.1.0

We had same issue for couple of days. We tried change CoreDNS config, move runners to different k8s cluster and so on. Finally today i checked my personal runner and found that i’m using different version. Runners in cluster had gitlab/gitlab-runner:alpine-v12.3.0, when mine had gitlab/gitlab-runner:alpine-v12.0.1. We added line
image: gitlab/gitlab-runner:alpine-v12.1.0
in values.yaml and this solved problem for us

其问题的根源应该在于alpine基础镜像对Kubernetes 集群支持有问题，

ndots breaks DNS resolving #64924

docker-alpine仓库对应也有未关闭的issue，其中就提到了关于DNS解析超时和异常的问题

DNS Issue #255

安装

一行命令安装即可

$ helm install /root/gitlab-runner/ --name k8s-gitlab-runner --namespace gitlab-runner

输出如下

NAME: k8s-gitlab-runner
LAST DEPLOYED: Tue Nov 26 21:51:57 2019
NAMESPACE: gitlab-runner
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
k8s-gitlab-runner-gitlab-runner 5 0s

==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
k8s-gitlab-runner-gitlab-runner 0/1 1 0 0s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
k8s-gitlab-runner-gitlab-runner-744d598997-xwh92 0/1 Pending 0 0s

==> v1/Role
NAME AGE
k8s-gitlab-runner-gitlab-runner 0s

==> v1/RoleBinding
NAME AGE
k8s-gitlab-runner-gitlab-runner 0s

==> v1/Secret
NAME TYPE DATA AGE
k8s-gitlab-runner-gitlab-runner Opaque 2 0s

==> v1/ServiceAccount
NAME SECRETS AGE
k8s-gitlab-runner-gitlab-runner 1 0s


NOTES:

Your GitLab Runner should now be registered against the GitLab instance reachable at: "https://gitlab.fjy8018.top/"

查看gitlab admin页面，发现已经有一个runner成功注册

工程配置

DinD方式构建所需配置

如果原本的ci文件是基于19.03 DinD镜像构建的则需要加上TLS相关配置

image: docker:19.03

variables:
 DOCKER_DRIVER: overlay
 DOCKER_HOST: tcp://localhost:2375
 DOCKER_TLS_CERTDIR: ""
...

其余配置保持不变，使用DinD构建

Kubectl和Kubernetes权限配置

由于使用k8s集群，而通过集群部署需要使用kubectl客户端，故手动创建了一个kubectl docker镜像，使用gitlab触发dockerhub构建，构建内容公开透明，可放心使用，如有其它版本的构建需求也可提pull request，会在后面补充，目前用到的只有1.14.0

fjy8018/kubectl

有kubectl客户端，还需要配置连接TLS和连接账户

为了保障安全，新建一个专门访问该工程命名空间的ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
 name: hmdt-gitlab-ci
 namespace: hmdt

利用集群提供的RBAC机制，为该账户授予该命名空间的admin权限

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
 name: hmdt-gitlab-role
 namespace: hmdt
subjects:
 - kind: ServiceAccount
 name: hmdt-gitlab-ci
 namespace: hmdt
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: admin

创建后在查询其在k8s集群中生成的唯一名称，此处为hmdt-gitlab-ci-token-86n89

$ kubectl describe sa hmdt-gitlab-ci -n hmdt
Name: hmdt-gitlab-ci
Namespace: hmdt
Labels: <none>
Annotations: kubectl.`Kubernetes`.io/last-applied-configuration:
 {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"hmdt-gitlab-ci","namespace":"hmdt"}}
Image pull secrets: <none>
Mountable secrets: hmdt-gitlab-ci-token-86n89
Tokens: hmdt-gitlab-ci-token-86n89
Events: <none>

然后根据上面的Secret找到CA证书

$ kubectl get secret hmdt-gitlab-ci-token-86n89 -n hmdt -o json | jq -r '.data["ca.crt"]' | base64 -d

再找到对应的 Token

$ kubectl get secret hmdt-gitlab-ci-token-86n89 -n hmdt -o json | jq -r '.data.token' | base64 -d

Kubernetes关联GitLab配置

进入gitlab Kubernetes集群配置页面，填写相关信息，让gitlab自动连接上集群环境

注意，需要将此处取消勾选，否则gitlab会自动创建新的用户账户，而不使用已经创建好的用户账户，在运行过程中会报无权限错误

不取消导致的报错如下，gitlab创建了新的用户账户hmdt-prod-service-account，但没有操作指定命名空间的权限

GitLab环境配置

创建环境

名称和url可以按需自定义

CI脚本配置

最终配置CI文件如下，该文件使用DinD方式构建Dockerfile

image: docker:19.03

variables:
 MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode -Dmaven.test.skip=true"
 MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
 DOCKER_DRIVER: overlay
 DOCKER_HOST: tcp://localhost:2375
 DOCKER_TLS_CERTDIR: ""
 SPRING_PROFILES_ACTIVE: docker
 IMAGE_VERSION: "1.8.6"
 DOCKER_REGISTRY_MIRROR: "https://XXX.mirror.aliyuncs.com"

stages:
 - test
 - package
 - review
 - deploy

maven-build:
 image: maven:3-jdk-8
 stage: test
 retry: 2
 script:
 - mvn $MAVEN_CLI_OPTS clean package -U -B -T 2C
 artifacts:
 expire_in: 1 week
 paths:
 - target/*.jar

maven-scan:
 stage: test
 retry: 2
 image: maven:3-jdk-8
 script:
 - mvn $MAVEN_CLI_OPTS verify sonar:sonar

maven-deploy:
 stage: deploy
 retry: 2
 image: maven:3-jdk-8
 script:
 - mvn $MAVEN_CLI_OPTS deploy

docker-harbor-build:
 image: docker:19.03
 stage: package
 retry: 2
 services:
 - name: docker:19.03-dind
 alias: docker
 before_script:
 - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
 script:
 - docker build --pull -t "$CI_REGISTRY_IMAGE:$IMAGE_VERSION" .
 - docker push "$CI_REGISTRY_IMAGE:$IMAGE_VERSION"
 - docker logout $CI_REGISTRY

deploy_live:
 image: fjy8018/kubectl:v1.14.0
 stage: deploy
 retry: 2
 environment:
 name: prod
 url: https://XXXX
 script:
 - kubectl version
 - kubectl get pods -n hmdt
 - cd manifests/
 - sed -i "s/__IMAGE_VERSION_SLUG__/${IMAGE_VERSION}/" deployment.yaml
 - kubectl apply -f deployment.yaml
 - kubectl rollout status -f deployment.yaml
 - kubectl get pods -n hmdt

若需要使用Kaniko构建Dockerfile，则配置如下

注意，其中依赖的镜像gcr.io/kaniko-project/executor:debug属于谷歌镜像仓库，可能存在无法拉取的情况

image: docker:19.03

variables:
 MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode -Dmaven.test.skip=true"
 MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
 DOCKER_DRIVER: overlay
 DOCKER_HOST: tcp://localhost:2375
 DOCKER_TLS_CERTDIR: ""
 SPRING_PROFILES_ACTIVE: docker
 IMAGE_VERSION: "1.8.6"
 DOCKER_REGISTRY_MIRROR: "https://XXX.mirror.aliyuncs.com"

cache:
 paths:
 - target/

stages:
 - test
 - package
 - review
 - deploy

maven-build:
 image: maven:3-jdk-8
 stage: test
 retry: 2
 script:
 - mvn $MAVEN_CLI_OPTS clean package -U -B -T 2C
 artifacts:
 expire_in: 1 week
 paths:
 - target/*.jar

maven-scan:
 stage: test
 retry: 2
 image: maven:3-jdk-8
 script:
 - mvn $MAVEN_CLI_OPTS verify sonar:sonar

maven-deploy:
 stage: deploy
 retry: 2
 image: maven:3-jdk-8
 script:
 - mvn $MAVEN_CLI_OPTS deploy


docker-harbor-build:
 stage: package
 retry: 2
 image:
 name: gcr.io/kaniko-project/executor:debug
 entrypoint: [""]
 script:
 - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
 - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$IMAGE_VERSION

deploy_live:
 image: fjy8018/kubectl:v1.14.0
 stage: deploy
 retry: 2
 environment:
 name: prod
 url: https://XXXX
 script:
 - kubectl version
 - kubectl get pods -n hmdt
 - cd manifests/
 - sed -i "s/__IMAGE_VERSION_SLUG__/${IMAGE_VERSION}/" deployment.yaml
 - kubectl apply -f deployment.yaml
 - kubectl rollout status -f deployment.yaml
 - kubectl get pods -n hmdt

执行流水线

runner自动扩缩容

Kubernetes中的runner会根据任务多少自动扩缩容，目前配置的上限为10个

Grafana也能监控到集群在构建过程中的资源使用情况

使用`DinD`构建`Dockerfile`结果

使用`Kaniko`构建`Dockerfile`的结果

部署结果

执行部署时gitlab会自动注入配置好的kubectl config

构建结果

部署完成后可在环境配置页中查看部署结果，只有成功的部署才会被记录

k8s共享存储以及Redis高可用集群快速搭建

Wed, 20 Nov 2019 15:55:00 +0800

介绍

StorageClass 为管理员提供了描述存储 "类" 的方法。不同的类型可能会映射到不同的服务质量等级或备份策略，通常由运维制定的任意策略

用户通过在 PersistentVolumeClaim 中包含存储类来请求动态供应的存储，动态卷供应的实现基于 storage.k8s.io API 组中的 StorageClass API 对象。集群管理员可以根据需要定义多个 StorageClass 对象，每个对象指定一个卷插件（又名 provisioner），卷插件向卷供应商提供在创建卷时需要的数据卷信息及相关参数。

环境

Kubernetes：v1.14.0

Docker：17.03.1-ce

机器数：$3 \times master + 2 \times worker$

其中，master3、worker1、worker2都挂载了2块磁盘，非系统盘为纯净未分区磁盘，用于提供给GlusterFs管理

安装GlusterFs

在所有可能被调度到pod的节点安装glusterfs

$ yum install -y glusterfs glusterfs-fuse net-tools

启动GlusterFs与heketi

使用yaml文件启动glusterfs及其操作工具heketi

glusterfs-daemonset.yaml

---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
 name: glusterfs
 labels:
 glusterfs: daemonset
 annotations:
 description: GlusterFS DaemonSet
 tags: glusterfs
spec:
 template:
 metadata:
 name: glusterfs
 labels:
 glusterfs: pod
 glusterfs-node: pod
 spec:
 nodeSelector:
 storagenode: glusterfs
 hostNetwork: true
 containers:
 - image: gluster/gluster-centos:latest
 imagePullPolicy: IfNotPresent
 name: glusterfs
 env:
 # alternative for /dev volumeMount to enable access to *all* devices
 - name: HOST_DEV_DIR
 value: "/mnt/host-dev"
 # set GLUSTER_BLOCKD_STATUS_PROBE_ENABLE to "1" so the
 # readiness/liveness probe validate gluster-blockd as well
 - name: GLUSTER_BLOCKD_STATUS_PROBE_ENABLE
 value: "1"
 - name: GB_GLFS_LRU_COUNT
 value: "15"
 - name: TCMU_LOGDIR
 value: "/var/log/glusterfs/gluster-block"
 resources:
 requests:
 memory: 100Mi
 cpu: 100m
 volumeMounts:
 - name: glusterfs-heketi
 mountPath: "/var/lib/heketi"
 - name: glusterfs-run
 mountPath: "/run"
 - name: glusterfs-lvm
 mountPath: "/run/lvm"
 - name: glusterfs-etc
 mountPath: "/etc/glusterfs"
 - name: glusterfs-logs
 mountPath: "/var/log/glusterfs"
 - name: glusterfs-config
 mountPath: "/var/lib/glusterd"
 - name: glusterfs-host-dev
 mountPath: "/mnt/host-dev"
 - name: glusterfs-misc
 mountPath: "/var/lib/misc/glusterfsd"
 - name: glusterfs-block-sys-class
 mountPath: "/sys/class"
 - name: glusterfs-block-sys-module
 mountPath: "/sys/module"
 - name: glusterfs-cgroup
 mountPath: "/sys/fs/cgroup"
 readOnly: true
 - name: glusterfs-ssl
 mountPath: "/etc/ssl"
 readOnly: true
 - name: kernel-modules
 mountPath: "/usr/lib/modules"
 readOnly: true
 securityContext:
 capabilities: {}
 privileged: true
 readinessProbe:
 timeoutSeconds: 3
 initialDelaySeconds: 40
 exec:
 command:
 - "/bin/bash"
 - "-c"
 - "if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh readiness; else systemctl status glusterd.service; fi"
 periodSeconds: 25
 successThreshold: 1
 failureThreshold: 50
 livenessProbe:
 timeoutSeconds: 3
 initialDelaySeconds: 40
 exec:
 command:
 - "/bin/bash"
 - "-c"
 - "if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh liveness; else systemctl status glusterd.service; fi"
 periodSeconds: 25
 successThreshold: 1
 failureThreshold: 50
 volumes:
 - name: glusterfs-heketi
 hostPath:
 path: "/var/lib/heketi"
 - name: glusterfs-run
 - name: glusterfs-lvm
 hostPath:
 path: "/run/lvm"
 - name: glusterfs-etc
 hostPath:
 path: "/etc/glusterfs"
 - name: glusterfs-logs
 hostPath:
 path: "/var/log/glusterfs"
 - name: glusterfs-config
 hostPath:
 path: "/var/lib/glusterd"
 - name: glusterfs-host-dev
 hostPath:
 path: "/dev"
 - name: glusterfs-misc
 hostPath:
 path: "/var/lib/misc/glusterfsd"
 - name: glusterfs-block-sys-class
 hostPath:
 path: "/sys/class"
 - name: glusterfs-block-sys-module
 hostPath:
 path: "/sys/module"
 - name: glusterfs-cgroup
 hostPath:
 path: "/sys/fs/cgroup"
 - name: glusterfs-ssl
 hostPath:
 path: "/etc/ssl"
 - name: kernel-modules
 hostPath:
 path: "/usr/lib/modules"

heketi-security.yaml 用户与权限相关

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: heketi-clusterrolebinding
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: heketi-clusterrole
subjects:
- kind: ServiceAccount
 name: heketi-service-account
 namespace: default

---

apiVersion: v1
kind: ServiceAccount
metadata:
 name: heketi-service-account
 namespace: default

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: heketi-clusterrole
rules:
- apiGroups:
 - ""
 resources:
 - pods
 - pods/status
 - pods/exec
 verbs:
 - get
 - list
 - watch
 - create

heketi-deployment.yaml 启动服务并使用ingress四层反向代理

kind: Service
apiVersion: v1
metadata:
 name: heketi
 labels: 
 glusterfs: heketi-service
 deploy-heketi: support
 annotations: 
 description: Exposes Heketi Service
spec:
 selector: 
 name: heketi
 ports:
 - name: heketi
 port: 80
 targetPort: 8080

---

apiVersion: v1
kind: ConfigMap
metadata:
 name: tcp-services
 namespace: ingress-nginx
data:
 "30001": default/heketi:80

---

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
 name: heketi
 labels: 
 glusterfs: heketi-deployment
 annotations: 
 description: Defines how to deploy Heketi
spec:
 replicas: 1
 template:
 metadata:
 name: heketi
 labels: 
 name: heketi
 glusterfs: heketi-pod
 spec:
 serviceAccountName: heketi-service-account
 containers:
 - image: heketi/heketi:6
 imagePullPolicy: Always
 name: heketi
 env:
 - name: HEKETI_EXECUTOR
 value: "kubernetes"
 - name: HEKETI_DB_PATH
 value: "/var/lib/heketi/heketi.db"
 - name: HEKETI_FSTAB
 value: "/var/lib/heketi/fstab"
 - name: HEKETI_SNAPSHOT_LIMIT
 value: "14"
 - name: HEKETI_KUBE_GLUSTER_DAEMONSET
 value: "y"
 ports:
 - containerPort: 8080
 volumeMounts:
 - name: db
 mountPath: /var/lib/heketi
 readinessProbe:
 timeoutSeconds: 3
 initialDelaySeconds: 3
 httpGet: 
 path: /hello
 port: 8080
 livenessProbe:
 timeoutSeconds: 3
 initialDelaySeconds: 30
 httpGet: 
 path: /hello
 port: 8080
 volumes:
 - name: db
 hostPath:
 path: "/heketi-data"

GlusterFS初始化

创建

使用heketi-cli初始化GlusterFS节点，找到heketi容器并进入

$ docker ps | grep hek
9c92cf25e4bb heketi/heketi@sha256:44058f20a91211234abc533c9b4b5c42764b8cd740e21d03377bee14ebca4497 "/usr/bin/heketi-s..." About a minute ago Up 58 seconds k8s_he eti_heketi-68f9dfdfbf-w4qrd_default_c0566bf2-09ff-11ea-b6ce-000c298872e2_0
ebafb92c4dc1 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" About a minute ago Up About a minute k8s_POD_heketi-68f9dfdfbf-w4qrd_default_c0566bf2-09ff-11ea-b6ce-000c298872e2_0
$ docker exec -it 9c92 /bin/bash

配置容器本地环境变量，并配置节点json文件

$ export HEKETI_CLI_SERVER=http://localhost:8080
$ vi topology.json

主机名需要唯一并可以访问

{
 "clusters": [
 {
 "nodes": [
 {
 "node": {
 "hostnames": {
 "manage": [
 "k8smaster3"
 ],
 "storage": [
 "192.168.1.17"
 ]
 },
 "zone": 1
 },
 "devices": ["/dev/sdb"]
 },
 {
 "node": {
 "hostnames": {
 "manage": [
 "k8sworker1"
 ],
 "storage": [
 "192.168.1.18"
 ]
 },
 "zone": 1
 },
 "devices": ["/dev/sdb"]
 },
 {
 "node": {
 "hostnames": {
 "manage": [
 "k8sworker2"
 ],
 "storage": [
 "192.168.1.19"
 ]
 },
 "zone": 1
 },
 "devices": ["/dev/sdb"]
 }
 ]
 }
 ]
}

装载json文件

$ heketi-cli topology load --json topology.json
 Found node k8smaster3 on cluster 84d8a3a128c43a2a7d6c36aa6833823f
 Adding device /dev/sdb ... OK
 Found node k8sworker1 on cluster 84d8a3a128c43a2a7d6c36aa6833823f
 Adding device /dev/sdb ... OK
 Found node k8sworker2 on cluster 84d8a3a128c43a2a7d6c36aa6833823f
 Adding device /dev/sdb ... OK

可能出现的问题

1. 无法初始化

报错信息为

Creating cluster ... ID: 84d8a3a128c43a2a7d6c36aa6833823f
 Allowing file volumes on cluster.
 Allowing block volumes on cluster.
 Creating node k8smaster3 ... ID: b97a53e296850d59bbdd4043b13ebf78
 Adding device /dev/sdb ... Unable to add device: Unable to execute command on glusterfs-4vgh8: WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/centos/root not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/sda1 not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/centos/swap not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/sda2 not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds.
 Can't initialize physical volume "/dev/sdb" of volume group "vg_9bc3bdaff53104647cd24abe9ca60745" without -ff
 /dev/sdb: physical volume not initialized.
 Creating node k8sworker1 ... ID: 3a57327dbd24d9aedcae01f72857ef5e
 Adding device /dev/sdb ... Unable to add device: Unable to execute command on glusterfs-wzcf8: WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/centos/root not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/sda1 not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/centos/swap not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/sda2 not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds.
 Can't initialize physical volume "/dev/sdb" of volume group "vg_c6a688843cd06e333c83e56814c95c55" without -ff
 /dev/sdb: physical volume not initialized.
 Creating node k8sworker2 ... ID: 141f497c0797b26a22a31288ca97acf1

通过查阅资料，发现如下issue，官方提供磁盘擦除命令，执行后重新初始化节点

deploy fails using k8s 1.7.1 when heketi tries to add devices to cluster #2 #279

Destroy the heketi and GlusterFS pods, then try the following on each of the nodes:
pvdisplay -C -o pv_name,vg_name # look for VGs on PVs on your target devices
lvremove -ff <any VGs found above>
vgremove -f <any VGs found above>
pvremove -f <any PVs found above>
wipefs -a <target device>
Then try gk-deploy from the beginning again. Naturally, double-check that you’re definitely removing something only on the target devices. :)

进入挂载磁盘节点，查看当前磁盘信息

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 40G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 39G 0 part
 ├─centos-root 253:0 0 35G 0 lvm /
 └─centos-swap 253:1 0 4G 0 lvm
sdb 8:16 0 40G 0 disk
sr0 11:0 1 1024M 0 rom

查看pvs名称

$ pvs
 PV VG Fmt Attr PSize PFree
 /dev/sda2 centos lvm2 a-- <39.00g 0
 /dev/sdb vg_c6a688843cd06e333c83e56814c95c55 lvm2 a-- 39.87g 39.87g

抹除磁盘

$ pvdisplay -C -o /dev/sdb,vg_c6a688843cd06e333c83e56814c95c55
$ lvremove -ff vg_c6a688843cd06e333c83e56814c95c55
$ vgremove -f vg_c6a688843cd06e333c83e56814c95c55
$ pvremove -f /dev/sdb
$ wipefs -a /dev/sdb

抹除后重新进入容器创建即可

验证

进入3个存储节点，检查创建情况，此处以worker1为例

$ netstat -ntlp | grep glusterd
tcp 0 0 0.0.0.0:24007 0.0.0.0:* LISTEN 25997/glusterd
$ docker ps | grep glus
eae6ef1d4a37 22a008905c35 "/usr/local/bin/up..." 31 minutes ago Up 31 minutes k8s_glusterfs_glusterfs-wzcf8_default_940d0f57-09ff-11ea-b6ce-000c298872e2_0
858fd84edbc2 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 31 minutes ago Up 31 minutes k8s_POD_glusterfs-wzcf8_default_940d0f57-09ff-11ea-b6ce-000c298872e2_0
$ docker exec -it eae6 /bin/bash
$ gluster pool list
UUID Hostname State
6d4eda3d-a7c2-4e27-999c-972106c54c3d k8smaster3 Connected
56a6e1d4-46db-4ff2-83eb-39c313bc9197 192.168.1.19 Connected
a8ac96bd-b596-433c-8e2a-e3a97873886b localhost Connected

创建storage-class

定义StorageClass，通过访问ingress-nginx四层代理暴露的端口管理glusterfs

glusterfs-storage-class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: glusterfs-storage-class
provisioner: kubernetes.io/glusterfs
parameters:
 resturl: "http://192.168.1.16:30001"
 restauthenabled: "false"

创建PVC

创建PVC，可以被单个节点映射为ReadWriteOnce,或者多个节点映射为ReadOnlyMany,但不能同时使用这两种方式来映射。

glusterfs-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: glusterfs-pvc
spec:
 storageClassName: glusterfs-storage-class
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi

检查pvc，可见已经是已绑定(Bound)状态

$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
glusterfs-pvc Bound pvc-7c4f94cd-0a04-11ea-b6ce-000c298872e2 1Gi RWO glusterfs-storage-class 128m

基于Helm安装Redis-HA

访问charts官方仓库：https://github.com/helm/charts

在stable目录下找到redis-ha目录，修改values.yaml文件如下，主要配置存储类为storageClass: glusterfs-storage-class，然后修改用户名和密码等相关信息即可

image:
 repository: redis
 tag: 5.0.5-alpine
 pullPolicy: IfNotPresent
replicas: 3
labels: {}
configure-service-account/
serviceAccount:
 create: truetemplate
Automatically proxies to Redis master.
haproxy:
 enabled: false
 readOnly:
 enabled: false
 port: 6380
 replicas: 1
 image:
 repository: haproxy
 tag: 2.0.4
 pullPolicy: IfNotPresent
 annotations: {}
 resources: {}
 service:
 type: ClusterIP
 loadBalancerIP:
 annotations: {}
 serviceAccount:
 create: true
 exporter:
 image:
 repository: quay.azk8s.cn/prometheus/haproxy-exporter
 tag: v0.9.0
 enabled: false
 port: 9101
 init:
 resources: {}
 timeout:
 connect: 4s
 server: 30s
 client: 30s
 securityContext:
 runAsUser: 1000
 fsGroup: 1000
 runAsNonRoot: true
rbac:
 create: true
sysctlImage:
 enabled: false
 command: []
 registry: docker.io
 repository: busybox
 tag: 1.31.1
 pullPolicy: Always
 mountHostSys: false
configure-multiple-schedulers/
redis:
 port: 6379
 masterGroupName: mymaster
 config:
 min-replicas-to-write: 1unlimited.instance. Default is volatile-lru.with a new slave. The only way to prevent this is to enable diskless replication.
 save: "900 1"the disk as intermediate storage. Default is false.
 repl-diskless-sync: "yes"
 rdbcompression: "yes"
 rdbchecksum: "yes"
 resources: {}

sentinel:
 port: 26379
 quorum: 2
 config:
 down-after-milliseconds: 10000
 failover-timeout: 180000
 parallel-syncs: 5
 resources: {}

securityContext:
 runAsUser: 1000
 fsGroup: 1000
 runAsNonRoot: true
assign-pod-node/#nodeselectorassign-pod-node/#taints-and-tolerations-beta-featureassign-pod-node/#affinity-and-anti-affinity
nodeSelector: {}
requiredDuringSchedulingIgnoredDuringExecution as opposed to preferred.assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature
hardAntiAffinity: true
additionalAffinities: {}
affinity: |
exporter:
 enabled: true
 image: oliver006/redis_exporter
 tag: v0.31.0
 pullPolicy: IfNotPresent
 port: 9121
 scrapePath: /metrics
 resources: {}
 extraArgs: {}
 serviceMonitor:
 enabled: falsePrometheus Operator
podDisruptionBudget: {}
auth: true
redisPassword: password
authKey: auth
persistentVolume:
 enabled: true
 storageClass: glusterfs-storage-class
 accessModes:
 - ReadWriteOnce
 size: 100Mi
 annotations: {}
init:
 resources: {}

hostPath:
 chown: true

从修改后的本地文件安装redis-ha

$ helm install /root/redis-ha/ --name k8s-redis-ha --namespace env-prod
NAME: k8s-redis-ha
LAST DEPLOYED: Mon Nov 18 22:13:20 2019
NAMESPACE: env-prod
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
k8s-redis-ha-configmap 4 1s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
k8s-redis-ha-server-0 0/3 Pending 0 0s

==> v1/Role
NAME AGE
k8s-redis-ha 1s

==> v1/RoleBinding
NAME AGE
k8s-redis-ha 1s

==> v1/Secret
NAME TYPE DATA AGE
k8s-redis-ha Opaque 1 1s

==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
k8s-redis-ha ClusterIP None <none> 6379/TCP,26379/TCP,9121/TCP 1s
k8s-redis-ha-announce-0 ClusterIP 10.101.55.132 <none> 6379/TCP,26379/TCP,9121/TCP 1s
k8s-redis-ha-announce-1 ClusterIP 10.98.159.185 <none> 6379/TCP,26379/TCP,9121/TCP 1s
k8s-redis-ha-announce-2 ClusterIP 10.102.215.171 <none> 6379/TCP,26379/TCP,9121/TCP 1s

==> v1/ServiceAccount
NAME SECRETS AGE
k8s-redis-ha 1 1s

==> v1/StatefulSet
NAME READY AGE
k8s-redis-ha-server 0/3 1s


NOTES:
Redis can be accessed via port 6379 and Sentinel can be accessed via port 26379 on the following DNS name from within your cluster:
k8s-redis-ha.env-prod.svc.cluster.local

To connect to your Redis server:
1. To retrieve the redis password:
 echo $(kubectl get secret k8s-redis-ha -o "jsonpath={.data['auth']}" | base64 --decode)

2. Connect to the Redis master pod that you can use as a client. By default the k8s-redis-ha-server-0 pod is configured as the master:

 kubectl exec -it k8s-redis-ha-server-0 sh -n env-prod

3. Connect using the Redis CLI (inside container):

 redis-cli -a <REDIS-PASS-FROM-SECRET>

检查pvc创建情况

$ kubectl get pvc -n env-prod
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-k8s-redis-ha-server-0 Bound pvc-9348a01f-0a0d-11ea-b6ce-000c298872e2 1Gi RWO glusterfs-storage-class 62m
data-k8s-redis-ha-server-1 Bound pvc-9a8defcf-0a0d-11ea-b6ce-000c298872e2 1Gi RWO glusterfs-storage-class 62m
data-k8s-redis-ha-server-2 Bound pvc-a1975bf8-0a0d-11ea-b6ce-000c298872e2 1Gi RWO glusterfs-storage-class 62m

检查pv

$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-7c4f94cd-0a04-11ea-b6ce-000c298872e2 1Gi RWO Delete Bound default/glusterfs-pvc glusterfs-storage-class 128m
pvc-9348a01f-0a0d-11ea-b6ce-000c298872e2 1Gi RWO Delete Bound env-prod/data-k8s-redis-ha-server-0 glusterfs-storage-class 63m
pvc-9a8defcf-0a0d-11ea-b6ce-000c298872e2 1Gi RWO Delete Bound env-prod/data-k8s-redis-ha-server-1 glusterfs-storage-class 63m
pvc-a1975bf8-0a0d-11ea-b6ce-000c298872e2 1Gi RWO Delete Bound env-prod/data-k8s-redis-ha-server-2 glusterfs-storage-class 63m

配置ingress-nginx四层反向代理

apiVersion: v1
kind: ConfigMap
metadata:
 name: tcp-services
 namespace: ingress-nginx
data:
 "30001": default/heketi:80
 "30002": env-prod/k8s-redis-ha:6379
 "30003": env-prod/k8s-redis-ha:26379

使用redis客户端连接即可

k8s监控平台搭建

Tue, 12 Nov 2019 21:33:00 +0800

架构

目前主流容器化监控平台采用Prometheus+Grafana架构，下图为Prometheus架构图，可以主动发现各种指标和采集数据，实现

监控每个节点基础指标
监控每个容器基础指标
监控k8s集群组件基础指标

基础环境

Kubernetes：v1.14.0

Docker：17.03.1-ce

机器数：$3 \times master + 2 \times worker$

部署

Helm

Helm作为k8s平台软件包管理工具，结合prometheus-operator可以类似插件的方式快速搭建监控体系，Helm旗下包含charts项目，为软件包仓库

Helm官方仓库：https://github.com/helm/helm

charts官方仓库：https://github.com/helm/charts

安装Helm

通过官方仓库查看目前最新版为v2.16.0，但由于仍有不稳定的情况，故不推荐该版本，原因可通过其issue查看

此处选择安装v2.15.2的版本，下载页面

解压并放到可执行目录下

$ tar -zxvf helm-v2.15.2-linux-amd64.tar.gz
$ mv linux-amd64/helm /usr/local/bin/

Tiller安装

Tiller 是以 Deployment 方式部署在 Kubernetes 集群中的，由于国内网络原因，需要配置为阿里云镜像

$ helm init --client-only --stable-repo-url https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts/
$ helm repo add incubator https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts-incubator/
$ helm repo update

因为官方的镜像无法拉取，使用-i指定自己的镜像

$ helm init --service-account tiller --upgrade -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.15.2 --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

创建TLS认证服务端

$ helm init --service-account tiller --upgrade -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.15.2 --tiller-tls-cert /etc/kubernetes/ssl/tiller001.pem --tiller-tls-key /etc/kubernetes/ssl/tiller001-key.pem --tls-ca-cert /etc/kubernetes/ssl/ca.pem --tiller-namespace kube-system --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

Tiller授权

Tiller 需要访问ApiServer去对集群进行操作。目前的 Tiller 部署时默认没有定义授权的 ServiceAccount，这会导致访问 API Server 时被拒绝。所以我们需要明确为 Tiller 部署添加授权

创建ServiceAccount和创建角色绑定

$ kubectl create serviceaccount --namespace kube-system tiller
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

验证

查看Tiller的ServiceAccount，需要跟创建的名字一致：tiller

$ kubectl get deploy --namespace kube-system tiller-deploy -o yaml|grep serviceAccount
 serviceAccount: tiller
 serviceAccountName: tiller

验证pods

$ kubectl -n kube-system get pods|grep tiller
tiller-deploy-b6c45d5bc-jhntj 1/1 Running 0 2d8h

验证版本

$ helm version
Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}

Operator

可以自定义资源类型(CDR)结合自定义控制器实现

样例

apiVersion: app.example.com/v1
kind: AppService
metadata:
 name: nginx-app
spec:
 size: 2
 image: nginx:1.7.9
 ports:
 - port: 80
 targetPort: 80
 nodePort: 30002

Charts

charts项目下stable目录里存放所有软件包内容，由于其默认从多处镜像仓库中拉取镜像，可能存在网络问题，需结合实际情况修改镜像配置，手动安装软件包

推荐配置地址：

quay.io 替换为 quay.azk8s.cn
gcr.io/google-containers 替换为 registry.cn-hangzhou.aliyuncs.com/google-containers

本次需要安装的软件包为prometheus-operator，但其还包含多个依赖软件如下

grafana
kube-state-metrics
prometheus-node-exporter

若需要手动安装则需要按照整理目录如下

prometheus-operator/
├── charts
│   ├── grafana
│   ├── kube-state-metrics
│   └── prometheus-node-exporter
...

手动安装

Etcd监控配置

监控etcd需要额外配置，而且较为复杂，具体步骤如下

查看etcd在集群中的配置，由于本k8s集群基于kubeadm搭建，故etcd通过容器方式运行，查看方法

# 查看etcd所在pod名称
$ kubectl get pod --all-namespaces | grep etcd
kube-system etcd-k8smaster1 1/1 Running 0 7d1h
# 展开etcd详细配置
$ kubectl get pod -n kube-system etcd-k8smaster1 -o yaml

配置内容查看其健康检查端点，--listen-client-urls指明其监听在https端点，故需要配置安全证书才能访问

apiVersion: v1
kind: Pod
metadata:
 annotations:
 kubernetes.io/config.hash: 6119961323d801d05a7dd23e429cda3f
 kubernetes.io/config.mirror: 6119961323d801d05a7dd23e429cda3f
 kubernetes.io/config.seen: "2019-11-05T19:17:28.480984831+08:00"
 kubernetes.io/config.source: file
 creationTimestamp: "2019-11-05T11:18:53Z"
 labels:
 component: etcd
 tier: control-plane
 name: etcd-k8smaster1
 namespace: kube-system
 resourceVersion: "876368"
 selfLink: /api/v1/namespaces/kube-system/pods/etcd-k8smaster1
 uid: 0ccf5e91-ffbe-11e9-a5d2-000c298872e2
spec:
 containers:
 - command:
 - etcd
 - --advertise-client-urls=https://192.168.1.15:2379
 - --cert-file=/etc/kubernetes/pki/etcd/server.crt
 - --client-cert-auth=true
 - --data-dir=/var/lib/etcd
 - --initial-advertise-peer-urls=https://192.168.1.15:2380
 - --initial-cluster=k8smaster1=https://192.168.1.15:2380
 - --key-file=/etc/kubernetes/pki/etcd/server.key
 - --listen-client-urls=https://127.0.0.1:2379,https://192.168.1.15:2379
 - --listen-peer-urls=https://192.168.1.15:2380
 - --name=k8smaster1
 - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
 - --peer-client-cert-auth=true
 - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
 - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
 - --snapshot-count=10000
 - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
 image: registry.cn-hangzhou.aliyuncs.com/imooc/etcd:3.3.10
 imagePullPolicy: IfNotPresent
 livenessProbe:
...
hostIP: 192.168.1.15
...

配置安全证书

k8s创建etcd时会自动创建相关证书，当前集群版本为1.14，证书位于/etc/kubernetes/pki/etcd/中

$ ls /etc/kubernetes/pki/etcd/
ca.crt ca.key healthcheck-client.crt healthcheck-client.key peer.crt peer.key server.crt server.key

可以通过curl命令加载证书测试能否访问健康检查端点

$ curl -k --cert /etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key https://192.168.1.15:2379/metrics

若有如下输出则正常

# HELP etcd_debugging_mvcc_db_compaction_keys_total Total number of db keys compacted.
# TYPE etcd_debugging_mvcc_db_compaction_keys_total counter
etcd_debugging_mvcc_db_compaction_keys_total 1.030132e+06
# HELP etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds Bucketed histogram of db compaction pause duration.
# TYPE etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds histogram
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="1"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="2"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="4"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="8"} 0
...

创建证书秘钥

关联三个证书文件，创建k8s访问秘钥，秘钥名称为etcd-certs

$ kubectl create secret generic etcd-certs -n monitoring --from-file=/etc/kubernetes/pki/etcd/ca.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key

写入prometheus配置文件values.yaml

注意，此处路径前缀固定写法为/etc/prometheus/secrets/+秘钥名称+文件名称，若写错证书路径则无法在指标中查看到对应的target

... 
 serviceMonitor:
 ## Scrape interval. If not set, the Prometheus default scrape interval is used.
 ##
 interval: ""
 scheme: https
 insecureSkipVerify: true
 serverName: ""
 caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
 certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt
 keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key
...

创建

指定命名空间和文件目录，若命名空间不存在会自动创建

$ helm install /root/prometheus-operator/ --name k8s-prometheus --namespace monitoring

等待一段时间，输出如下则创建成功

NAME: k8s-prometheus
LAST DEPLOYED: Tue Nov 12 20:27:18 2019
NAMESPACE: monitoring
STATUS: DEPLOYED

RESOURCES:
==> v1/Alertmanager
NAME AGE
k8s-prometheus-prometheus-alertmanager 33s

==> v1/ClusterRole
NAME AGE
k8s-prometheus-grafana-clusterrole 34s
k8s-prometheus-prometheus-alertmanager 34s
k8s-prometheus-prometheus-operator 34s
k8s-prometheus-prometheus-operator-psp 34s
k8s-prometheus-prometheus-prometheus 34s
k8s-prometheus-prometheus-prometheus-psp 34s
psp-k8s-prometheus-kube-state-metrics 34s
psp-k8s-prometheus-prometheus-node-exporter 34s

==> v1/ClusterRoleBinding
NAME AGE
k8s-prometheus-grafana-clusterrolebinding 34s
k8s-prometheus-prometheus-alertmanager 34s
k8s-prometheus-prometheus-operator 34s
k8s-prometheus-prometheus-operator-psp 34s
k8s-prometheus-prometheus-prometheus 34s
k8s-prometheus-prometheus-prometheus-psp 34s
psp-k8s-prometheus-kube-state-metrics 34s
psp-k8s-prometheus-prometheus-node-exporter 34s

==> v1/ConfigMap
NAME DATA AGE
k8s-prometheus-grafana 1 34s
k8s-prometheus-grafana-config-dashboards 1 34s
k8s-prometheus-grafana-test 1 34s
k8s-prometheus-prometheus-apiserver 1 34s
k8s-prometheus-prometheus-cluster-total 1 34s
k8s-prometheus-prometheus-controller-manager 1 34s
k8s-prometheus-prometheus-etcd 1 34s
k8s-prometheus-prometheus-grafana-datasource 1 34s
k8s-prometheus-prometheus-k8s-coredns 1 34s
k8s-prometheus-prometheus-k8s-resources-cluster 1 34s
k8s-prometheus-prometheus-k8s-resources-namespace 1 34s
k8s-prometheus-prometheus-k8s-resources-node 1 34s
k8s-prometheus-prometheus-k8s-resources-pod 1 34s
k8s-prometheus-prometheus-k8s-resources-workload 1 34s
k8s-prometheus-prometheus-k8s-resources-workloads-namespace 1 34s
k8s-prometheus-prometheus-kubelet 1 34s
k8s-prometheus-prometheus-namespace-by-pod 1 34s
k8s-prometheus-prometheus-namespace-by-workload 1 34s
k8s-prometheus-prometheus-node-cluster-rsrc-use 1 34s
k8s-prometheus-prometheus-node-rsrc-use 1 34s
k8s-prometheus-prometheus-nodes 1 34s
k8s-prometheus-prometheus-persistentvolumesusage 1 34s
k8s-prometheus-prometheus-pod-total 1 34s
k8s-prometheus-prometheus-pods 1 34s
k8s-prometheus-prometheus-prometheus 1 34s
k8s-prometheus-prometheus-proxy 1 34s
k8s-prometheus-prometheus-scheduler 1 34s
k8s-prometheus-prometheus-statefulset 1 34s
k8s-prometheus-prometheus-workload-total 1 34s

==> v1/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
k8s-prometheus-prometheus-node-exporter 5 5 5 5 5 <none> 33s

==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
k8s-prometheus-grafana 1/1 1 1 33s
k8s-prometheus-kube-state-metrics 1/1 1 1 33s
k8s-prometheus-prometheus-operator 1/1 1 1 33s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
k8s-prometheus-grafana-769d6dfb69-mtbdg 2/2 Running 0 33s
k8s-prometheus-kube-state-metrics-76b8ff78cd-kjxvl 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-jqkvf 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-jxpzm 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-k8n6x 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-kpjzk 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-qc6kv 1/1 Running 0 33s
k8s-prometheus-prometheus-operator-5f894bc75d-kvkvs 2/2 Running 0 33s

==> v1/Prometheus
NAME AGE
k8s-prometheus-prometheus-prometheus 33s

==> v1/PrometheusRule
NAME AGE
k8s-prometheus-prometheus-alertmanager.rules 32s
k8s-prometheus-prometheus-etcd 32s
k8s-prometheus-prometheus-general.rules 32s
k8s-prometheus-prometheus-k8s.rules 32s
k8s-prometheus-prometheus-kube-apiserver.rules 32s
k8s-prometheus-prometheus-kube-prometheus-node-recording.rules 32s
k8s-prometheus-prometheus-kube-scheduler.rules 32s
k8s-prometheus-prometheus-kubernetes-absent 32s
k8s-prometheus-prometheus-kubernetes-apps 32s
k8s-prometheus-prometheus-kubernetes-resources 32s
k8s-prometheus-prometheus-kubernetes-storage 32s
k8s-prometheus-prometheus-kubernetes-system 32s
k8s-prometheus-prometheus-kubernetes-system-apiserver 32s
k8s-prometheus-prometheus-kubernetes-system-controller-manager 32s
k8s-prometheus-prometheus-kubernetes-system-kubelet 32s
k8s-prometheus-prometheus-kubernetes-system-scheduler 32s
k8s-prometheus-prometheus-node-exporter 32s
k8s-prometheus-prometheus-node-exporter.rules 32s
k8s-prometheus-prometheus-node-network 32s
k8s-prometheus-prometheus-node-time 32s
k8s-prometheus-prometheus-node.rules 32s
k8s-prometheus-prometheus-prometheus 32s
k8s-prometheus-prometheus-prometheus-operator 32s

==> v1/Role
NAME AGE
k8s-prometheus-grafana-test 34s

==> v1/RoleBinding
NAME AGE
k8s-prometheus-grafana-test 34s

==> v1/Secret
NAME TYPE DATA AGE
alertmanager-k8s-prometheus-prometheus-alertmanager Opaque 1 34s
k8s-prometheus-grafana Opaque 3 34s

==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
k8s-prometheus-grafana ClusterIP 10.100.203.67 <none> 80/TCP 33s
k8s-prometheus-kube-state-metrics ClusterIP 10.98.35.173 <none> 8080/TCP 33s
k8s-prometheus-prometheus-alertmanager ClusterIP 10.97.189.252 <none> 9093/TCP 33s
k8s-prometheus-prometheus-coredns ClusterIP None <none> 9153/TCP 34s
k8s-prometheus-prometheus-kube-controller-manager ClusterIP None <none> 10252/TCP 34s
k8s-prometheus-prometheus-kube-etcd ClusterIP None <none> 2379/TCP 34s
k8s-prometheus-prometheus-kube-proxy ClusterIP None <none> 10249/TCP 34s
k8s-prometheus-prometheus-kube-scheduler ClusterIP None <none> 10251/TCP 34s
k8s-prometheus-prometheus-node-exporter ClusterIP 10.100.146.95 <none> 9100/TCP 33s
k8s-prometheus-prometheus-operator ClusterIP 10.105.23.58 <none> 8080/TCP,443/TCP 33s
k8s-prometheus-prometheus-prometheus ClusterIP 10.99.7.136 <none> 9090/TCP 34s

==> v1/ServiceAccount
NAME SECRETS AGE
k8s-prometheus-grafana 1 34s
k8s-prometheus-grafana-test 1 34s
k8s-prometheus-kube-state-metrics 1 34s
k8s-prometheus-prometheus-alertmanager 1 34s
k8s-prometheus-prometheus-node-exporter 1 34s
k8s-prometheus-prometheus-operator 1 34s
k8s-prometheus-prometheus-prometheus 1 34s

==> v1/ServiceMonitor
NAME AGE
k8s-prometheus-prometheus-alertmanager 32s
k8s-prometheus-prometheus-apiserver 32s
k8s-prometheus-prometheus-coredns 32s
k8s-prometheus-prometheus-grafana 32s
k8s-prometheus-prometheus-kube-controller-manager 32s
k8s-prometheus-prometheus-kube-etcd 32s
k8s-prometheus-prometheus-kube-proxy 32s
k8s-prometheus-prometheus-kube-scheduler 32s
k8s-prometheus-prometheus-kube-state-metrics 32s
k8s-prometheus-prometheus-kubelet 32s
k8s-prometheus-prometheus-node-exporter 32s
k8s-prometheus-prometheus-operator 32s
k8s-prometheus-prometheus-prometheus 32s

==> v1beta1/ClusterRole
NAME AGE
k8s-prometheus-kube-state-metrics 34s

==> v1beta1/ClusterRoleBinding
NAME AGE
k8s-prometheus-kube-state-metrics 34s

==> v1beta1/MutatingWebhookConfiguration
NAME AGE
k8s-prometheus-prometheus-admission 33s

==> v1beta1/PodSecurityPolicy
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
k8s-prometheus-grafana false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
k8s-prometheus-grafana-test false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,downwardAPI,emptyDir,projected,secret
k8s-prometheus-kube-state-metrics false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false secret
k8s-prometheus-prometheus-alertmanager false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
k8s-prometheus-prometheus-node-exporter false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath
k8s-prometheus-prometheus-operator false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
k8s-prometheus-prometheus-prometheus false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim

==> v1beta1/Role
NAME AGE
k8s-prometheus-grafana 34s

==> v1beta1/RoleBinding
NAME AGE
k8s-prometheus-grafana 34s

==> v1beta1/ValidatingWebhookConfiguration
NAME AGE
k8s-prometheus-prometheus-admission 32s


NOTES:
The Prometheus Operator has been installed. Check its status by running:
 kubectl --namespace monitoring get pods -l "release=k8s-prometheus"

Visit https://github.com/coreos/prometheus-operator for instructions on how
to create & configure Alertmanager and Prometheus instances using the Operator.

升级软件包

一般用于修改yaml配置文件重载应用

$ helm upgrade k8s-prometheus /root/prometheus-operator/ -f /root/prometheus-operator/charts/grafana/values.yaml

删除

使用Helm的优势在于安装软件包和删除都非常方便，通过命令即可快速删除

普通删除，会放入回收站

$ helm delete k8s-prometheus

强制删除，不会放入回收站

$ helm delete k8s-prometheus --purge

对于当前监控组件来说，删除后同时要删除CRD

$ kubectl get crd | grep coreos

删除所有的CRD

$ kubectl delete crd $(kubectl get crd | grep coreos | awk '{print $1}')

访问指标面板

由于当前集群配置了ingress-nginx负载均衡，故新增两个配置文件即可访问Prometheus和Grafana

访问Grafana

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: prom-grafana
 namespace: monitoring
spec:
 rules:
 - host: your.domain
 http:
 paths:
 - backend:
 serviceName: k8s-prometheus-grafana
 servicePort: 80
 path: /

访问Prometheus

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: prometheus
 namespace: monitoring
spec:
 rules:
 - host: your.domain
 http:
 paths:
 - backend:
 serviceName: k8s-prometheus-prometheus-prometheus
 servicePort: web
 path: /

Prometheus

访问Prometheus web站点，首页支持执行PromeQL查询采集指标信息

访问target可见所有指标采集器都处于正常状态

访问监控与告警可查看所有告警规则

Grafana

首次访问会要求登陆

查看用户名和密码

在grafana软件包的values.yaml配置中有如下描述，声明密码被存储在k8s的secret中

# Administrator credentials when not using an existing secret (see below)
adminUser: admin
# adminPassword: strongpassword

# Use an existing secret for the admin user.
admin:
 existingSecret: ""
 userKey: admin-user
 passwordKey: admin-password

查看监控命名空间的秘钥

$ kubectl get secrets -n monitoring
NAME TYPE DATA AGE
alertmanager-k8s-prometheus-prometheus-alertmanager Opaque 1 34m
default-token-hrhgk kubernetes.io/service-account-token 3 2d10h
etcd-certs Opaque 3 2d8h
k8s-prometheus-grafana Opaque 3 34m
k8s-prometheus-grafana-test-token-nnnvk kubernetes.io/service-account-token 3 34m
k8s-prometheus-grafana-token-cdw2n kubernetes.io/service-account-token 3 34m
k8s-prometheus-kube-state-metrics-token-6rxp4 kubernetes.io/service-account-token 3 34m
k8s-prometheus-prometheus-admission Opaque 3 2d10h
k8s-prometheus-prometheus-alertmanager-token-hxqm9 kubernetes.io/service-account-token 3 34m
k8s-prometheus-prometheus-node-exporter-token-vjrcs kubernetes.io/service-account-token 3 34m
k8s-prometheus-prometheus-operator-token-bkhrb kubernetes.io/service-account-token 3 34m
k8s-prometheus-prometheus-prometheus-token-wf2v6 kubernetes.io/service-account-token 3 34m
prometheus-k8s-prometheus-prometheus-prometheus Opaque 1 37m
prometheus-k8s-prometheus-prometheus-prometheus-tls-assets Opaque 0 34m

k8s-prometheus-grafana 即目标秘钥，展示其详情

$ kubectl get secrets -n monitoring k8s-prometheus-grafana -o yaml
apiVersion: v1
data:
 admin-password: b3BlcmF0b3ItYWRtaW4=
 admin-user: cm9vdA==
 ldap-toml: ""
kind: Secret
metadata:
 creationTimestamp: "2019-11-12T12:27:55Z"
 labels:
 app: grafana
 chart: grafana-4.0.2
 heritage: Tiller
 release: k8s-prometheus
 name: k8s-prometheus-grafana
 namespace: monitoring
 resourceVersion: "1033230"
 selfLink: /api/v1/namespaces/monitoring/secrets/k8s-prometheus-grafana
 uid: da7e86d0-0547-11ea-a5d2-000c298872e2
type: Opaque

秘钥默认通过Base64 加密，通过在线网站对该Base64解密即可得到用户名和密码

登录后可查看Grafana已经被配置好的多种可视化面板提供指标展示

点击左上角即可查看已经配置好的指标展示面板

Etcd监控面板

节点监控

计算资源监控

Elasticsearch Docker高可用环境搭建

Tue, 10 Sep 2019 22:52:00 +0800

介绍

作为Elastic Stack 的核心，Elasticsearch 是一个分布式、RESTful 风格的搜索和数据分析引擎，本次主要用于存储业务数据和基于业务日志进行数据统计分析。

目标

本次目标是搭建个人项目所使用的小型es高可用环境，目前需要使用es的服务有些部署在美国的服务器，有些部署在国内的服务器，带宽延迟各不相同，为了保证应用连接es延迟最小，计划同时在美国和中国都部署节点，不同区域的应用连接不同的节点

基础环境

本次配置全局使用真实服务器，服务器配置如下

名称	CPU	内存	系统	Docker版本	磁盘大小	区域
es01	1	2G	CentOS 7	19.03.1	40 G	亚洲
es02	1	2G	CentOS 7	19.03.1	20G	美洲
es03	4	4G	CentOS 7	19.03.1	36G	美洲
es04	1	2G	CentOS 7	19.03.1	50G	亚洲

所有服务器部署的es版本均为：6.6.2

从表中可见内存和磁盘都不大，如果是公司项目配置肯定比这个好很多，但对于个人使用来说足够了，至于为什么是需要4台而不是3台在最后搭建完成后会进行分析

部署前准备

就算使用了Docker容器，elasticsearch仍然不像普通镜像那么简单启动，es对虚拟内存敏感，因此服务器必须是内核虚拟化KVM架构，不支持OpenVZ虚拟，参考官方说明

Production modeedit
The vm.max_map_count kernel setting needs to be set to at least 262144 for production use. Depending on your platform:
Linux

The vm.max_map_count setting should be set permanently in /etc/sysctl.conf:
$ grep vm.max_map_count /etc/sysctl.conf
vm.max_map_count=262144
To apply the setting on a live system type: sysctl -w vm.max_map_count=262144
macOS with Docker for Mac

The vm.max_map_count setting must be set within the xhyve virtual machine:
$ screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
Just press enter and configure the sysctl setting as you would for Linux:
sysctl -w vm.max_map_count=262144

按照指引设置内存参数

$ sudo sysctl -w vm.max_map_count=262144

前置知识

裂脑事件

Elasticsearch牺牲了一致性，以确保可用性和分区容错。其背后的原因是短期的不良行为比短期的不可用性问题少。换句话说，当群集中的Elasticsearch节点无法复制对数据的更改时，它们将继续为应用程序提供服务。当节点能够复制其数据时，它们将尝试聚合副本并实现最终的一致性。

Elasticsearch通过选举主节点来解决之前的问题，主节点负责数据库操作，例如创建新索引，在群集节点周围移动分片等等。主节点与其他节点主动协调其操作，确保数据可以由非主节点汇聚。

在某些情况下，先前的机制可能会失败，从而导致裂脑事件。当Elasticsearch集群分为两个区块时，若每个区块都有一个节点都认为它们是主节点，因为主节点将在数据上独立工作，数据一致性就会丢失。因此，节点将对相同的查询做出不同的响应。这将会是灾难性的事件，因为来自两个主节点的数据无法自动重新加入，并且需要相当多的手动工作来纠正这种情况。

节点部署

3个主节点节点均使用docker-compose命令部署，需要自行安装好相关环境

version: '2'
services:

 es01:
 image: docker.elastic.co/elasticsearch/elasticsearch:6.6.2
 container_name: es01
 environment:
 - cluster.name=docker-cluster
 - bootstrap.memory_lock=true
 - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
 - "discovery.zen.ping.unicast.hosts=es2.xxx.com,es3.xxx.com,es4.xxx.com"
 - "discovery.zen.minimum_master_nodes=2"
 - "network.publish_host=es1..xxx.com"
 - "node.name=es01-XXX"
 ulimits:
 memlock:
 soft: -1
 hard: -1
 volumes:
 - ./elasticsearch/data:/usr/share/elasticsearch/data
 ports:
 - 9200:9200
 - 9300:9300
 networks:
 - fjyesnet
 mem_limit: 1g

networks:
 fjyesnet:

其他节点配置均相同，修改服务发现域名配置即可，放行防火墙部分不再赘述，需要可以查看我以前发的文章

非主节点配置只需加入node.master=false配置即可

version: '2'
services:

 es01:
 image: docker.elastic.co/elasticsearch/elasticsearch:6.6.2
 container_name: es04
 environment:
 - cluster.name=docker-cluster
 - bootstrap.memory_lock=true
 - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
 - "discovery.zen.ping.unicast.hosts=es2.xxx.com,es3.xxx.com,es1.xxx.com"
 - "discovery.zen.minimum_master_nodes=2"
 - "network.publish_host=es4..xxx.com"
 - "node.name=es04-XXX"
 - "node.master=false"
 ulimits:
 memlock:
 soft: -1
 hard: -1
 volumes:
 - ./elasticsearch/data:/usr/share/elasticsearch/data
 ports:
 - 9200:9200
 - 9300:9300
 networks:
 - fjyesnet
 mem_limit: 1g

networks:
 fjyesnet:

参数解析

discovery.zen.ping.unicast.hosts配置广播地址，用于节点发现，此处配置的是指向其他节点IP的域名地址，该域名在任何节点均可以被访问
discovery.zen.minimum_master_nodes配置主节点最小数量，该参数十分重要，用于防止脑裂（split-brain）事件的发生。

原理：前面提到了脑裂事件发生的原因，而避免的方法就是保证至少3个节点可靠地工作，因为一个或两个节点不能形成多数投票，这也是为什么选择使用4个节点并将该值配置为2，确保分布式环境还可以随机宕机一个节点以保证不会出现脑裂问题，因此第4个节点最好配置为无法成为主节点的slave节点。而若仅配置为3个节点的高可用状态，其实是一种“伪高可用”，此时随机宕机一个节点将可能发生“脑裂”问题，2个节点的es环境无法进行主节点的选举，并且可能出现独立的分块。
network.publish_host用于提供其他节点服务发现的主机名，默认为本地主机名映射为IP的地址，但由于使用容器提供服务发现，并且不在同一个网段，故需要手工指定
mem_limit容器内存限制为1G

节点数量与配置的主节点数对应关系

`Master nodes`	`minimum_master_nodes`	备注
1	1
2	1	如果另一个节点宕机，集群将停止工作!
3	2
4	3
5	3
6	4

对可能产生疑问的解释

1.当前`docker-compose`已经更新到`3.7`了为什么还使用`2`这个老版本？

原因：最新文档地址，官方文档指出3.x版本已经不再支持mem_limit参数

Note: This replaces the older resource constraint options for non swarm mode in Compose files prior to version 3 (cpu_shares, cpu_quota, cpuset, mem_limit, memswap_limit, mem_swappiness), as described in Upgrading version 2.x to 3.x.

但对小内存机器以及数据量不会非常的大的环境，内存限制参数尤为重要，经过实测，空载或者小负载（1000 doc）以下的es节点内存占用大约为1.5G左右，故各环境限制为1G不会影响性能，不仅如此，es官方提供的部署文档使用的也是2.2版本

2.当前`Elasticsearch`已经更新到`7.3`了为什么还使用`6.6`这个老版本？

原因：7.x版本目前尚有许多外部服务不能兼容，例如graylog，并且Spring官方组件暂时也只能兼容到6.x的版本，所以综合考虑使用6.x的最新版6.6.2

graylog官方文档

Caution

Graylog 3.x does not work with Elasticsearch 7.x!

Spring官方文档

Compatibility Matrix

Spring Data Elasticsearch Elasticsearch

3.2.x (not yet released) 6.8.1

3.1.x 6.2.2

3.0.x 5.5.0

2.1.x 2.4.0

2.0.x 2.2.0

1.3.x 1.5.2

Spring Data Elasticsearch	Elasticsearch
3.2.x (not yet released)	6.8.1
3.1.x	6.2.2
3.0.x	5.5.0
2.1.x	2.4.0
2.0.x	2.2.0
1.3.x	1.5.2

3.一般高可用不是部署3个节点就足够了吗，为什么要4个节点？

原因：本次总共部署了3个主节点和一个非主节点，即4个都是数据节点，但只有3个参与选举。如果仅部署3个节点，那么3个都必须为主节点，此时若其中一个主节点宕机，那么不满足选举“至少3个节点可靠地工作”的条件，故新增多一个冗余数据节点，其不能竞选为主节点，但能对节点竞选做出响应，依然满足条件，详细算法可以参考Raft协议的实现原理

节点启动

使用docker-compose命令启动即可

$ docker-compose up -d

这种直接启动的方式会报文件夹无访问权限的错误，需要在命令执行后手动赋予文件夹访问权限

$ chmod 777 -R elasticsearch/
$ chown 1000:1000 -R elasticsearch/

若此时es容器已经停止则再执行一次启动命令即可

部署结果

使用chrome插件elasticsearch-head连接任意一个节点查看情况

4节点在线，接入业务系统后各节点情况如下图。

注意所有索引副本数都要是2或者3，否则都无法实现高可用效果

由于网络问题或者服务器问题导致其中一个节点下线，es会自动对各个节点的副本重新路由，此时集群颜色为黄色，仍然可用，稍等片刻会实现最终一致性

若此时主节点下线，整个集群会立刻重新选举出新的主节点并路由各个分片，保证24小时服务都处在可用状态

橙色表示尚未路由完成，但所有索引数据均可用

注意

若某些索引的副本数设置为1，则当某个节点宕机且其刚好存储该索引的副本，则整个集群的状态将立刻变成红色，意思是数据丢失

灰色表示已经下线的分片并且无法被重新路由

基于Mesos+Marathon的DevOps分布式环境搭建

Sun, 18 Aug 2019 22:01:00 +0800

基础环境

本地准备了4台虚拟机配置如下

虚拟机名称	IP	CPU	内存	磁盘
129	192.168.133.129	1	2G	20G
130	192.168.133.130	1	2G	20G
131	192.168.133.131	1	2G	20G
132	192.168.133.132	1	2G	20G

部署架构

部署架构图如下

各服务详细部署情况

Zookeeper

服务器	节点	端口	版本
129	zk-1-docker	2181	3.4.14
131	zk-2-docker	2181	3.4.14
132	zk-3-docker	2181	3.4.14

MesosMaster

服务器	节点	端口	版本
132	master	5050	1.7.0
130	master	5050	1.7.0
131	master	5050	1.7.0

MesosSlave

服务器	节点	端口	版本
131	slave2	5051	1.7.0
130	slave1	5051	1.7.0

marathon

服务器	服务	端口	版本
132	marathonMaster	8080	v1.6.496

marathon-lb

服务器	服务	端口	版本
129	marathon-lb	9090	v1.14.0

环境准备

开放防火墙

本次服务各主机将可能使用到以下端口

8080
9090
2181
2888
3888
5050
5051

逐台登陆服务器执行放行防火墙脚本

firewall-cmd --zone=public --add-port=8080/tcp --permanent
firewall-cmd --zone=public --add-port=2181/tcp --permanent
firewall-cmd --zone=public --add-port=5050/tcp --permanent
firewall-cmd --zone=public --add-port=5051/tcp --permanent
firewall-cmd --zone=public --add-port=9090/tcp --permanent
firewall-cmd --zone=public --add-port=2888/tcp --permanent
firewall-cmd --zone=public --add-port=3888/tcp --permanent
firewall-cmd --reload

安装Docker

本次所有服务均采用Docker方式部署，服务调度通过Docker in Docker的方式进行

由于Docker有可能出现版本影响环境问题，参考文章相关文章

故建议安装时指定版本

$ yum-config-manager \
 --add-repo \
 https://download.docker.com/linux/centos/docker-ce.repo
$ yum install docker-ce-19.03.1 docker-ce-cli-19.03.1 containerd.io

设置开机启动并运行服务

$ systemctl enable docker
$ systemctl start docker

安装docker-compose，脚本参考官方文档

$ sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

安装Zookeeper

鉴于zk 3.4和3.5版本的配置有一定差别，编写脚本时注意版本，此处使用的是3.4.14，以下配置文件注意hosts的微小差别

docker-compose-zoo-129.yml

version: '3.1'

services:
 zoo1:
 image: zookeeper:3.4.14
 restart: always
 network_mode: host
 extra_hosts:
 - "zoo2: 192.168.133.131"
 - "zoo3: 192.168.133.132"
 ports:
 - 2181:2181
 - 2888:2888
 - 3888:3888
 environment:
 ZOO_MY_ID: 1
 ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888

docker-compose-zoo-131.yml

version: '3.1'

services:

 zoo2:
 image: zookeeper:3.4.14
 restart: always
 network_mode: host
 extra_hosts:
 - "zoo1: 192.168.133.129"
 - "zoo3: 192.168.133.132"
 ports:
 - 2181:2181
 - 2888:2888
 - 3888:3888
 environment:
 ZOO_MY_ID: 2
 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=0.0.0.0:2888:3888 server.3=zoo3:2888:3888

docker-compose-zoo-132.yml

version: '3.1'

services:

 zoo3:
 image: zookeeper:3.4.14
 restart: always
 network_mode: host
 extra_hosts:
 - "zoo1: 192.168.133.129"
 - "zoo2: 192.168.133.131"
 ports:
 - 2181:2181
 - 2888:2888
 - 3888:3888
 environment:
 ZOO_MY_ID: 3
 ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=0.0.0.0:2888:3888

安装MesosMaster

MesosMaster也采用3节点高可用方案，由于要在各台机器上配置hosts过于麻烦，此处选择使用非hosts注册方式，故需要配置各自的IP地址

docker-compose-mesos-master-130.yml

version: '3.1'

services:

 mesos-master:
 image: mesosphere/mesos-master:1.7.0
 restart: always
 network_mode: host
 hostname: master
 command: --no-hostname_lookup --ip=192.168.133.130
 ports:
 - 5050:5050
 environment:
 MESOS_PORT: 5050
 MESOS_ZK: zk://192.168.133.129:2181/mesos
 MESOS_QUORUM: 1
 MESOS_REGISTRY: in_memory
 MESOS_LOG_DIR: /var/log/mesos
 MESOS_WORK_DIR: /var/tmp/mesos

docker-compose-mesos-master-131.yml

version: '3.1'

services:

 mesos-master:
 image: mesosphere/mesos-master:1.7.0
 restart: always
 network_mode: host
 hostname: master
 command: --no-hostname_lookup --ip=192.168.133.131
 ports:
 - 5050:5050
 environment:
 MESOS_PORT: 5050
 MESOS_ZK: zk://192.168.133.129:2181/mesos
 MESOS_QUORUM: 1
 MESOS_REGISTRY: in_memory
 MESOS_LOG_DIR: /var/log/mesos
 MESOS_WORK_DIR: /var/tmp/mesos

docker-compose-mesos-master-132.yml

version: '3.1'

services:

 mesos-master:
 image: mesosphere/mesos-master:1.7.0
 restart: always
 network_mode: host
 hostname: master
 command: --no-hostname_lookup --ip=192.168.133.132
 ports:
 - 5050:5050
 environment:
 MESOS_PORT: 5050
 MESOS_ZK: zk://192.168.133.129:2181/mesos
 MESOS_QUORUM: 1
 MESOS_REGISTRY: in_memory
 MESOS_LOG_DIR: /var/log/mesos
 MESOS_WORK_DIR: /var/tmp/mesos

安装MesosSlave

需要选择与master对应的版本号，同时也使用IP注册的方式

docker-compose-mesos-slave1.yml

version: '3.1'

services:

 mesos-slave1:
 image: mesosphere/mesos-slave:1.7.0
 restart: always
 network_mode: host
 hostname: slave1
 command: --no-systemd_enable_support --no-hostname_lookup --ip=192.168.133.131
 ports:
 - 5051:5051
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock
 - /cgroup:/cgroup
 - /sys:/sys
 - /usr/bin/docker:/usr/local/bin/docker
 environment:
 MESOS_PORT: 5051
 MESOS_MASTER: zk://192.168.133.129:2181/mesos
 MESOS_SWITCH_USER: 0
 MESOS_CONTAINERIZERS: docker,mesos
 MESOS_LOG_DIR: /var/log/mesos
 MESOS_WORK_DIR: /var/tmp/mesos
 privileged: true

docker-compose-mesos-slave2.yml

version: '3.1'

services:

 mesos-slave2:
 image: mesosphere/mesos-slave:1.7.0
 restart: always
 network_mode: host
 hostname: slave2
 command: --no-systemd_enable_support --no-hostname_lookup --ip=192.168.133.130
 ports:
 - 5051:5051
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock
 - /cgroup:/cgroup
 - /sys:/sys
 - /usr/bin/docker:/usr/local/bin/docker
 environment:
 MESOS_PORT: 5051
 MESOS_MASTER: zk://192.168.133.129:2181/mesos
 MESOS_SWITCH_USER: 0
 MESOS_CONTAINERIZERS: docker,mesos
 MESOS_LOG_DIR: /var/log/mesos
 MESOS_WORK_DIR: /var/tmp/mesos
 privileged: true

Marathon

marathon在高版本会出现收不到mesos心跳自动停止的问题，暂时没找到可以配置的方式，故使用1.6的版本，问题链接如下

Missing heartbeats from Mesos, eventually disconnecting and dying

docker-compose-marathon.yml

version: '3.1'

services:

 marathon:
 image: mesosphere/marathon:v1.6.496
 hostname: marathon
 restart: always
 command: --master zk://192.168.133.129:2181/mesos --zk zk://192.168.133.129:2181/marathon
 ports:
 - 8080:8080

安装marathon-lb

marathon-lb 与marathon没有版本的对应关系，故使用当前最新的版本1.14，新版和旧版的配置也有所不同，在早期版本无需指定端口的环境变量，翻阅官网也指出其应该是自动配置的，但如果使用Docker默认运行脚本则需要手动指定

if you use the built-in run script, you’ll need to pass an env var that contains a comma separated list of the ports HAProxy binds to

参考ISSUES：

unbound variable PORTS v1.1.0

docker-compose-marathon-lb.yml

version: '3.1'

services:

 marathon-lb:
 image: mesosphere/marathon-lb:v1.14.0
 network_mode: host
 hostname: marathon-lb
 restart: always
 command: sse --group external --marathon http://192.168.133.132:8080
 environment:
 PORTS: 9090
 ports:
 - 9090:9090

启动

启动顺序：

zookeeper
mesosMaster
mesosSlave
marathon
marathon-lb

由于全使用docker-compose脚本编写，启动时只需指定文件名即可

$ docker-compose -f xx.yml up -d

查看日志命令，-f表示跟踪最新的日志

$ docker-compose -f xx.yml logs [-f]

效果

随便打开一个节点的mesos端口，可看到当前的服务注册数量和当前的leader，其采用与ZK类似的选举算法来选出leader实现高可用

slave信息

查看框架注册情况

查看marathon-lb情况

marathon面板

marathon支持通过面板创建任务和通过rest接口创建任务

使用postman创建任务测试

请求路径http://192.168.133.132:8080/v2/apps

内容

{
 "id": "basic-0", 
 "cmd": "while [ true ] ; do echo 'Hello Marathon' ; sleep 5 ; done",
 "cpus": 0.1,
 "mem": 10.0,
 "instances": 1
}

结果

面板同步显示

运行状态正常

查看详情可知其运行在130服务器上

通过面板创建

切换为json格式可以编辑更多内容

{
 "id": "demo-springboot",
 "cmd": null,
 "cpus": 0.3,
 "mem": 128,
 "disk": 0,
 "instances": 1,
 "container": {
 "docker": {
 "image": "fjy8018/demo-springboot:2.1.6.RELEASE",
 "forcePullImage": true
 },
 "type": "DOCKER",
 "portMappings": [
 {
 "containerPort": 8080,
 "protocol": "tcp",
 "servicePort": 18080
 }
 ]
 },
 "networks": [
 {
 "mode": "container/bridge"
 }
 ]
}

运行成功

访问成功

查看资源调度情况

配置扩缩容

配置成功

两个实例被分别调度到不同的机器上

配置健康检查

自动轮询接口

错误告警

DevOps实践心得一

Fri, 16 Aug 2019 19:32:00 +0800

初遇DevOps

最早接触到DevOps是在2018年8月26日，那时首先接触到的就是大名鼎鼎的Jenkins，当时看了多篇关于持续集成和持续部署的文章，蠢蠢欲动，便开始了第一套DevOps环境的搭建

Jenkins

第一套DevOps是用Jenkins+Ansible+GitLab搭建的，那时只是了解Docker的好处，并没有使用Docker搭建整套环境，还是基于传统虚拟服务器搭建的整套流程。

Jenkins不仅可以编写groove脚本、pipeline 流水线等，还能与maven集成自动构建，深深感受到了Jenkins的强大，同时大量的用户为其贡献了许多插件，有很多实践问题都可以在网上搜索到解决方案

搭建过程：

最后顺利实现了静态网站持续集成、持续部署，详见：相关文章

Docker+GitLab

随着容器化技术在国内越来越流行，容器普及度越来越高，其带来的环境隔离极大推动了DevOps的落地，使得无需使用python环境隔离，可以直接利用容器技术实现跨平台和环境隔离，并且利于迁移和部署

于是在2019年4月开始研究GitLab官方文档，发现其早已原生支持流水线，并依托其GitLab-runner可以直接实现整套DevOps环境

因此，经过不断摸索，完整的搭建的基于Docker和GitLab的DevOps环境，搭建过程见：

由于国内当时对GitLab流水线使用的并不多，因此遇到的很多坑都需要翻查官方文档和国外的实践记录，比如：

DevOps 带来的好处

2019年7月将手上的一个小型项目尝试使用完整的DevOps流程开发，开发小组选择敏捷开发方法，所有的开发工具都尽量的适应快速的需求变化，同时投入非常少量的开发人员，分工明确，从一开始一周发布一次到最后一天可以发布多次

开发人员已经完全从运维中脱离，全团队没有专门的运维人员，极大的提高了开发效率，精简了人员构成，一旦构建失败，开发人员可以根据邮件通知的内容排查上线失败问题，调整代码和配置后重新提交

面板提供了流水线统计图，如果出现频繁的失败还是需要人工介入，查看问题原因

当前的问题

尽管使用了基于Docker+GitLab的流水线实现了完整的DevOps流程，但对于监控和日志部分仍然处在很多问题

尽管上线的是单体应用，服务器和应用的健康状态都无法得知，也没有相应的告警
应用的日志没有被外部服务收集，无法进行快速分析
没有实现完整的测试流程，生成测试覆盖率报告等
没有实现0停机上线，在容器切换时仍需停机1-2分钟
没有实现灰度上线、蓝绿部署等，上线失败需要手动回滚

未来探索方向

全面容器化服务治理方向
- 国内对容器化有非常多的实践
- kubernetes作为容器调度成为大多数服务治理的选择
- 容器隔离、迁移和伸缩带来的巨大优势
容器与传统应用部署结合方式
- 以半全自动方式落地DevOps，采用Mesos+marathon方式同时兼容传统服务部署和容器化部署
- 兼容分布式服务治理
- 国外有大量的实践案例
- 日志收集体系完备

使用GitHub和DockerHub自动构建镜像

Wed, 31 Jul 2019 22:36:41 +0800

环境

DockerHub账号、GitHub账号

步骤

构建脚本

此处以构建mycat的docker镜像为例，在本地准备好可以运行的构建脚本Dockerfile，尽可能不使用准备好的配置文件，保证镜像的安全、纯净，并且利用环境变量配置精简构建脚本

FROM openjdk:8-jre
MAINTAINER fjy8018 fjy8018@gmail.com

# Mycat 版本
ENV MYCAT_VERSION 1.6-RELEASE
ENV MYCAT_PKG Mycat-server-1.6-RELEASE-20161028204710-linux.tar.gz

WORKDIR /usr/local

# 缓存层
RUN wget http://dl.mycat.io/$MYCAT_VERSION/$MYCAT_PKG
RUN apt-get update && \
 apt-get install -y procps && \
 apt-get clean && \
 tar -zxf /usr/local/$MYCAT_PKG && \
 rm -rf /usr/local/$MYCAT_PKG

VOLUME /usr/local/mycat/conf

EXPOSE 8066 9066

ENTRYPOINT /usr/local/mycat/bin/mycat console

创建GitHub仓库

在GitHub上创建对应的仓库，并推送Dockerfile

链接DockerHub与GitHub

登陆DockerHub选择repository新建仓库

首次链接可以参照指引操作，主要是做OAuth 授权

创建构建分支和标签

连接GitHub后可以选择需要构建的仓库，同时展开构建选项，可以指定不同分支构建出的标签，默认master分支构建为latest版本

触发构建

第一次会自动触发构建，随后代码推送都会触发构建

构建结果

在管理页面可以查看构建时长和构建结果

结果

用户视角

用户视角可以看到构建的GitHub仓库

自动构建产生的分支

构建历史记录

Docker 19.03 版本对原Gitlab-Runner的影响

Sun, 28 Jul 2019 13:09:00 +0800

版本变化始末

在 2019.07.24 Docker社区发布了19.03的新版本，在此前版本号为18.0x

新版本带来的坑

原本没太在意docker版本的变化，因为多数情况都是兼容旧版的所有特性和功能的

结果在近期的CICD构建中，在ci脚本没有变化的情况下突然所有runner的构建都失败了

随后立即采取以下措施尝试修复

工程版本回退到上一次可用状态
更换runner服务器和容器
更改ci脚本
重启docker服务

均无效

执行构建步骤出现了以往没有出现过的警告信息

Running with gitlab-runner 11.8.0 (4745a6f3)
 on docker-in-docker-runner 4iAG2sYM
Using Docker executor with image docker:stable ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:fd0c64832f7e46b63a180e6000dbba7ad7a63542c5764841cba73429ba74a39e for docker:dind ...
Waiting for services to be up and running...

*** WARNING: Service runner-4iAG2sYM-project-89-concurrent-0-docker-0 probably didn't start properly.

Health check error:
service "runner-4iAG2sYM-project-89-concurrent-0-docker-0-wait-for-service" timeout

Health check container logs:

...

2019-07-24T03:11:35.365911823Z time="2019-07-24T03:11:35.365537741Z" level=info msg="API listen on [::]:2376"
2019-07-24T03:11:35.366107674Z time="2019-07-24T03:11:35.365626706Z" level=info msg="API listen on /var/run/docker.sock"

*********

Pulling docker image docker:stable ...
Using docker image sha256:c4154a2b47a18fe9437956ab981bd5924b19e7ae3eb3ed60c42cf8dfa394d550 for docker:stable ...

最终报错信息

Downloading artifacts for maven-build (280)...
Downloading artifacts from coordinator... ok id=280 responseStatus=200 OK token=L9iuC5Ss
$ docker build -t $CONTAINER_IMAGE:$IMAGE_VERSION .
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?
ERROR: Job failed: exit code 1

寻找解决方法

该错误在早期版本就出现过，但与本次情况均不符合，以往通过挂载容器路径即可，而本次情况是runner服务和构建脚本均不变的情况下突然出现构建失败的情况

最终在社区ISSUS讨论中发现了相关讨论，并且问题发生时间相近

Jobs failing with “Cannot connect to the Docker daemon”

官方开发者解释当天几个小时前发布的新版本 19.03默认开启了TLS确实会影响已有的Docker in Docker 模式的构建环境

同时官方开发人员建议自行部署的runner建议开启TLS功能

该开发者在另一个讨论中提供了19.03环境下的runner启动和配置样板，同时还有样例ci脚本

Update DinD example for 19.03

解决方法

最好可以卸载旧版本的Docker，尽管运行runner和dind环境都基于docker容器中
加入TLS相关runner注册配置

此处采用更新Docker版本并使用最新版的runner镜像进行注册

更新docker版本

官方文档

卸载旧docker

$ sudo yum remove docker \
 docker-client \
 docker-client-latest \
 docker-common \
 docker-latest \
 docker-latest-logrotate \
 docker-logrotate \
 docker-engine

配置生产环境的依赖仓库

$ sudo yum-config-manager \
 --add-repo \
 https://download.docker.com/linux/centos/docker-ce.repo

官方建议的安装新版Docker命令

$ sudo yum install docker-ce docker-ce-cli containerd.io

但经过这次事件后个人不建议使用该命令，而是应该手动指定所有Docker环境版本号，防止以后版本更新导致不兼容问题

查看当前可以安装的所有版本

$ sudo yum list docker-ce --showduplicates | sort -r
 * updates: repos.lax.quadranet.com
Loaded plugins: fastestmirror
 * extras: mirror.hostduplex.com
 * epel: d2lzkl7pfhq30w.cloudfront.net
 * elrepo-kernel: repos.lax-noc.com
docker-ce.x86_64 3:19.03.1-3.el7 docker-ce-stable
docker-ce.x86_64 3:19.03.0-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.8-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.7-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.6-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.5-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.4-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.3-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.2-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.1-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.0-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.3.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.2.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.1.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.0.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.12.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.12.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.3.ce-1.el7 docker-ce-stable
docker-ce.x86_64 17.03.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.0.ce-1.el7.centos docker-ce-stable
Determining fastest mirrors
 * base: mirror.hostduplex.com
Available Packages

指定最新版本

$ sudo yum install docker-ce-19.03.1 docker-ce-cli-19.03.1 containerd.io

启动docker

$ sudo systemctl start docker

设置开机自启

$ sudo systemctl enable docker

安装Gitlab Runner

官方对新版启用TLS的注册样例

sudo gitlab-runner register -n \
 --url https://gitlab.com/ \
 --registration-token REGISTRATION_TOKEN \
 --executor docker \
 --description "My Docker Runner" \
 --docker-image "docker:stable" \
 --docker-privileged \
 --docker-volumes "/certs/client"

此处采用容器化注册runner命令为

$ docker run --rm -t -i \
-v /etc/gitlab-runner/config:/etc/gitlab-runner \
gitlab/gitlab-runner:alpine-v12.1.0 register \
--url "https://gitlab.com/" \
--registration-token REGISTRATION_TOKEN \
--executor docker \
--non-interactive \
--run-untagged \
--tag-list "Runner,Docker" \
--run-untagged="true" \
--locked="false" \
--description "My Docker Runner" \
--docker-image "docker:stable" \
--docker-privileged \
--docker-volumes "/certs/client"

主要的变化至于增加了证书挂载--docker-volumes "/certs/client"配置，同时指定了runner的版本号

启动runner

$ docker run --name gitlab-runner --restart always -d \
-v /etc/gitlab-runner/config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:alpine-v12.1.0

修改构建脚本

官方对新版启用TLS的构建脚本样例，指定了构建时使用的docker版本与外部一直，防止不兼容的问题

同时加入了TLS环境变量配置

 image: docker:19.03

 variables:
 # When using dind service we need to instruct docker, to talk with
 # the daemon started inside of the service. The daemon is
 # available with a network connection instead of the default
 # /var/run/docker.sock socket. docker:19.03-dind does this
 # automatically by setting the DOCKER_HOST in
 # https://github.com/docker-library/docker/blob/d45051476babc297257df490d22cbd806f1b11e4/19.03/docker-entrypoint.sh#L23-L29
 #
 # The 'docker' hostname is the alias of the service container as described at
 # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services.
 #
 # Note that if you're using the Kubernetes executor, the variable should be set to
 # tcp://localhost:2376/ because of how the Kubernetes executor connects services
 # to the job container
 # DOCKER_HOST: tcp://localhost:2376/
 #
 # When using dind, it's wise to use the overlayfs driver for
 # improved performance.
 DOCKER_DRIVER: overlay2
 # Specify to Docker where to create the certificates, Docker will
 # create them automatically on boot, and will create
 # `/certs/client` that will be shared between the service and
 # build container.
 DOCKER_TLS_CERTDIR: "/certs"

 services:
 - docker:19.03-dind

 before_script:
 - echo $DOCKER_HOST
 - docker info

 build:
 stage: build
 script:
 - docker build -t my-docker-image .

在此前自己的构建脚本是按照官网提供的样板，没有指定版本，因此会导致每次构建都会使用当前最新的版本作为基础，最终导致外部Docker和内部Docker容器版本不一致

官网示例DinD构建脚本

image: docker:stable
services:
 - docker:dind
...

按照社区配置修改后重新构建则成功

总结

对于Docker等云服务平台仍然需要特别注意版本变化，为了防止不兼容的情况，在生产环境尽量不要使用latest版本标签，而是应该手动指定所有参与构建的版本环境

DevOps GitLab CICD 实践3——CI文件编写

Wed, 03 Apr 2019 16:32:00 +0800

前置步骤：

Docker 版 GitLab CICD 实践1——GitLab 部署 [[Docker 版 GitLab CICD 实践]]
Docker 版 GitLab CICD 实践2——Runner 部署 [[Docker 版 GitLab CICD 实践——Runner 部署]]

官方文档

编写结构类似Jenkins pineline流水线

GitLab CI/CD Pipeline Configuration Reference

GitLab CI/CD Pipeline Configuration Reference
GitLab CI/CD pipelines are configured using a YAML file called .gitlab-ci.yml within each project.

The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:

What to execute using GitLab Runner.

What decisions to make when specific conditions are encountered. For example, when a process succeeds or fails.

This topic covers CI/CD pipeline configuration. For other CI/CD configuration information, see:

GitLab CI/CD Variables, for configuring the environment the pipelines run in.

GitLab Runner advanced configuration, for configuring GitLab Runner.

We have complete examples of configuring pipelines:

For a quick introduction to GitLab CI, follow our quick start guide.

For a collection of examples, see GitLab CI/CD Examples.

To see a large .gitlab-ci.yml file used in an enterprise, see the .gitlab-ci.yml file for gitlab-ce.

目标

maven项目自动打包
~~自动测试~~
自动镜像构建并上传

官方参数表列出了作业的可用参数：

关键词	描述
`script`	由Runner执行的Shell脚本。
`image`	使用泊坞窗图像。也可用：`image:name`和`image:entrypoint`。
`services`	使用docker services图像。也可用：`services:name`，`services:alias`，`services:entrypoint`，和`services:command`。
`before_script`	覆盖在作业之前执行的一组命令。
`after_script`	覆盖作业后执行的一组命令。
`stages`	定义管道中的阶段。
`stage`	定义作业阶段（默认值:) `test`。
`only`	创建作业时限制。也可用：`only:refs`，`only:kubernetes`，`only:variables`，和`only:changes`。
`except`	在未创建作业时限制。也可用：`except:refs`，`except:kubernetes`，`except:variables`，和`except:changes`。
`tags`	用于选择Runner的标签列表。
`allow_failure`	让工作失败。失败的作业无助于提交状态。
`when`	什么时候开始工作。也可用：`when:manual`和`when:delayed`。
`environment`	作业部署到的环境的名称。也可用：`environment:name`，`environment:url`，`environment:on_stop`，和`environment:action`。
`cache`	后续运行之间应缓存的文件列表。也可用：`cache:paths`，`cache:key`，`cache:untracked`，和`cache:policy`。
`artifacts`	成功附加到作业的文件和目录列表。也可用：`artifacts:paths`，`artifacts:name`，`artifacts:untracked`，`artifacts:when`，`artifacts:expire_in`，`artifacts:reports`，和`artifacts:reports:junit`。在GitLab 企业版，这些都是可供选择：`artifacts:reports:codequality`，`artifacts:reports:sast`，`artifacts:reports:dependency_scanning`，`artifacts:reports:container_scanning`，`artifacts:reports:dast`，`artifacts:reports:license_management`，和`artifacts:reports:performance`。
`dependencies`	作业所依赖的其他作业，以便您可以在它们之间传递工件。
`coverage`	给定作业的代码覆盖率设置。
`retry`	在发生故障的情况下，可以自动重试作业的次数和次数。
`parallel`	应该并行运行多少个作业实例。
`trigger`	定义下游管道触发器。
`include`	允许此作业包含外部YAML文件。也可用：`include:local`，`include:file`，`include:template`，和`include:remote`。
`extends`	此作业将继承的配置条目。
`pages`	上传作业结果以用于GitLab Pages。
`variables`	在作业级别定义作业变量。

注：参数types和type被弃用。

参考

官方构建案例

Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD

参考其脚本

Configure GitLab CI/CD to deploy your application
Now we need to add the GitLab CI/CD configuration file (.gitlab-ci.yml) to our project’s root. This is how GitLab figures out what commands need to be run whenever code is pushed to our repository. We will add the following .gitlab-ci.yml file to the root directory of the repository, GitLab will detect it automatically and run the steps defined once we push our code:
image: java:8

stages:
 - build
 - deploy

build:
 stage: build
 script: ./mvnw package
 artifacts:
 paths:
 - target/demo-0.0.1-SNAPSHOT.jar

production:
 stage: deploy
 script:
 - curl --location "https://cli.run.pivotal.io/stable?release=linux64-binary&source=github" | tar zx
 - ./cf login -u $CF_USERNAME -p $CF_PASSWORD -a api.run.pivotal.io
 - ./cf push
 only:
 - master

配置

全局变量

进入工程CI设置配置全局变量脚本，包含镜像仓库登陆名称、密码、打包名称等

配置全局变量目的在于配置脚本中不应该包含密码等敏感信息

若希望使用GitLab内置环境变量，可参考官方表格

GitLab CI/CD environment variables

CI脚本

结合可用参数和样例配置，根据已经存在的SpringBoot项目编写响应的CI脚本.gitlab-ci.yml

image: docker:stable
services:
 - docker:dind

variables:
 DOCKER_DRIVER: overlay
 SPRING_PROFILES_ACTIVE: gitlab-ci

stages:
 - build
 - package

maven-build:
 image: maven:3-jdk-8
 stage: build
 script: "mvn package -B"
 artifacts:
 paths:
 - target/*.jar

docker-build:
 stage: package
 script:
 - docker build -t $CONTAINER_IMAGE:latest .
 - docker login -u $DOCKER_HUB_USER -p $DOCKER_HUB_PASS
 - docker push $CONTAINER_IMAGE:latest

该脚本对于满足脚本目标的工程都可复用，若需要测试，则增加相关的测试步骤即可

此处由于工程包含外部依赖关系，不在构建时测试

自动构建

默认的触发构建事件为commit等

触发后会自动执行任务

Maven自动触发构建，自动镜像打包

自动上传（推送）镜像

执行结果

若执行失败则会发送邮件给开发人员

CICD全套流程目标实现！

总结

不管使用GitLab CICD或者是使用Jenkins+Ansible的方式，目标都在于践行DevOps，打通开发与运维流程，一旦配置好CICD流程，往后开发人员便不再操心打包、构建等操作，可以专注开发

DevOps GitLab CICD 实践2——Runner 部署

Wed, 03 Apr 2019 16:31:00 +0800

前置步骤：DevOps GitLab CICD 实践1——GitLab 部署

官方文档

文档地址

虽然有官方指引，但个人感觉指引的不够清晰，导致初次配置可能频繁失败

Run GitLab Runner in a container
This is how you can run GitLab Runner inside a Docker container.

General GitLab Runner Docker image usage
GitLab Runner Docker images (based on Ubuntu or Alpine Linux) are designed as wrappers around the standard gitlab-runner command, like if GitLab Runner was installed directly on the host.

The general rule is that every GitLab Runner command that normally would be executed as:
gitlab-runner [Runner command and options...]
can be executed with:
docker run [chosen docker options...] gitlab/gitlab-runner [Runner command and options...]
For example, getting the top-level help information for GitLab Runner command could be executed as:
docker run --rm -t -i gitlab/gitlab-runner --help

NAME:
 gitlab-runner - a GitLab Runner

USAGE:
 gitlab-runner [global options] command [command options] [arguments...]

VERSION:
 10.7.0 (7c273476)

(...)
In short, the gitlab-runner part of the command is replaced with docker run [docker options] gitlab/gitlab-runner, while the rest of Runner’s command stays as it is described in the register documentation. The only difference is that the gitlab-runner command is executed inside of a Docker container.

Docker image installation and configuration
Install Docker first:
 curl -sSL https://get.docker.com/ | sh
You need to mount a config volume into the gitlab-runner container to be used for configs and other resources:
 docker run -d --name gitlab-runner --restart always \
 -v /srv/gitlab-runner/config:/etc/gitlab-runner \
 -v /var/run/docker.sock:/var/run/docker.sock \
 gitlab/gitlab-runner:latest
Tip: On macOS, use /Users/Shared instead of /srv.

Or, you can use a config container to mount your custom data volume:
 docker run -d --name gitlab-runner-config \
 -v /etc/gitlab-runner \
 busybox:latest \
 /bin/true
And then, run the Runner:
 docker run -d --name gitlab-runner --restart always \
 -v /var/run/docker.sock:/var/run/docker.sock \
 --volumes-from gitlab-runner-config \
 gitlab/gitlab-runner:latest
Register the runner you just launched by following the instructions in the Docker section of Registering Runners. The runner won’t pick up any jobs until it’s registered.

安装步骤

获取Gitlab Runner秘钥

可以通过Gitlab管理员账号获取，也可以让每一个用户自行配置

普通用户查看秘钥

进入任意一个仓库的设置中，查看CICD配置

准备注册专用Runner令牌

管理员查看令牌

进入总设置页面，配置全局Runner令牌

Runner注册

由于Runner一般运行复杂构建、打包任务，推荐配置在性能、带宽更大的机房

准备Docker环境

$ curl -sSL https://get.docker.com/ | sh
$ systemctl start docker

注册

可以根据需要选择注册Runner类型

同时，为了方便配置，使用单行注册并且关闭交互

单行注册官方文档

命令说明：

-v挂载的目的是为了将注册后的配置文件持久化，用于运行容器
--rm指定容器运行结束后自动删除停止的容器
-it指定使用命令行交互方式运行，便于查看注册结果

$ docker run --rm -it \
 -v /www/wwwroot/gitlab/srv/gitlab-runner/config:/etc/gitlab-runner \
 gitlab/gitlab-runner:alpine-v11.8.0 register \
 --non-interactive \
 --executor "docker" \
 --docker-image docker:stable \
 --url "Gitlab URL" \
 --registration-token "令牌" \
 --description "描述" \
 --tag-list "标签1,标签2" \
 --run-untagged \
 --docker-privileged \
 --locked="false"

注册成功提示

此时管理面板显示新的Runner已经注册

Runner 启动

挂载本地配置信息并启动Runner

$ docker run -d --name gitlab-runner --restart always \
 	-v /www/wwwroot/gitlab/srv/gitlab-runner/config:/etc/gitlab-runner \
 -v /var/run/docker.sock:/var/run/docker.sock \
 gitlab/gitlab-runner:alpine-v11.8.0

此时可见Runner已经保持和Gitlab的联系

DevOps GitLab CICD 实践1——GitLab 部署

Wed, 03 Apr 2019 16:30:00 +0800

配置目标

邮件提示
GitHub第三方授权登陆
GitLab Runner
Docker私服注册

官方介绍

目前微服务盛行环境下，服务部署优先考虑Docker方式，便于迁移和弹性伸缩

官方镜像介绍 GitLab Docker images

GitLab Docker images
Both GitLab CE and EE are in Docker Hub:

GitLab CE Docker image

GitLab EE Docker image

The GitLab Docker images are monolithic images of GitLab running all the necessary services on a single container.

In the following examples we are using the image of GitLab CE. To use GitLab EE instead of GitLab CE, replace the image name to gitlab/gitlab-ee:latest.

If you want to use the latest RC image, use gitlab/gitlab-ce:rc or gitlab/gitlab-ee:rc for GitLab CE and GitLab EE respectively.

The GitLab Docker images can be run in multiple ways:

Run the image in Docker Engine

Install GitLab into a cluster

Install GitLab using docker-compose

docker-compose 脚本

此处选择社区版(CE)安装，同时为了便于参数配置，使用docker-compose方式编写脚本文件

Install GitLab using docker-compose
With Docker compose you can easily configure, install, and upgrade your Docker-based GitLab installation.
Install Docker Compose
Create a docker-compose.yml file (or download an example):
 web:
 image: 'gitlab/gitlab-ce:latest'
 restart: always
 hostname: 'gitlab.example.com'
 environment:
 GITLAB_OMNIBUS_CONFIG: |
 external_url 'https://gitlab.example.com'
 # Add any other gitlab.rb configuration here, each on its own line
 ports:
 - '80:80'
 - '443:443'
 - '22:22'
 volumes:
 - '/srv/gitlab/config:/etc/gitlab'
 - '/srv/gitlab/logs:/var/log/gitlab'
 - '/srv/gitlab/data:/var/opt/gitlab'
Make sure you are in the same directory as docker-compose.yml and run docker-compose up -d to start GitLab
Read “Pre-configure Docker container” to see how the GITLAB_OMNIBUS_CONFIG variable works.

Below is another docker-compose.yml example with GitLab running on a custom HTTP and SSH port. Notice how the GITLAB_OMNIBUS_CONFIG variables match the ports section:
web:
 image: 'gitlab/gitlab-ce:latest'
 restart: always
 hostname: 'gitlab.example.com'
 environment:
 GITLAB_OMNIBUS_CONFIG: |
 external_url 'http://gitlab.example.com:9090'
 gitlab_rails['gitlab_shell_ssh_port'] = 2224
 ports:
 - '9090:9090'
 - '2224:22'
 volumes:
 - '/srv/gitlab/config:/etc/gitlab'
 - '/srv/gitlab/logs:/var/log/gitlab'
 - '/srv/gitlab/data:/var/opt/gitlab'
This is the same as using --publish 9090:9090 --publish 2224:22.

官方提示说明Docker CE版基于Omnibus版本，故环境配置也可参考相关文档

Omnibus文档目录

Installation and Configuration using omnibus package
Note: This section describes the commonly used configuration settings. Check configuration section of the documentation for complete configuration settings.

Installing GitLab

Manually downloading and installing a GitLab package

Setting up a domain name/URL for the GitLab Instance so that it can be accessed easily

Enabling HTTPS

Enabling notification EMails

Enabling replying via email

Installing and configuring postfix

Enabling container registry on GitLab

You will require SSL certificates for the domain used for container registry

Enabling GitLab Pages

If you want HTTPS enabled, you will have to get wildcard certificates

Enabling Elasticsearch

GitLab Mattermost Set up the Mattermost messaging app that ships with Omnibus GitLab package.

GitLab Prometheus Set up the Prometheus monitoring included in the Omnibus GitLab package.

GitLab High Availability Roles

结合配置目标编写yaml文件

注意：

此处邮件使用163邮箱（官方没有提供163邮箱支持案例）
Docker私服公钥执行从私服上获取
由于特殊原因，目标配置未启动SSL安全连接，但GitLab可以通过简单配置支持SSL并自动更新证书

配置文档

Let’s Encrypt Integration
Primary GitLab Instance

Note: Introduced in GitLab version 10.5 and disabled by default. Enabled by default in GitLab version 10.7 and later if external_url is set with the httpsprotocol and no certificates are configured.

Note: In order for Let’s Encrypt verification to work correctly, ports 80 and 443 will need to be accessible to the Let’s Encrypt servers that run the validation. Also note that the validation currently does not work with non-standard ports.
Caution Administrators installing or upgrading to GitLab version 10.7 or later and do not plan on using Let’s Encrypt should set the following in /etc/gitlab/gitlab.rb to disable:
letsencrypt['enable'] = false
Add the following entries to /etc/gitlab/gitlab.rb to enable Let’s Encrypt support for the primary domain:
letsencrypt['enable'] = true # GitLab 10.5 and 10.6 require this option
external_url "https://gitlab.example.com"	 # Must use https protocol
letsencrypt['contact_emails'] = ['foo@email.com'] # Optional

生成163邮箱授权密码

生成GitHub授权秘钥

最终配置

version: '3.1'

services:

 gitlab:
 environment:
 GITLAB_OMNIBUS_CONFIG: |
 external_url '外部访问地址'
 gitlab_rails['gitlab_shell_ssh_port'] = 外部SSH端口
 registry_external_url 'Docker私服地址'
 registry_nginx['ssl_certificate'] = "Docker 私服CA证书 crt文件"
 registry_nginx['ssl_certificate_key'] = "Docker 私服公钥 pem文件"
 gitlab_rails['smtp_enable'] = true
 gitlab_rails['smtp_address'] = "smtp.163.com"
 gitlab_rails['smtp_port'] = 465
 gitlab_rails['smtp_user_name'] = "邮件发送者名称"
 gitlab_rails['gitlab_email_from'] = '邮件发送地址'
 gitlab_rails['smtp_password'] = "授权密码"
 gitlab_rails['smtp_domain'] = "163.com"
 gitlab_rails['smtp_authentication'] = "login"
 gitlab_rails['smtp_enable_starttls_auto'] = true
 gitlab_rails['smtp_tls'] = true
 gitlab_rails['omniauth_enabled'] = true
 gitlab_rails['omniauth_allow_single_sign_on'] = true
 gitlab_rails['omniauth_block_auto_created_users'] = true
 gitlab_rails['omniauth_providers'] = [
 {
 "name" => "github",
 "app_id" => "Client ID",
 "app_secret" => "Client Secret",
 "url" => "https://github.com/",
 "args" => { "scope" => "user:email" }
 }
 ]
 image: gitlab/gitlab-ce:latest
 hostname: gitlab.fjy8018.top
 restart: always
 networks:
 - devops-service-bridge
 ports:
 - '8443:443'
 - '8099:8099'
 - '8022:22'
 volumes:
 - ./srv/gitlab/config:/etc/gitlab
 - ./srv/gitlab/logs:/var/log/gitlab
 - ./srv/gitlab/data:/var/opt/gitlab
 - /etc/docker/certs.d:/etc/docker/certs.d


networks:
 devops-service-bridge:
 driver: bridge

Tyk网关Docker安装

Sun, 31 Mar 2019 15:57:00 +0800

文档指南

官方文档

查看官方Docker安装指南，发现只提供了镜像地址，并未给出详细操作步骤

With Docker
Tyk has three containers that are available to set up a Docker installation:

The Tyk Gateway container

The Tyk Dashboard container

The Tyk Pump container

All three are required for a full deployment. We recommend that each container is installed on a separate machine for optimum performance.

实践

Tyk Gateway container

参考镜像指南，此处的目标是搭建带Dashboard的Tyk网关

Gateway - Pro installation with Dashboard
The gateway in a Pro installation is dependent on the dashboard service. We will assume that the dashboard service is installed, up and running. If not, we would recommend that you follow the dashboard installation guide here:

https://github.com/TykTechnologies/tyk-dashboard-docker

The gateway relies upon the dashboard service to load it’s api definitions & proxy configurations. As such, there is no need to mount any app directory.
docker run -d \
 --name tyk_gateway \
 --network tyk \
 -p 8080:8080 \
 -v $(pwd)/tyk.with_dashboard.conf:/opt/tyk-gateway/tyk.conf \
 tykio/tyk-gateway:latest

文档中说明搭建网关需要先搭建Dashboard

Tyk Dashboard container

安装指南，指南的第六步指向的连接已经失效，由此推断之前存在一键安装脚本

Quickstart

Ensure you have set up the Redis, MongoDB and Tyk Gateway containers

Set up the docker instance IP as the Dashboard hostname (in your /etc/hosts file or as a DNS):

127.0.0.1 dashboard.tyk.docker

Run the Dashboard

docker run -d --name tyk_dashboard -p 3000:3000 --link tyk_redis:redis --link tyk_mongo:mongo --link tyk_gateway:tyk_gateway tykio/tyk-dashboard

You should now be able to access your Dashboard at http://dashboard.tyk.docker:3000/ (note for OSX users, replace 127.0.0.1 with whatever IP address your docker VM runs)

Enter your Dashboard License. Go to http://dashboard.tyk.docker:3000/. You will see a screen asking for a license, enter it in the section marked “Already have a license?” and click Use this license.

Grab the bootstrap script from our tyk-dashboard github repo and run:

./bootstrap.sh dashboard.tyk.docker

To use an external configuration files, use the -v option to mount it over /opt/tyk-dashboard/tyk_analytics.conf

经过查找发现了旧版指南链接仍然可以访问，但不建议使用，旧版指南

按照文档的说法需要申请许可证，官方提供开发许可证，提供邮箱即可获取12个月的授权，申请地址

申请后根据文档提供的Docker启动语句准备环境和Docker-compose文件便于一键启动

docker-compose.yml文件

version: "3.1"

services:

 tyk_gateway:
 image: tykio/tyk-gateway:latest
 restart: always
 networks:
 - tyk-service-bridge
 volumes:
 - /www/wwwroot/tyk-free/apps:/opt/tyk-gateway/apps
 depends_on:
 - tyk_dashboard

 tyk_dashboard:
 image: tykio/tyk-dashboard
 restart: always
 networks:
 - tyk-service-bridge
 ports:
 - 3000:3000
 depends_on:
 - mongo
 - redis

 mongo:
 image: mongo
 restart: always
 networks:
 - tyk-service-bridge

 redis:
 image: redis:4.0-alpine
 restart: always
 networks:
 - tyk-service-bridge

networks:
 tyk-service-bridge:
 driver: bridge

启动

$ docker-compose up

放行防火墙

$ firewall-cmd --zone=public --add-port=3000/tcp --permanent 
$ firewall-cmd --reload

访问浏览器IP:3000输入秘钥

许可证已经发送到注册邮件中，激活后可登陆

但此时没有默认的用户名和密码，并且暂时未找到配置的方式（差评），需要通过官方脚本注册用户名和密码

但官方脚本依赖Python环境，若主机没有python环境则无法执行

启动脚本：

#!/bin/bash
# Usage ./bootstrap.sh DASHBOARD_HOSTNAME

LOCALIP=$1
RANDOM_USER=$(env LC_CTYPE=C tr -dc "a-z0-9" < /dev/urandom | head -c 10)
PASS="test123"

echo "Creating Organisation"
ORGDATA=$(curl --silent --header "admin-auth: 12345" --header "Content-Type:application/json" --data '{"owner_name": "Default Org.","owner_slug": "default", "cname_enabled": true, "cname": ""}' http://$LOCALIP:3000/admin/organisations 2>&1)
#echo $ORGDATA
ORGID=$(echo $ORGDATA | python -c 'import json,sys;obj=json.load(sys.stdin);print(obj["Meta"])')
echo "ORGID: $ORGID"

echo "Adding new user"
USER_DATA=$(curl --silent --header "admin-auth: 12345" --header "Content-Type:application/json" --data '{"first_name": "John","last_name": "Smith","email_address": "'$RANDOM_USER'@example.com","password":"'$PASS'", "active": true,"org_id": "'$ORGID'"}' http://$LOCALIP:3000/admin/users 2>&1)
#echo $USER_DATA
USER_CODE=$(echo $USER_DATA | python -c 'import json,sys;obj=json.load(sys.stdin);print(obj["Message"])')
echo "USER AUTH: $USER_CODE"

USER_LIST=$(curl --silent --header "authorization: $USER_CODE" http://$LOCALIP:3000/api/users 2>&1)
#echo $USER_LIST

USER_ID=$(echo $USER_LIST | python -c 'import json,sys;obj=json.load(sys.stdin);print(obj["users"][0]["id"])')
echo "NEW ID: $USER_ID"

echo "Setting password"
OK=$(curl --silent --header "authorization: $USER_CODE" --header "Content-Type:application/json" http://$LOCALIP:3000/api/users/$USER_ID/actions/reset --data '{"new_password":"'$PASS'"}')

echo ""

echo "DONE"
echo "===="
echo "Login at http://$LOCALIP:3000/"
echo "User: $RANDOM_USER@example.com"
echo "Pass: $PASS"
echo ""

为了方便使用，此处创建了包含启动脚本的Docker镜像

仓库地址：fjy8018/tyk-bootstrap

使用方法：

$ docker run --rm -it --network tyk-free_tyk-service-bridge -e TARGET_IP=tyk_dashboard fjy8018/tyk-bootstrap:0.1

随即获得登陆名和密码

登陆成功

Mesos集群安装

Wed, 03 Oct 2018 22:10:00 +0800

基础环境

4台服务器，一台外部客户机

安装指南

官方支持C++源码编译安装和Docker安装

C++源码编译安装地址

Docker镜像仓库

停止所有容器命令

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

安装

mesos主节点

node1

docker pull mesosphere/mesos-master:1.7.0

node2(host)

docker run --name mesos-zookeeper -p 2181:2181 --restart always -d zookeeper:3.5

node1执行脚本(需替换zk IP地址)

MESOS_QUORUM=1即1为单机运行，3个节点为2，计算方式为 n÷2+1 \

若不手动指定hostname映射的IP，则所有节点默认通过主机名进行通信

使用host通信脚本

docker run -d --net=host \
 -e MESOS_PORT=5050 \
 -e MESOS_ZK=zk://192.168.0.32:2181/mesos \
 -e MESOS_QUORUM=1 \
 -e MESOS_REGISTRY=in_memory \
 -e MESOS_LOG_DIR=/var/log/mesos \
 -e MESOS_WORK_DIR=/var/tmp/mesos \
 -v "$(pwd)/log/mesos:/var/log/mesos" \
 -v "$(pwd)/work/mesos:/var/tmp/mesos" \
 -p 5050:5050 \
 --name=mesos-master \
 mesosphere/mesos-master:1.7.0 \

使用IP通信脚本

docker run -d --net=host \
 --hostname=192.168.0.32 \
 -e MESOS_PORT=5050 \
 -e MESOS_ZK=zk://192.168.0.32:2181/mesos \
 -e MESOS_QUORUM=1 \
 -e MESOS_REGISTRY=in_memory \
 -e MESOS_LOG_DIR=/var/log/mesos \
 -e MESOS_WORK_DIR=/var/tmp/mesos \
 -v "$(pwd)/log/mesos:/var/log/mesos" \
 -v "$(pwd)/work/mesos:/var/tmp/mesos" \
 -p 5050:5050 \
 --name=mesos-master \
 mesosphere/mesos-master:1.7.0 \
 --no-hostname_lookup \
 --ip=192.168.0.33

查看日志

docker logs mesos-master

成功输出日志

I1003 08:41:29.868827 14 group.cpp:341] Group process (zookeeper-group(2)@192.168.0.33:5050) connected to ZooKeeper
I1003 08:41:29.868902 14 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1003 08:41:29.868921 14 group.cpp:419] Trying to create path '/mesos' in ZooKeeper
I1003 08:41:29.902194 14 contender.cpp:268] New candidate (id='1') has entered the contest for leadership
I1003 08:41:29.904409 8 detector.cpp:152] Detected a new leader: (id='1')
I1003 08:41:29.904513 8 group.cpp:700] Trying to get '/mesos/json.info_0000000001' in ZooKeeper
I1003 08:41:29.912968 8 zookeeper.cpp:262] A new leading master (UPID=master@192.168.0.33:5050) is detected
I1003 08:41:29.913091 8 master.cpp:2113] Elected as the leading master!
I1003 08:41:29.913115 8 master.cpp:1668] Recovering from registrar
I1003 08:41:29.916913 11 registrar.cpp:383] Successfully fetched the registry (0B) in 3.734016ms
I1003 08:41:29.917054 11 registrar.cpp:487] Applied 1 operations in 26us; attempting to update the registry
I1003 08:41:29.918582 11 registrar.cpp:544] Successfully updated the registry in 1.449984ms
I1003 08:41:29.918630 11 registrar.cpp:416] Successfully recovered registrar
I1003 08:41:29.918772 8 master.cpp:1782] Recovered 0 agents from the registry (121B); allowing 10mins for agents to reregister

访问IP:5050，对于PWD，可以访问http://SSHIP-5050.direct.labs.play-with-docker.com/#/，将SSH ip替换成实际提供的IP即可

mesos从节点

node3 (slave-1)、node4(slave-2)

启动脚本

使用host通信脚本

mkdir /cgroup
docker run -d --net=host --privileged \
 -e MESOS_PORT=5051 \
 -e MESOS_MASTER=zk://192.168.0.32:2181/mesos \
 -e MESOS_SWITCH_USER=0 \
 -e MESOS_CONTAINERIZERS=docker,mesos \
 -e MESOS_LOG_DIR=/var/log/mesos \
 -e MESOS_WORK_DIR=/var/tmp/mesos \
 -v "$(pwd)/log/mesos:/var/log/mesos" \
 -v "$(pwd)/tmp/mesos:/var/tmp/mesos" \
 -v /var/run/docker.sock:/var/run/docker.sock \
 -v /cgroup:/cgroup \
 -v /sys:/sys \
 -v "$(which docker):/usr/local/bin/docker" \
 --name=mesos-slave \
 -p 5051:5051 \
 mesosphere/mesos-slave:1.7.0 \
 --no-systemd_enable_support

使用IP通信脚本，--ip需要改成实际IP

mkdir /cgroup
docker run -d --net=host --privileged \
 -e MESOS_PORT=5051 \
 -e MESOS_MASTER=zk://192.168.0.32:2181/mesos \
 -e MESOS_SWITCH_USER=0 \
 -e MESOS_CONTAINERIZERS=docker,mesos \
 -e MESOS_LOG_DIR=/var/log/mesos \
 -e MESOS_WORK_DIR=/var/tmp/mesos \
 -v "$(pwd)/log/mesos:/var/log/mesos" \
 -v "$(pwd)/tmp/mesos:/var/tmp/mesos" \
 -v /var/run/docker.sock:/var/run/docker.sock \
 -v /cgroup:/cgroup \
 -v /sys:/sys \
 -v "$(which docker):/usr/local/bin/docker" \
 --name=mesos-slave \
 -p 5051:5051 \
 mesosphere/mesos-slave:1.7.0 \
 --no-systemd_enable_support \
 --no-hostname_lookup \
 --ip=192.168.0.31

查看日志

docker logs mesos-slave

启动成功日志

I1003 09:05:08.615039 12 group.cpp:341] Group process (zookeeper-group(1)@192.168.0.31:5051) connected to ZooKeeper
I1003 09:05:08.615104 12 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1003 09:05:08.615121 12 group.cpp:419] Trying to create path '/mesos' in ZooKeeper
I1003 09:05:08.618291 13 state.cpp:66] Recovering state from '/var/tmp/mesos/meta'
I1003 09:05:08.618405 15 slave.cpp:6930] Finished recovering checkpointed state from '/var/tmp/mesos/meta', beginning agent recovery
I1003 09:05:08.618475 8 task_status_update_manager.cpp:207] Recovering task status update manager
I1003 09:05:08.618628 8 docker.cpp:908] Recovering Docker containers
I1003 09:05:08.626359 13 detector.cpp:152] Detected a new leader: (id='3')
I1003 09:05:08.626487 15 group.cpp:700] Trying to get '/mesos/json.info_0000000003' in ZooKeeper
I1003 09:05:08.628664 15 zookeeper.cpp:262] A new leading master (UPID=master@192.168.0.33:5050) is detected
I1003 09:05:08.997036 15 docker.cpp:921] Got the list of Docker containers
I1003 09:05:08.997324 15 docker.cpp:1157] Finished processing orphaned Docker containers
I1003 09:05:08.997436 8 slave.cpp:7159] Recovering executors
I1003 09:05:08.997553 8 slave.cpp:7312] Finished recovery
I1003 09:05:08.998044 11 task_status_update_manager.cpp:181] Pausing sending task status updates
I1003 09:05:08.998045 8 slave.cpp:1275] New master detected at master@192.168.0.33:5050
I1003 09:05:08.998159 8 slave.cpp:1329] No credentials provided. Attempting to register without authentication
I1003 09:05:08.998181 8 slave.cpp:1340] Detecting new master
I1003 09:05:09.446756 11 slave.cpp:1500] Registered with master master@192.168.0.33:5050; given agent ID b597695d-6e37-418a-9879-4ffe46117e17-S0
I1003 09:05:09.449091 10 task_status_update_manager.cpp:188] Resuming sending task status updates

访问主节点web界面可看见资源邀约有效

节点情况

marathon安装

node5

启动脚本

docker run -d --net=host \
 -p 8080:8080 \
 mesosphere/marathon:latest \
 --master zk://192.168.0.32:2181/mesos \
 --zk zk://192.168.0.32:2181/marathon

通过IP:8080查看运行结果

marathon-lb安装

node6

启动脚本

docker run -d -e PORTS=9090 --net=host \
 -p 9090:9090 \
 mesosphere/marathon-lb sse \
 --group external \
 --marathon http://192.168.0.29:8080

访问IP:9090/haproxy?stats可查看启动状态

从零开始部署Rancher

Sat, 11 Aug 2018 11:41:00 +0800

基础环境部署

基础环境使用CentOS 7，版本号为1804，部署在本地虚拟机VM上

安装CentOS+docker

CentOS下载地址

使用最小安装，ISO镜像大小为906MB，在VM中导入镜像，给4G内存（Rancher2.0最低要求），安装过程系统配置均为默认配置

安装好后为纯净环境，使用ip addr即可查看ip，使用外部远程连接工具连接虚拟机ip

使用yum install docker安装docker环境

安装好后使用命令systemctl start docker启动docker 使用 docker info或者docker version查看版本

[root@localhost ~]# docker version
Client:
 Version: 1.13.1
 API version: 1.26
 Package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64
 Go version: go1.9.4
 Git commit: dded712/1.13.1
 Built: Tue Jul 17 18:34:48 2018
 OS/Arch: linux/amd64

Server:
 Version: 1.13.1
 API version: 1.26 (minimum version 1.12)
 Package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64
 Go version: go1.9.4
 Git commit: dded712/1.13.1
 Built: Tue Jul 17 18:34:48 2018
 OS/Arch: linux/amd64
 Experimental: false

安装Rancher

为了加速Rancher下载速度，需要配置docker镜像下载地址，此处提供一个阿里云的镜像地址

{
 "registry-mirrors": ["https://fy707np5.mirror.aliyuncs.com"]
}

使用命令编辑docker镜像地址 vi /etc/docker/daemon.json 保存后重载配置systemctl daemon-reload 重启docker服务systemctl restart docker 镜像地址配置完成

访问Rancher官网查看docker命令 Rancher英文官网

Rancher中文官网

访问quickstart查看docker命令

目前Rancher官方开始推荐2.0版本，首次使用推荐使用1.6稳定版

在下方找到1.6文档，使用命令在虚拟机中安装Rancher1.6

sudo docker run -d --restart=unless-stopped -p 8080:8080 rancher/server:stable

配置防火墙

安装后若没有打开8080端口则会报错，配置CentOS防火墙来源：[莫小安-CentOS7使用firewalld打开关闭防火墙与端口][10]

1、firewalld的基本使用启动： systemctl start firewalld 关闭： systemctl stop firewalld 查看状态： systemctl status firewalld 开机禁用： systemctl disable firewalld 开机启用： systemctl enable firewalld

2.systemctl是CentOS7的服务管理工具中主要的工具，它融合之前service和chkconfig的功能于一体。启动一个服务：systemctl start firewalld.service 关闭一个服务：systemctl stop firewalld.service 重启一个服务：systemctl restart firewalld.service 显示一个服务的状态：systemctl status firewalld.service 在开机时启用一个服务：systemctl enable firewalld.service 在开机时禁用一个服务：systemctl disable firewalld.service 查看服务是否开机启动：systemctl is-enabled firewalld.service 查看已启动的服务列表：systemctl list-unit-files|grep enabled 查看启动失败的服务列表：systemctl –failed

3.配置firewalld-cmd

查看版本： firewall-cmd –version 查看帮助： firewall-cmd –help 显示状态： firewall-cmd –state 查看所有打开的端口： firewall-cmd –zone=public –list-ports 更新防火墙规则： firewall-cmd –reload 查看区域信息: firewall-cmd –get-active-zones 查看指定接口所属区域： firewall-cmd –get-zone-of-interface=eth0 拒绝所有包：firewall-cmd –panic-on 取消拒绝状态： firewall-cmd –panic-off 查看是否拒绝： firewall-cmd –query-panic

那怎么开启一个端口呢添加 firewall-cmd –zone=public –add-port=80/tcp –permanent （–permanent永久生效，没有此参数重启后失效）重新载入 firewall-cmd –reload 查看 firewall-cmd –zone= public –query-port=80/tcp 删除 firewall-cmd –zone= public –remove-port=80/tcp –permanent

通过命令放行8080端口

firewall-cmd --zone=public --add-port=8080/tcp --permanent

重载防火墙

firewall-cmd --reload

查看端口放行情况

 firewall-cmd --zone=public --list-ports

设置Rancher

设置Rancher界面语言

通过虚拟机ip+端口号访问Rancher：http://192.168.79.142:8080

默认为英文界面，Rancher官方有中文支持修改右下方语言选项即可

设置Rancher主机

首次访问需要设置主机

按照默认识别的IP即可，点击保存

选择主机类型，包含自定义主机、亚马逊主机等，此处选择自定义主机

依照指示操作，为了确保通信测试，暂时关闭防火墙

为了测试多台Rancher Server和agent服务，使用VM保存快照并复制一份虚拟机启动目前测试：128为server，129为agent，注意此处填的是客户端的地址，非主机地址

设置Rancher客户端

复制Server的配置命令，在agent服务器上运行，等待下载

按照指示，运行后关闭Server主机配置查看主机列表

也可以添加环境、测试环境、生产环境等

Docker部署中遇到的坑

Thu, 05 Jul 2018 14:21:00 +0800

镜像无法删除

问题复现

当前的镜像

PS D:\JAVA WORKSPACE\dockerTemp> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
orderbook latest 6514a4df3367 7 seconds ago 300MB
hub.c.163.com/fjy8018/mysql-orderbooks latest 37a82bac5f71 About an hour ago 407MB
hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB
hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB

运行一个镜像

PS D:\JAVA WORKSPACE\dockerTemp> docker run -d -p 8888:8080 orderbook:latest
678dcf28c1c3206cd78a824aab51cc5d1f6c333ed85f48d4c364a51526049f20

查看进程，发现进程没有出现（3306是已经启动的MySQL容器），由于容器制作过程出现问题导致进程无法成功启动

PS D:\JAVA WORKSPACE\dockerTemp> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
43146ddfc771 hub.c.163.com/library/mysql:latest "docker-entrypoint.s…" 2 hours ago Up 2 hours 0.0.0.0:8006->3306/tcp hopeful_euclid

检查镜像

PS D:\JAVA WORKSPACE\dockerTemp> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
orderbook latest 6514a4df3367 44 seconds ago 300MB
hub.c.163.com/fjy8018/mysql-orderbooks latest 37a82bac5f71 About an hour ago 407MB
hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB
hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB

删除没有成功运行的镜像报错，报该容器已经被占用

PS D:\JAVA WORKSPACE\dockerTemp> docker rmi 6514a4df3367
Error response from daemon: conflict: unable to delete 6514a4df3367 (must be forced) - image is being used by stopped container 678dcf28c1c3

删除报错中提示的容器ID

PS D:\JAVA WORKSPACE\dockerTemp> docker rmi 678dcf28c1c3
Error: No such image: 678dcf28c1c3

报无该镜像

解决

官方对rm与rmi的解释： rm Remove one or more containers rmi Remove one or more images 可知rm删除的是容器，rmi删除的是镜像，两者的区别如下：

容器（container）的定义和镜像（image）几乎一模一样，也是一堆层的统一视角，唯一区别在于容器的最上面那一层是可读可写的。镜像（Image）就是一堆只读层（read-only layer）的统一视角要点：容器 = 镜像 + 可读层。并且容器的定义并没有提及是否要运行容器。

故对于不在镜像列表的ID，要删除的是容器，使用rm命令 PS D:\JAVA WORKSPACE\dockerTemp> docker rm 678dcf28c1c3 678dcf28c1c3

列出所有的镜像

PS D:\JAVA WORKSPACE\dockerTemp> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
orderbook latest 6514a4df3367 2 minutes ago 300MB
hub.c.163.com/fjy8018/mysql-orderbooks latest 37a82bac5f71 About an hour ago 407MB
hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB
hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB

再删除目标镜像 PS D:\JAVA WORKSPACE\dockerTemp> docker rmi 6514a4df3367 Untagged: orderbook:latest Deleted: sha256:6514a4df33678bd98e32b5738150c88c5d72b9a3a4d86d1c370406f0f7f18788 Deleted: sha256:68a70a0ac20590c8412286b3c34ab2bc5c5b7c8845088536f03702185ac34105 Deleted: sha256:bf15e37a31e4266bc8a962a2b11fc3223c07de328ff85b17c70a15653c71061b Deleted: sha256:b447a606517609703f1efe4c560c42b9aeeaee669d22f12a65d2d2929d8ecc20

删除成功
PS D:\JAVA WORKSPACE\dockerTemp> docker images REPOSITORY TAG IMAGE ID CREATED SIZE hub.c.163.com/fjy8018/mysql-orderbooks latest 37a82bac5f71 About an hour ago 407MB hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB

停止并删除所有容器

批处理命令

docker stop $(docker ps -aq)
docker rm $(docker ps -aq)

自定义docker容器

Wed, 04 Jul 2018 17:06:00 +0800

准备docker环境

去官网下载docker环境，此处使用win10的docker环境，若早期版本的windows要安装额外的工具

检查docker版本

下载基础环境

本次决定搭建的docker为订书系统，其基于jsp编写，运行在tomcat上，数据库依赖MySQL，故需要一个MySQL和tomcat环境（tomcat环境自带Java）去网易镜像中心下载镜像，网易镜像中心与官方镜像中心保持同步更新，而且对于国内来说速度更快

部署MySQL镜像

下载

复制下载地址

mysql镜像下方也有使用说明

以及环境变量的描述，是否需要在创建时建立数据库等

启动

使用命令启动mysql的docker，指定本机8006端口映射到docker的3306端口，同时设置数据库root用户密码为root -d指定为后台运行

docker run -d -p 8006:3306 -e MYSQL_ROOT_PASSWORD=root hub.c.163.com/library/mysql:latest

使用命令查看当前运行的docker

docker ps

测试

使用Navicat连接docker的数据库

建立相关数据库，不使用环境变量创建是为了设置字符集

启用命令行界面创建相关用户并授权

导入数据库文件

部署Tomcat镜像

查看本机的ip

下载

从镜像中心下载tomcat

查看镜像中配置的tomcat目录

查看当前已经下载的镜像

PS D:\JAVA WORKSPACE\dockerTemp> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB
hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB

编写Dockerfile文件将工程移到tomcat的webapps目录下

from hub.c.163.com/library/tomcat

MAINTAINER fjy8018 fjy8018@gmail.com

COPY /OrderBooks /usr/local/tomcat/webapps/OrderBooks

修改工程中的数据库地址，地址为ip+暴露的docker端口

使用docker build -t orderbook .命令将docker打包成镜像

启动

使用命令启动带工程的tomcat，指定访问端口为8888

docker run -d -p 8888:8080 orderbook:latest

配置

由于使用了jsp+jdbc作为数据库连接，故需要进入容器内部将jdbc的jar复制到tomcat的lib目录下，否则tomcat会报无法加载驱动，使用命令进入容器内部

docker exec -it {容器ID} /bin/sh

使用exec命令比attach优势在于，exec进入容器后再使用exit命令不会令容器停止进入容器将工程的lib复制到tomcat中的lib

PS D:\JAVA WORKSPACE\dockerTemp> docker exec -it 6fe288a25be0 /bin/sh
# cp /usr/local/tomcat/webapps/OrderBooks/WebRoot/WEB-INF/lib/mysql-connector-java-5.1.45-bin.jar /usr/local/tomcat/lib
# exit
PS D:\JAVA WORKSPACE\dockerTemp> docker restart 6fe288a25be0
6fe288a25be0

测试

外部浏览器访问工程

工程正常访问

提交到网易镜像仓库

官方推送指南

提交修改的镜像

docker commit提交修改的镜像

PS D:\JAVA WORKSPACE\dockerTemp> docker commit 6fe288a25be0 orderbooks
sha256:30c6a878adc80f54118f5c67c39c271baa5dc68006aa26b64ef5281832fb4b37
PS D:\JAVA WORKSPACE\dockerTemp> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
orderbooks latest 30c6a878adc8 5 seconds ago 301MB
orderbook latest a57e26354272 10 minutes ago 300MB
hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB
hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB

提交mysql容器的修改 PS D:\JAVA WORKSPACE\dockerTemp> docker commit 43146ddfc771 hub.c.163.com/fjy8018/mysql-orderbooks sha256:37a82bac5f71d54e3ff31b0ddd795604e46d4f2e4dde8a6a73d556cc5f26411a PS D:\JAVA WORKSPACE\dockerTemp> docker images REPOSITORY TAG IMAGE ID CREATED SIZE hub.c.163.com/fjy8018/mysql-orderbooks latest 37a82bac5f71 3 seconds ago 407MB orderbooks latest 30c6a878adc8 13 minutes ago 301MB hub.c.163.com/fjy8018/tomcat-orderbooks latest 30c6a878adc8 13 minutes ago 301MB orderbook latest a57e26354272 23 minutes ago 300MB hub.c.163.com/library/tomcat latest 72d2be374029 10 months ago 292MB hub.c.163.com/library/mysql latest 9e64176cd8a2 14 months ago 407MB

推送镜像

将镜像推送到网易仓库

登录到网易镜像仓库查看，默认推送会设置为private，推送完成

Docker on F嘉阳

Kubesphere落地DevOps

创建企业空间

安装helm

找不到环境变量报错

安装SonarQube

配置 SonarQube 服务器

创建 SonarQube 管理员令牌 (Token)

创建 Webhook 服务器

将 SonarQube 配置添加到 ks-installer

将 SonarQube 服务器添加至 Jenkins

将 sonarqubeURL 添加到 KubeSphere 控制台

重启服务

为新工程创建 SonarQube Token

创建流水线工程

创建凭证

创建流水线

在 KubeSphere 控制台查看结果

Kubesphere运维Kubernetes集群

背景

解决方案

安装步骤

依赖包安装

配置DNS解析

按照官方教程安装多节点kubernetes

使用

利用Docker自建DNS服务器

背景

步骤

搭建DNS服务器

服务器指向新DNS服务器

补充

Sentinel持久化存储和镜像构建

背景

解决方法

进一步完善

添加Dockerfile

使用方法

GitHub DevOps工作流构建Java项目并上传到DockerHub

工程准备

准备Dockerfile

编写GitHub工作流

安全配置

触发构建

检查结果

总结

Docker MySQL主从复制方案

背景

现状

改造过程

配置主服务器

新增同步用户并授权

开启binlog日志

备份主数据库数据

配置从服务器

可能出现的问题

死锁——waiting for global read lock

k8s环境下GitLab+Helm+GitLab Runner Java项目CICD落地实践

背景

优势

环境

GitLab-Runner部署

配置介绍

使用DinD方式构建已经不再推荐

Use docker-in-docker workflow with Docker executor

Using docker:dind

使用Kaniko构建Docker镜像

使用Helm部署

GitLab-Runner选择可能导致的坑

安装

工程配置

DinD方式构建所需配置

Kubectl和Kubernetes权限配置

Kubernetes关联GitLab配置

GitLab环境配置

CI脚本配置

执行流水线

runner自动扩缩容

使用DinD构建Dockerfile结果

使用Kaniko构建Dockerfile的结果

Using `docker:dind`

使用`DinD`构建`Dockerfile`结果

使用`Kaniko`构建`Dockerfile`的结果

1.当前`docker-compose`已经更新到`3.7`了为什么还使用`2`这个老版本？

2.当前`Elasticsearch`已经更新到`7.3`了为什么还使用`6.6`这个老版本？