Kubernetes on F嘉阳

ArgoCD落地GitOps

Sat, 23 Aug 2025 22:49:50 +0800

背景

在过去的DevOps实践中，跨环境的基础设施部署和迁移经常存在遗漏，导致环境差异，部署一个新环境往往需要几天。传统CICD流水线采用“推”模式，将变更推送到不同的环境，推送后也不对目标环境状态进行管理和监控，容易导致推送后部署失败。随着云原生容器化已经成为行业事实标准，GitOps成为一种更可靠的应用部署和基础设置运维的范式。

技术选型

ArgoCD遵循GitOps的核心原则，已经成为落地GitOps的默认技术选型，因此此处不探讨其他选型方案。

原理

Git 作为唯一事实来源（GitOps思想）
自动化和持续同步：Argo CD会持续监控 Git 仓库的状态，并自动将集群的实际状态与 Git 中声明的目标状态进行同步。
- 利用K8s的Kustomize能力完成对集群apply操作

实践

环境

安装ArgoCD

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

检查安装结果

查看argo cd登录密码

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

设计代码库结构

落地ArgoCD通常有以下几种仓库设计范式

单仓库模式
多仓库模式
App of Apps模式

本次采用App of Apps的单仓库模式，仓库结构如下

.
├── apps
│ ├── app-one
│ ├── app-two
├── cluster-infra
│ ├── argocd
│ │ ├── base
│ │ └── overlays
│ ├── ingress
│ │ ├── base
│ │ └── overlays
│ │ └── dev
│ └── prometheus-stack
│ ├── base
│ └── overlays
│ └── dev
└── environments
 └── dev
 └── root.yaml

该结构特点如下

基础设施和应用部署分离（apps和cluster-infra）
环境基础配置和环境定制配置分离（base和overlays）
环境根节点配置（environments控制git仓库配置、应用apps注册）

本次将该代码库存放到阿里云代码仓库中。

环境配置迁移

目前的k3s环境中已经手动运维安装了argocd，现在需要将已安装的配置迁移到git库中

将官方安装配置https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml写入到cluster-infra/argocd/base/argocd.yaml中

配置``cluster-infra/argocd/base/kustomization.yaml`文件，注册同目录的安装文件

resources:
 - argocd.yaml

本次依赖阿里云代码仓库，通过ssh连接，需要覆盖默认的argocd配置，编写cluster-infra/argocd/overlays/with-known-hosts/ssh-known-hosts.patch.yaml

apiVersion: v1 
kind: ConfigMap 
metadata: 
 name: argocd-ssh-known-hosts-cm 
data: 
 ssh_known_hosts: |- 
 # BEGIN >>> 自定义条目（全局生效） 
 codeup.aliyun.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6WbValczSDKC4ktjRzDNC1qINDAViw/oGA8ud7qH5F9XFZ4QAWhZmFBsp+cppBIOM63exjc4O9vbzMrd7AUItXigymGfTqrGt5VykIiaNhjttxSwbZf3N9eDcAvX4gGm3fcm6D1ACvPxvxMsCwOpdz5tDHqDxGOLPAXN+FSAvjdcN4uMz0MXghGh9Q3zSpy8z1TLkAKT2QBnr4q2tsdZKwWON8bBHIPwP3RSAdSUL6bAyotyncMDAvG53kwRKJ3wJ5ZezXqZNNV7EzHds+pvwThtuWtWsBuuRpL/oZiviWlXy6hILoWbkRfgONPwT0faaOn1GLzjHLuqHHfALGOen 
 # END <<< 自定义条目 
 
 # This file was automatically generated by hack/update-ssh-known-hosts.sh. DO NOT EDIT 
 [ssh.github.com]:443 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg= 
 [ssh.github.com]:443 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl 
 [ssh.github.com]:443 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk= 
 bitbucket.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPIQmuzMBuKdWeF4+a2sjSSpBK0iqitSQ+5BM9KhpexuGt20JpTVM7u5BDZngncgrqDMbWdxMWWOGtZ9UgbqgZE= 
 bitbucket.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO 
 bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQeJzhupRu0u0cdegZIa8e86EG2qOCsIsD1Xw0xSeiPDlCr7kq97NLmMbpKTX6Esc30NuoqEEHCuc7yWtwp8dI76EEEB1VqY9QJq6vk+aySyboD5QF61I/1WeTwu+deCbgKMGbUijeXhtfbxSxm6JwGrXrhBdofTsbKRUsrN1WoNgUa8uqN1Vx6WAJw1JHPhglEGGHea6QICwJOAr/6mrui/oB7pkaWKHj3z7d1IC4KWLtY47elvjbaTlkN04Kc/5LFEirorGYVbt15kAUlqGM65pk6ZBxtaO3+30LVlORZkxOh+LKL/BvbZ/iRNhItLqNyieoQj/uh/7Iv4uyH/cV/0b4WDSd3DptigWq84lJubb9t/DnZlrJazxyDCulTmKdOR7vs9gMTo+uoIrPSb8ScTtvw65+odKAlBj59dhnVp9zd7QUojOpXlL62Aw56U4oO+FALuevvMjiWeavKhJqlR7i5n9srYcrNV7ttmDw7kf/97P5zauIhxcjX+xHv4M= 
 github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg= 
 github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl 
 github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk= 
 gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY= 
 gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf 
 gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9 
 ssh.dev.azure.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H 
 vs-ssh.visualstudio.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

配置``cluster-infra/argocd/overlays/with-known-hosts/kustomization.yaml`文件，注册hosts patch文件

resources: 
 - ../../base 
 
patchesStrategicMerge: 
 - ssh-known-hosts.patch.yaml

按这样配置后，argocd会先执行base的安装文件apply基础环境，再使用ssh-known-hosts.patch.yaml覆盖ssh的配置，保证环境与git配置一致。

接下来，继续在apps根目录注册应用，配置environments/dev/kustomization.yaml

resources:
 - ../../cluster-infra/argocd/overlays/with-known-hosts

编写根目录git配置

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: dev-root 
 namespace: argocd 
spec: 
 project: default 
 source: 
 repoURL: git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git 
 targetRevision: master 
 path: environments/dev 
 kustomize: 
 enableHelm: true 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: argocd 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

到此，所有配置迁移完成，将git仓库提交到远程后，在master节点添加私钥

argocd repo add git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git --ssh-private-key-path /root/.ssh/gitops_ed25519

同步dev根应用

kubectl apply -f dev-root.yaml

本地使用kubectl暴露临时端口，再使用argocd cli连接集群查看状态

kubectl port-forward svc/argocd-server -n argocd --address 0.0.0.0 30443:443
argocd login xxxx:30443 --username admin
argocd app list

部署ingress

一直使用临时端口+命令行查看应用状态也不方便，官方提供webui界面查看部署状态，考虑使用ingress暴露固定域名访问

ingress的部署也遵循相同步骤

配置官方安装文件
配置环境个性化
1. 端口暴露
根应用添加配置

将官方https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml安装文件配置到cluster-infra/ingress/base/ingress-nginx.yaml

配置端口暴露cluster-infra/ingress/overlays/dev/ingress-nginx-controller-nodeport.yaml

apiVersion: v1 
kind: Service 
metadata: 
 name: ingress-nginx-controller 
 namespace: ingress-nginx 
spec: 
 type: NodePort 
 ports: 
 - name: http 
 port: 80 
 targetPort: http 
 protocol: TCP 
 nodePort: 30080 
 - name: https 
 port: 443 
 targetPort: https 
 protocol: TCP 
 nodePort: 30443

根应用添加配置

resources: 
 - ../../cluster-infra/ingress/overlays/dev 
 - ../../cluster-infra/argocd/overlays/with-known-hosts

在argocd环境目录下添加ingress域名配置

apiVersion: networking.k8s.io/v1 
kind: Ingress 
metadata: 
 name: argocd-ingress 
 namespace: argocd 
 annotations: 
 nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" 
spec: 
 ingressClassName: nginx 
 rules: 
 - host: xxxx 
 http: 
 paths: 
 - path: / 
 pathType: Prefix 
 backend: 
 service: 
 name: argocd-server 
 port: 
 number: 443

git提交后即可自动同步并创建ingress

访问域名查看argocd webui

部署Prometheus+Grafana

为了完善集群监控，argocd 支持通过helm charts部署应用

配置cluster-infra/prometheus-stack/base/prometheus-stack.yaml

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: prometheus-stack 
 namespace: argocd 
 annotations: 
 argocd.argoproj.io/sync-options: ServerSideApply=true 
spec: 
 project: default 
 source: 
 repoURL: https://prometheus-community.github.io/helm-charts 
 chart: kube-prometheus-stack 
 targetRevision: 75.16.1 
 helm: 
 valueFiles: 
 - values.yaml 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: monitoring 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

注意，由于Helm仓库部署prometheus时伴随大量CRD资源文件apply，需要开启ServerSideApply模式避免通过接口apply导致字符超长报错。参考：https://argocd.devops.gold/user-guide/sync-options/#_9

使用helm模式安装的应用无法使用patchesStrategicMerge策略覆盖，目前覆盖个性化配置只能将所有内容写到一个资源文件内，因此dev环境的个性化配置values如下

prometheus: 
 prometheusSpec: 
 retention: 3d 
 retentionSize: "2Gi" 
 resources: 
 requests: 
 memory: 128Mi 
 limits: 
 memory: 1Gi 
 storageSpec: 
 volumeClaimTemplate: 
 spec: 
 resources: 
 requests: 
 storage: 5Gi 
 ingress: 
 enabled: true 
 ingressClassName: nginx 
 annotations: {} 
 hosts: 
 - prometheus.fjy8018.top 
 paths: 
 - / 
 tls: 
 - hosts: 
 - xxx.xxx
 secretName: prometheus-tls 
 prometheusSpec: 
 externalUrl: https://xxx.xxx 
 routePrefix: / 
 
grafana: 
 adminPassword: "2EvwJJ0NOLzhYa" 
 resources: 
 requests: 
 memory: 64Mi 
 limits: 
 memory: 512Mi 
 ingress: 
 enabled: true 
 ingressClassName: nginx 
 annotations: {} 
 hosts: 
 - xxx.xxx 
 tls: 
 - hosts: 
 - xxx.xxx
 secretName: grafana-tls

需要在dev中覆盖base的资源文件，通过multi repo模式合并helm配置

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: prometheus-stack 
 namespace: argocd 
 annotations: 
 argocd.argoproj.io/sync-options: ServerSideApply=true 
spec: 
 project: default 
 sources: 
 - repoURL: https://prometheus-community.github.io/helm-charts 
 chart: kube-prometheus-stack 
 targetRevision: 75.16.1 
 helm: 
 valueFiles: 
 - $values/cluster-infra/prometheus-stack/overlays/dev/values-dev.yaml 
 - repoURL: 'git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git' 
 targetRevision: master 
 ref: values 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: monitoring 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

prometheus依赖CRD提前部署的namespace需要注册为独立应用提前配置好

apiVersion: v1 
kind: Namespace 
metadata: 
 name: monitoring

命名空间的配置和普通应用一致

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: common-ns 
 namespace: argocd 
spec: 
 project: default 
 source: 
 repoURL: git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git 
 targetRevision: master 
 path: cluster-infra/common-ns 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: argocd 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

最终根目录应用配置如下

resources: 
 - ../../cluster-infra/ingress/overlays/dev 
 - ../../cluster-infra/argocd/overlays/with-known-hosts 
 - ../../cluster-infra/prometheus-stack/overlays/dev 
 - common-ns-app.yaml

同步后效果如下

查看监控部署拓扑

通过域名访问grafana

Kubesphere落地DevOps

Sat, 19 Jun 2021 22:16:00 +0800

创建企业空间

创建一个企业空间和一个帐户 (project-admin)，必须邀请该帐户至该企业空间并赋予 workspace-self-provisioner 角色。

进入企业空间，切换到DevOps工程，点击创建

创建工程

安装helm

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

找不到环境变量报错

安装helm遇到报错

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
 % Total % Received % Xferd Average Speed Time Time Time Current
unset -f pathmunge
 Dload Upload Total Spent Left Speed
100 11248 100 11248 0 0 61543 0 --:--:-- --:--:-- --:--:-- 61802
Helm v3.6.0 is available. Changing from version v3.2.1.
Downloading https://get.helm.sh/helm-v3.6.0-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
helm not found. Is /usr/local/bin on your $PATH?
Failed to install helm
 For support, go to https://github.com/helm/helm.

则需要配置环境变量

$ vi /etc/profile
# 添加配置
export PATH=/usr/local/bin:$PATH

$ source /etc/profile

验证

$ helm version
version.BuildInfo{Version:"v3.6.0", GitCommit:"7f2df6467771a75f5646b7f12afb408590ed1755", GitTreeState:"clean", GoVersion:"go1.16.3"}

安装SonarQube

执行以下命令安装 SonarQube 服务器

$ helm upgrade --install sonarqube sonarqube --repo https://charts.kubesphere.io/main -n kubesphere-devops-system --create-namespace --set service.type=NodePort

输出

执行以下命令以获取 SonarQube NodePort

$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services sonarqube-sonarqube)
$ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo http://$NODE_IP:$NODE_PORT

输出

此时可在kubesphere控制台中看到sonar服务

配置 SonarQube 服务器

访问地址，点击右上角的 Log in，然后使用默认帐户 admin/admin 登录。

创建 SonarQube 管理员令牌 (Token)

点击右上角字母 A，然后从菜单中选择 My Account 以转到 Profile 页面。

生成token

创建 Webhook 服务器

执行以下命令获取 SonarQube Webhook 的地址。

$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services ks-jenkins)
$ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo http://$NODE_IP:$NODE_PORT/sonarqube-webhook/

输出

依次点击 Administration、Configuration 和 Webhooks 创建一个 Webhook。

点击 Create

由于webhook都通过内部网络调用，故可以使用内部网络地址

将 SonarQube 配置添加到 ks-installer

执行以下命令编辑 ks-installer

$ kubectl edit cc -n kubesphere-system ks-installer

搜寻至 devops。添加字段 sonarqube 并在其下方指定 externalSonarUrl 和 externalSonarToken。

 devops:
 enabled: true
 jenkinsJavaOpts_MaxRAM: 2g
 jenkinsJavaOpts_Xms: 512m
 jenkinsJavaOpts_Xmx: 512m
 jenkinsMemoryLim: 2Gi
 jenkinsMemoryReq: 1500Mi
 jenkinsVolumeSize: 8Gi
 sonarqube:
 externalSonarUrl: http://10.170.0.5:32108
 externalSonarToken: c6a6e2e15a9d52e30b744df0eef4c639bbea88a1

完成操作后保存此文件。

将 SonarQube 服务器添加至 Jenkins

执行以下命令获取 Jenkins 的地址。

$ export NODE_PORT=$(kubectl get --namespace kubesphere-devops-system -o jsonpath="{.spec.ports[0].nodePort}" services ks-jenkins)
$ export NODE_IP=$(kubectl get nodes --namespace kubesphere-devops-system -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo http://$NODE_IP:$NODE_PORT

输出

使用地址 http://{$Public IP}:30180 访问 Jenkins。安装 KubeSphere 时，默认情况下也会安装 Jenkins 仪表板。此外，Jenkins 还配置有 KubeSphere LDAP，这意味着您可以直接使用 KubeSphere 帐户（例如 admin/P@88w0rd）登录 Jenkins。

使用kubesphere admin账户登录

前往系统管理下的 Manage Credentials 并点击 Stores scoped to Jenkins 下的 Jenkins，再点击全局凭据 (unrestricted)

点击左侧导航栏的添加凭据，用 SonarQube 管理员令牌添加凭证

点击左侧的系统管理，向下翻页找到并点击系统配置。

搜寻到 SonarQube servers，然后点击 Add SonarQube。

添加sonar服务器地址和名称，点击应用完成操作。

将 sonarqubeURL 添加到 KubeSphere 控制台

指定 sonarqubeURL，以便可以直接从 KubeSphere 控制台访问 SonarQube。

执行以下命令：

$ kubectl edit cm -n kubesphere-system ks-console-config

搜寻到 client，添加 devops 字段并指定 sonarqubeURL。

 client:
 version:
 kubesphere: v3.1.0
 kubernetes: v1.20.4
 openpitrix: v0.3.5
 enableKubeConfig: true
 devops:
 sonarqubeURL: http://34.92.90.181:32108

结果

重启服务

执行以下命令。

$ kubectl -n kubesphere-system rollout restart deploy ks-apiserver
$ kubectl -n kubesphere-system rollout restart deploy ks-console

为新工程创建 SonarQube Token

创建一个 SonarQube 令牌，以便您的流水线可以在运行时与 SonarQube 通信。

在 SonarQube 控制台上，点击 Create new project。

设置工程名称

生成token

创建令牌后，点击 Continue。

分别选择 Java 和 Maven。复制下图所示绿色框中的序列号，如果要在流水线中使用，则需要在凭证中添加此序列号。

创建流水线工程

创建凭证

凭证 ID	类型	用途
dockerhub-id	帐户凭证	Docker Hub
demo-kubeconfig	kubeconfig	Kubernetes

demo-kubeconfig

dockerhub-id

创建一个凭证 ID (sonar-token)

列表中看到已创建的三个凭证。

创建流水线

在流水线页面点击创建，为了使用图形化流水线搭建，不在此处引用代码仓库，直接点击下一步

保持默认配置，点击创建

创建成功

进入流水线，点击编辑

此时选择CICD模板

可视化编辑流水线

编辑后运行

流水线支持交互式操作

在 KubeSphere 控制台查看结果

流水线执行成功后，即可查看到代码质量标签

sonar控制台也可以查看对应结果

jenkins视图

blue ocean

Kubesphere运维Kubernetes集群

Sun, 23 May 2021 22:58:00 +0800

背景

使用原生Kubernetes-dashboard和Grafana+Prometheus管理较大规模集群比较困难，有如下缺陷

每次执行命令都要登陆服务器
容器日志采集方案、监控方案搭建比较繁琐
添加节点不方便
管理存储插件、秘钥、配置不方便

解决方案

目前国内流行的k8s管理平台主要有rancher、kubesphere、kubeboard

rancher主打原生集群管理，侧重运维人员
kubesphere主打开发运维工作流，侧重研发人员
kubeboard与kubesphere类似，侧重微服务管理，但功能上较少，对多节点管理需要收费

经过评估，最终决定采用kubesphere进行集群管理

Rancher — 多云混合云多集群Kubernetes管理平台

[post url=“https://kubesphere.io/zh/” title=“Kubesphere” intro=“面向云原生应用的容器混合云” cover=“https://kubesphere.io/images/logo.svg” /]

Kuboard — 快速在 Kubernetes 上落地微服务

安装步骤

目标是面对小规模集群管理，节点数最多为5个，故不考虑高可用集群配置，采用单主节点+4个worker节点配置

采用纯净CentOS 7 Linux系统，需要做以下前置准备

依赖包安装

安装官方要求的依赖包，由于计划安装NFS存储插件，还需要安装nfs工具包

$ yum install -y socat conntrack ebtables ipset nfs-utils

配置DNS解析

将各节点DNS配置为私有DNS服务器

$ tee /etc/resolv.conf <<-'EOF'
# Generated by NetworkManager
nameserver 192.168.1.13
EOF

私有DNS搭建可以参考文章

[[利用Docker自建Dns服务器]]

按照官方教程安装多节点kubernetes

强烈建议先用KubeKey安装纯净k8s集群+控制面板，关闭所有插件。等安装完毕再一个个插件打开，防止因为内存不够而无法打开kubesphere

安装步骤参考官方文档一键安装方式

多节点安装 — Kubesphere

启动成功

启动成功后按照官方指南打开插件

启用可插拔组件 — Kubesphere

所有插件打开后如下

使用

节点状态

存储卷管理

监控告警

日志采集

日志查询

操作审计

工具箱

企业空间权限控制

内置helm应用商店，支持上线审批，在线编辑等

支持关联外部仓库

应用流量治理

链路追踪

灰度发布

费用统计

更多的使用场景包括：DevOps、边缘节点、多集群管理还在探索中

利用Docker自建DNS服务器

Sun, 09 May 2021 18:14:12 +0800

背景

当Linux集群中机器不断增多，集群内通过配置/hosts实现域名内网访问。

目标

相同域名在集群之外通过公有DNS访问外网地址
相同域名在集群内通过DNS解析访问内网地址

步骤

搭建DNS服务器

DNS服务器可以通过jpillora/dnsmasq容器实现

jpillora/dnsmasq — DockerHub

新增配置文件 dnsmasq.conf

# 完整配置文件参考以下地址
# http://oss.segetech.com/intra/srv/dnsmasq.conf
# 记录所有DNS查询请求
log-queries
# 不使用本地配置的域名解析器
no-resolv
# 不使用本地hosts
no-hosts
# 配置上游DNS服务器地址，用于解析国内域名
server=114.114.114.114
# 配置国际DNS服务器地址
server=8.8.8.8
strict-order
# 配置内网解析
address=/gitlab.fjy8018.top/192.168.1.11

配置docker-compose文件，DNS服务器默认通过53 udp端口解析，不建议修改53端口的外部地址

version: "3.5"

services:

 dnsmasq:
 image: jpillora/dnsmasq:1.1.0
 networks:
 - dns-net
 ports:
 - 53:53/udp
 - 5380:8080
 volumes:
 - ./dnsmasq.conf:/etc/dnsmasq.conf
 environment:
 - "HTTP_USER=admin"
 - "HTTP_PASS=admin"
 restart: always

networks:
 dns-net:

启动服务即可

后续需要修改DNS解析只需要访问5380的web UI界面即可动态调整

服务器指向新DNS服务器

将所有服务器/etc/resolv.conf文件指向新DNS服务器地址

$ tee /etc/resolv.conf <<-'EOF'
# Generated by NetworkManager
nameserver 192.168.1.13
EOF

补充

对于kubernetes集群，由于集群内使用了coredns和nodelocaldns缓存服务，还需要重启两个dns服务组件

$ kubectl -n kube-system rollout restart daemonset nodelocaldns
$ kubectl -n kube-system rollout restart deploy coredns

DevOps实践心得二

Sun, 27 Dec 2020 12:25:47 +0800

背景

上一篇DevOps实践心得编写于2019年08月16日

文章最后提到未来探索方向是

全面容器化服务治理方向

国内对容器化有非常多的实践

kubernetes作为容器调度成为大多数服务治理的选择

容器隔离、迁移和伸缩带来的巨大优势

容器与传统应用部署结合方式

以半全自动方式落地DevOps，采用Mesos+marathon方式同时兼容传统服务部署和容器化部署

兼容分布式服务治理

国外有大量的实践案例

日志收集体系完备

经过一年半的时间，完成对完整DevOps的探索

Mesos+Marathon的DevOps分布式环境探索

参考文章

该架构更加适合还未进行容器化改造的项目，其支持较为通用，但各组件都是由开源组件协调而成，故搭建较为繁琐，而且后续版本升级并不方便

全面容器化服务探索

探索历程

最终完成了DevOps容器化自动部署实践

采用kuberneties方案，其统一性更加一致，依赖的基础设施也多支持容器化部署，机器利用率较高，版本升级、维护较为方便，适用于云原生项目部署

遇到的坑

最终生产环境采用kuberneties集群方案，5个节点，但由于实施经验不足，依然遇到不少坑

后续发展展望

云原生技术还在不断发展中，不少公司也研发并开源关于kuberneties集群管理方案，例如kubesphere

kubesphere — The Kubernetes platform tailored for hybrid multicloud

经过测试，其2.x版本资源占用较高，还处于刚起步阶段，暂时不适用于小规模集群

k8s证书失效问题排查

Sun, 27 Dec 2020 12:00:58 +0800

问题背景

生产环境kuberneties集群搭建于2019年11月7日，使用kubeadm结合国内镜像搭建，参考文章

2020年11月8日突然发现生产环境集群服务全都失效并且不可用，进行紧急排查处理

处理过程

1. 排查服务是否启动

[root@k8smaster1 ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
65dff767f422 b95b1efa0436 "kube-controller-m..." 20 hours ago Up 20 hours k8s_kube-controller-manager_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_11
1b26acecbcea registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_1
9c6449df0a18 00638a24688b "kube-scheduler --..." 20 hours ago Up 20 hours k8s_kube-scheduler_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_10
028a1177cc9b registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_1
6c6bdc2c2644 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1
fe4a12daa7d7 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1

容器服务正常，但kubelet服务有报错信息，报连接不上apiserver

[root@k8smaster1 ~]$ systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
 Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
 Drop-In: /usr/lib/systemd/system/kubelet.service.d
 └─10-kubeadm.conf
 Active: active (running) since 日 2020-11-08 22:16:07 CST; 20h ago
 Docs: https://kubernetes.io/docs/
 Main PID: 15313 (kubelet)
 Tasks: 16
 Memory: 68.4M
 CGroup: /system.slice/kubelet.service
 └─15313 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1

11月 09 18:43:06 k8smaster1 kubelet[15313]: E1109 18:43:06.967416 15313 reflector.go:126] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://192.168.1.14:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.14:6443: connect: connection refused
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.009411 15313 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.109665 15313 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:43:07 k8smaster1 kubelet[15313]: I1109 18:43:07.165755 15313 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.167106 15313 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSIDriver: Get https://192.168.1.14:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 192.168.1.14:6443: connect: connection refused
11月 09 18:43:07 k8smaster1 kubelet[15313]: I1109 18:43:07.169000 15313 kubelet_node_status.go:72] Attempting to register node k8smaster1
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.209831 15313 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.310013 15313 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.362049 15313 kubelet_node_status.go:94] Unable to register node "k8smaster1" with API server: Post https://192.168.1.14:6443/api/v1/nodes: dial tcp 192.168.1.14:6443: connect: connection refused
11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.410174 15313 kubelet.go:2244] node "k8smaster1" not found

2. 检查APIServer

apiserver 端口连接正常

[root@k8smaster1 ~]$ telnet 192.168.1.14 6443
Trying 192.168.1.14...
telnet: connect to address 192.168.1.14: Connection refused

重设防火墙，并重新检查，连接正常

[root@k8smaster1 ~]$ iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT
[root@k8smaster1 ~]$ telnet 192.168.1.14 6443
Trying 192.168.1.14...
Connected to 192.168.1.14.

但服务仍然无法访问

3. 重启kubelet

[root@k8smaster1 ~]$ systemctl restart kubelet
[root@k8smaster1 ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4383926ef6c9 ecf910f40d6e "kube-apiserver --..." 3 seconds ago Up 2 seconds k8s_kube-apiserver_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1362
88eb6134fe3c 2c4adeb21b4f "etcd --advertise-..." 3 seconds ago Up 2 seconds k8s_etcd_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1768
65dff767f422 b95b1efa0436 "kube-controller-m..." 20 hours ago Up 20 hours k8s_kube-controller-manager_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_11
1b26acecbcea registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_1
9c6449df0a18 00638a24688b "kube-scheduler --..." 20 hours ago Up 20 hours k8s_kube-scheduler_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_10
028a1177cc9b registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_1
6c6bdc2c2644 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1
fe4a12daa7d7 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 20 hours ago Up 20 hours k8s_POD_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1

检查kubelet状态，仍然报错

[root@k8smaster1 ~]$ systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
 Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
 Drop-In: /usr/lib/systemd/system/kubelet.service.d
 └─10-kubeadm.conf
 Active: active (running) since 一 2020-11-09 18:52:24 CST; 24s ago
 Docs: https://kubernetes.io/docs/
 Main PID: 31759 (kubelet)
 Tasks: 15
 Memory: 30.1M
 CGroup: /system.slice/kubelet.service
 └─31759 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1

11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.063832 31759 controller.go:115] failed to ensure node lease exists, will retry in 3.2s, error: Get https://192.168.1.14:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/k8smaster1?timeout=10s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.152723 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.252964 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.353226 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.453452 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.553682 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.653965 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.754181 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.854401 31759 kubelet.go:2244] node "k8smaster1" not found
11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.954576 31759 kubelet.go:2244] node "k8smaster1" not found

4.检查API Server日志

查容器ID

[root@k8smaster1 ~]$ docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
27a3b7787748 ecf910f40d6e "kube-apiserver --..." 18 seconds ago Up 18 seconds

查看日志

[root@k8smaster1 ~]$ docker logs 27a
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I1109 10:55:42.281078 1 server.go:559] external host was not specified, using 192.168.1.15
I1109 10:55:42.281264 1 server.go:146] Version: v1.14.0
I1109 10:55:42.759000 1 plugins.go:158] Loaded 9 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I1109 10:55:42.759031 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
E1109 10:55:42.759943 1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.759976 1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.760007 1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.760038 1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.760065 1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E1109 10:55:42.760086 1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
I1109 10:55:42.760102 1 plugins.go:158] Loaded 9 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I1109 10:55:42.760110 1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I1109 10:55:42.763844 1 client.go:352] parsed scheme: ""
I1109 10:55:42.763866 1 client.go:352] scheme "" not registered, fallback to default scheme
I1109 10:55:42.764295 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I1109 10:55:42.764449 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
W1109 10:55:42.771517 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
I1109 10:55:43.756910 1 client.go:352] parsed scheme: ""
I1109 10:55:43.756939 1 client.go:352] scheme "" not registered, fallback to default scheme
I1109 10:55:43.756999 1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}]
I1109 10:55:43.757077 1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
W1109 10:55:43.763355 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:43.770837 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:44.764994 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:45.541971 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:46.657328 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:47.938720 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:48.730961 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:51.899665 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:52.961615 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:55:59.193916 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
W1109 10:56:00.025333 1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting...
F1109 10:56:02.764150 1 storage_decorator.go:57] Unable to create storage backend: config (&{ /registry {[https://127.0.0.1:2379] /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/etcd/ca.crt} false true 0xc0007a94d0 apiextensions.k8s.io/v1beta1 <nil> 5m0s 1m0s}), err (context deadline exceeded)

日志最后提示：authentication handshake failed: x509: certificate has expired or is not yet valid

以此信息查询解决方案

解决方案

外部推荐方案

最终在该网站找到解决方案

Kubeadm证书过期时间调整 — Devops、小铁匠

采用方案

由于生产环境集群版本为1.14，是支持自动轮换证书的，故首先解决生产环境问题，先手动重新签发证书，再开启证书自动续签

重新签发证书

国内网络直接签发证书会因为无法连接Google而报错

[root@k8smaster1 ~]$ kubeadm alpha certs renew all
I1109 19:33:40.266778 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:33:40.266978 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:33:51.139051 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:33:51.139111 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:01.511909 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1109 19:34:01.511967 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:11.945411 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:34:11.945504 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:22.145905 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:34:22.145975 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:32.532514 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1109 19:34:32.532561 22487 version.go:97] falling back to the local client version: v1.14.0
I1109 19:34:43.335574 22487 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1109 19:34:43.335628 22487 version.go:97] falling back to the local client version: v1.14.0

先准备一份连接国内镜像的kubeadm.conf文件

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.14.0 #-->这里改成你集群对应的版本
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers 
#这里使用国内的镜像仓库，否则在重新签发的时候会报错：could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt"

重新签发，即可，并重启kubelet

[root@k8smaster1 ~]$ kubeadm alpha certs renew all --config=/root/kubeadm.conf
[root@k8smaster1 ~]$ systemctl restart kubelet

重新签发后若要使用kubectl命令，要重新生成~/.kube/config配置文件

[root@k8smaster1 kubernetes]$ kubeadm init phase kubeconfig all --config=/root/kubeadm.conf
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[root@k8smaster1 kubernetes]$ cp -i /etc/kubernetes/admin.conf ~/.kube/config
cp：是否覆盖"/root/.kube/config"？ y

命令正常使用

[root@k8smaster1 kubernetes]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
glusterfs-hfdmh 1/1 Running 0 351d
glusterfs-k766z 1/1 Running 0 351d
glusterfs-rrc7x 1/1 Running 0 351d
heketi-68f9dfdfbf-2k58b 1/1 Running 0 351d
nginx 1/1 Running 1 370d
nginx-ds-29dbc 1/1 Running 5 366d
nginx-ds-4w6cn 1/1 Running 1 370d
nginx-ds-6lhsk 0/1 Evicted 0 216d
nginx-ds-xq4h7 1/1 Running 3 366d
tomcat-demo-6bc7d5b6f4-75rgc 0/1 Evicted 0 351d

重新启动集群上的所有服务，可见服务正常

证书自动续签

kubelet证书分为server和client两种， k8s 1.9默认启用了client证书的自动轮换，但server证书自动轮换需要用户开启

查询kubelet配置文件位置

[root@k8smaster1 ~]$ find / -name 10-kubeadm.conf
/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
[root@k8smaster1 ~]$ vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

添加配置如下

# 在/etc/systemd/system/kubelet.service.d/10-kubeadm.conf 增加如下参数
Environment="KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true"

添加Controller-Manager参数

apiVersion: v1
kind: Pod
metadata:
 creationTimestamp: null
 labels:
 component: kube-controller-manager
 tier: control-plane
 name: kube-controller-manager
 namespace: kube-system
spec:
 containers:
 - command:
 - kube-controller-manager
 - --allocate-node-cidrs=true
 # 证书有效期
 - --experimental-cluster-signing-duration=87600h0m0s
 # 证书自动签发
 - --feature-gates=RotateKubeletServerCertificate=true
 - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
 - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
 - --bind-address=127.0.0.1
 - --client-ca-file=/etc/kubernetes/pki/ca.crt
 - --cluster-cidr=172.22.0.0/16
 - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
 - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
 - --controllers=*,bootstrapsigner,tokencleaner
 - --kubeconfig=/etc/kubernetes/controller-manager.conf
 - --leader-elect=true
 - --node-cidr-mask-size=24
 - --requestheader-client-ca-file=/etc/k

创建RBAC对象

cat > ca-update.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 annotations:
 rbac.authorization.kubernetes.io/autoupdate: "true"
 labels:
 kubernetes.io/bootstrapping: rbac-defaults
 name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
 - certificates.k8s.io
 resources:
 - certificatesigningrequests/selfnodeserver
 verbs:
 - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: kubeadm:node-autoapprove-certificate-server
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
 kind: Group
 name: system:nodes
EOF

[root@k8smaster1 ~]$ kubectl create -f ca-update.yaml
clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeserver created
clusterrolebinding.rbac.authorization.k8s.io/kubeadm:node-autoapprove-certificate-server created

k8s环境下GitLab+Helm+GitLab Runner Java项目CICD落地实践

Wed, 27 Nov 2019 10:24:00 +0800

背景

目前使用5台服务器搭建了Kubernetes集群环境，监控、日志采集均已落地，业务也手工迁移到集群中顺利运行，故需要将原本基于原生docker环境的CICD流程迁移到Kubernetes集群中

优势

Kubernetes集群实现CICD有几个显著优势

Deployment 天然支持滚动部署、结合其他Kubernetes特性还能实现蓝绿部署、金丝雀部署等
新版本的GitLab与GitLab Runner天然支持Kubernetes集群，支持runner自动伸缩，减小资源占用

环境

Kubernetes版本：1.14

GitLab版本：12.2.5

GitLab-Runner版本：12.1.0

Docker环境版本：17.03.1

GitLab-Runner部署

配置介绍

原始环境的gitlab runner通过手动执行官网提供的注册命令和启动命令，分成两部部署，需要较多的手工操作，而在Kubernetes中，其支持使用Helm一键部署，官方文档如下

GitLab Runner Helm Chart

其实官方文档的指引并不清晰，许多配置在文档中没有介绍用法，推荐去其源码仓库查看详细的参数使用文档

The Kubernetes executor

其中介绍了几个关键配置，在后面修改工程的ci配置文件时会用到

使用DinD方式构建已经不再推荐

官方文档介绍

Use docker-in-docker workflow with Docker executor
The second approach is to use the special docker-in-docker (dind) Docker image with all tools installed (docker) and run the job script in context of that image in privileged mode.

Note: docker-compose is not part of docker-in-docker (dind). To use docker-compose in your CI builds, follow the docker-compose installation instructions.

Danger: By enabling --docker-privileged, you are effectively disabling all of the security mechanisms of containers and exposing your host to privilege escalation which can lead to container breakout. For more information, check out the official Docker documentation on Runtime privilege and Linux capabilities.

Docker-in-Docker works well, and is the recommended configuration, but it is not without its own challenges:

When using docker-in-docker, each job is in a clean environment without the past history. Concurrent jobs work fine because every build gets it’s own instance of Docker engine so they won’t conflict with each other. But this also means jobs can be slower because there’s no caching of layers.

By default, Docker 17.09 and higher uses --storage-driver overlay2 which is the recommended storage driver. See Using the overlayfs driver for details.

Since the docker:19.03.1-dind container and the Runner container don’t share their root filesystem, the job’s working directory can be used as a mount point for child containers. For example, if you have files you want to share with a child container, you may create a subdirectory under /builds/$CI_PROJECT_PATH and use it as your mount point (for a more thorough explanation, check issue #41227):

总之使用DinD进行容器构建并非不可行，但面临许多问题，例如使用overlay2网络需要Docker版本高于 17.09

Using docker:dind
Running the docker:dind also known as the docker-in-docker image is also possible but sadly needs the containers to be run in privileged mode. If you’re willing to take that risk other problems will arise that might not seem as straight forward at first glance. Because the docker daemon is started as a service usually in your .gitlab-ci.yaml it will be run as a separate container in your Pod. Basically containers in Pods only share volumes assigned to them and an IP address by which they can reach each other using localhost. /var/run/docker.sock is not shared by the docker:dind container and the docker binary tries to use it by default.

To overwrite this and make the client use TCP to contact the Docker daemon, in the other container, be sure to include the environment variables of the build container:

DOCKER_HOST=tcp://localhost:2375 for no TLS connection.

DOCKER_HOST=tcp://localhost:2376 for TLS connection.

Make sure to configure those properly. As of Docker 19.03, TLS is enabled by default but it requires mapping certificates to your client. You can enable non-TLS connection for DIND or mount certificates as described in Use Docker In Docker Workflow wiht Docker executor

在Docker 19.03.1版本之后默认开启了TLS配置，在构建的环境变量中需要声明，否则报连接不上docker的错误，并且使用DinD构建需要runner开启特权模式，以访问主机的资源，并且由于使用了特权模式，在Pod中对runner需要使用的资源限制将失效

使用Kaniko构建Docker镜像

目前官方提供另一种方式在docker容器中构建并推送镜像，实现更加优雅，可以实现无缝迁移，那就是kaniko

Building a Docker image with kaniko

其优势官网描述如下

在Kubernetes集群中构建Docker映像的另一种方法是使用kaniko。iko子

允许您构建没有特权访问权限的映像。

无需Docker守护程序即可工作。

在后面的实践中会使用两种方式构建Docker镜像，可根据实际情况选择

使用Helm部署

拉取Helm Gitlab-Runner仓库到本地，修改配置

GitLab Runner

将原有的gitlab-runner配置迁移到Helm中，迁移后如下

image: alpine-v12.1.0
imagePullPolicy: IfNotPresent
gitlabUrl: https://gitlab.fjy8018.top/
runnerRegistrationToken: "***"
unregisterRunners: true
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
rbac:
 create: true
 clusterWideAccess: false
metrics:
 enabled: true
 listenPort: 9090
runners:
 image: ubuntu:16.04
 imagePullSecrets:
 - name: registry-secret
 locked: false
 tags: "k8s"
 runUntagged: true
 privileged: true
 pollTimeout: 180
 outputLimit: 4096
 cache: {}
 builds: {}
 services: {}
 helpers: {}
resources:
 limits:
 memory: 2048Mi
 cpu: 1500m
 requests:
 memory: 128Mi
 cpu: 200m
affinity: {}
nodeSelector: {}
tolerations: []
hostAliases:
 - ip: "192.168.1.13"
 hostnames:
 - "gitlab.fjy8018.top"
 - ip: "192.168.1.30"
 hostnames:
 - "harbor.fjy8018.top"
podAnnotations: {}

其中配置了私钥、内网harbor地址、harbor拉取资源私钥，资源限制策略

GitLab-Runner选择可能导致的坑

选择runner镜像为alpine-v12.1.0，这一点单独说一下，目前最新的runner版本为12.5.0，但其有许多问题，alpine新版镜像在Kubernetes中间断发生无法解析DNS的问题，反映到GitLab-Runner中就是Could not resolve host和server misbehaving

查阅解决方法

通过查询发现，其官方仓库还有多个相关issue没有关闭

官方gitlab：Kubernetes runner: Could not resolve host

stackoverflow：Gitlab Runner is not able to resolve DNS of Gitlab Server

给出的解决方案无一例外都是降级到alpine-v12.1.0

We had same issue for couple of days. We tried change CoreDNS config, move runners to different k8s cluster and so on. Finally today i checked my personal runner and found that i’m using different version. Runners in cluster had gitlab/gitlab-runner:alpine-v12.3.0, when mine had gitlab/gitlab-runner:alpine-v12.0.1. We added line
image: gitlab/gitlab-runner:alpine-v12.1.0
in values.yaml and this solved problem for us

其问题的根源应该在于alpine基础镜像对Kubernetes 集群支持有问题，

ndots breaks DNS resolving #64924

docker-alpine仓库对应也有未关闭的issue，其中就提到了关于DNS解析超时和异常的问题

DNS Issue #255

安装

一行命令安装即可

$ helm install /root/gitlab-runner/ --name k8s-gitlab-runner --namespace gitlab-runner

输出如下

NAME: k8s-gitlab-runner
LAST DEPLOYED: Tue Nov 26 21:51:57 2019
NAMESPACE: gitlab-runner
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
k8s-gitlab-runner-gitlab-runner 5 0s

==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
k8s-gitlab-runner-gitlab-runner 0/1 1 0 0s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
k8s-gitlab-runner-gitlab-runner-744d598997-xwh92 0/1 Pending 0 0s

==> v1/Role
NAME AGE
k8s-gitlab-runner-gitlab-runner 0s

==> v1/RoleBinding
NAME AGE
k8s-gitlab-runner-gitlab-runner 0s

==> v1/Secret
NAME TYPE DATA AGE
k8s-gitlab-runner-gitlab-runner Opaque 2 0s

==> v1/ServiceAccount
NAME SECRETS AGE
k8s-gitlab-runner-gitlab-runner 1 0s


NOTES:

Your GitLab Runner should now be registered against the GitLab instance reachable at: "https://gitlab.fjy8018.top/"

查看gitlab admin页面，发现已经有一个runner成功注册

工程配置

DinD方式构建所需配置

如果原本的ci文件是基于19.03 DinD镜像构建的则需要加上TLS相关配置

image: docker:19.03

variables:
 DOCKER_DRIVER: overlay
 DOCKER_HOST: tcp://localhost:2375
 DOCKER_TLS_CERTDIR: ""
...

其余配置保持不变，使用DinD构建

Kubectl和Kubernetes权限配置

由于使用k8s集群，而通过集群部署需要使用kubectl客户端，故手动创建了一个kubectl docker镜像，使用gitlab触发dockerhub构建，构建内容公开透明，可放心使用，如有其它版本的构建需求也可提pull request，会在后面补充，目前用到的只有1.14.0

fjy8018/kubectl

有kubectl客户端，还需要配置连接TLS和连接账户

为了保障安全，新建一个专门访问该工程命名空间的ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
 name: hmdt-gitlab-ci
 namespace: hmdt

利用集群提供的RBAC机制，为该账户授予该命名空间的admin权限

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
 name: hmdt-gitlab-role
 namespace: hmdt
subjects:
 - kind: ServiceAccount
 name: hmdt-gitlab-ci
 namespace: hmdt
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: admin

创建后在查询其在k8s集群中生成的唯一名称，此处为hmdt-gitlab-ci-token-86n89

$ kubectl describe sa hmdt-gitlab-ci -n hmdt
Name: hmdt-gitlab-ci
Namespace: hmdt
Labels: <none>
Annotations: kubectl.`Kubernetes`.io/last-applied-configuration:
 {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"hmdt-gitlab-ci","namespace":"hmdt"}}
Image pull secrets: <none>
Mountable secrets: hmdt-gitlab-ci-token-86n89
Tokens: hmdt-gitlab-ci-token-86n89
Events: <none>

然后根据上面的Secret找到CA证书

$ kubectl get secret hmdt-gitlab-ci-token-86n89 -n hmdt -o json | jq -r '.data["ca.crt"]' | base64 -d

再找到对应的 Token

$ kubectl get secret hmdt-gitlab-ci-token-86n89 -n hmdt -o json | jq -r '.data.token' | base64 -d

Kubernetes关联GitLab配置

进入gitlab Kubernetes集群配置页面，填写相关信息，让gitlab自动连接上集群环境

注意，需要将此处取消勾选，否则gitlab会自动创建新的用户账户，而不使用已经创建好的用户账户，在运行过程中会报无权限错误

不取消导致的报错如下，gitlab创建了新的用户账户hmdt-prod-service-account，但没有操作指定命名空间的权限

GitLab环境配置

创建环境

名称和url可以按需自定义

CI脚本配置

最终配置CI文件如下，该文件使用DinD方式构建Dockerfile

image: docker:19.03

variables:
 MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode -Dmaven.test.skip=true"
 MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
 DOCKER_DRIVER: overlay
 DOCKER_HOST: tcp://localhost:2375
 DOCKER_TLS_CERTDIR: ""
 SPRING_PROFILES_ACTIVE: docker
 IMAGE_VERSION: "1.8.6"
 DOCKER_REGISTRY_MIRROR: "https://XXX.mirror.aliyuncs.com"

stages:
 - test
 - package
 - review
 - deploy

maven-build:
 image: maven:3-jdk-8
 stage: test
 retry: 2
 script:
 - mvn $MAVEN_CLI_OPTS clean package -U -B -T 2C
 artifacts:
 expire_in: 1 week
 paths:
 - target/*.jar

maven-scan:
 stage: test
 retry: 2
 image: maven:3-jdk-8
 script:
 - mvn $MAVEN_CLI_OPTS verify sonar:sonar

maven-deploy:
 stage: deploy
 retry: 2
 image: maven:3-jdk-8
 script:
 - mvn $MAVEN_CLI_OPTS deploy

docker-harbor-build:
 image: docker:19.03
 stage: package
 retry: 2
 services:
 - name: docker:19.03-dind
 alias: docker
 before_script:
 - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
 script:
 - docker build --pull -t "$CI_REGISTRY_IMAGE:$IMAGE_VERSION" .
 - docker push "$CI_REGISTRY_IMAGE:$IMAGE_VERSION"
 - docker logout $CI_REGISTRY

deploy_live:
 image: fjy8018/kubectl:v1.14.0
 stage: deploy
 retry: 2
 environment:
 name: prod
 url: https://XXXX
 script:
 - kubectl version
 - kubectl get pods -n hmdt
 - cd manifests/
 - sed -i "s/__IMAGE_VERSION_SLUG__/${IMAGE_VERSION}/" deployment.yaml
 - kubectl apply -f deployment.yaml
 - kubectl rollout status -f deployment.yaml
 - kubectl get pods -n hmdt

若需要使用Kaniko构建Dockerfile，则配置如下

注意，其中依赖的镜像gcr.io/kaniko-project/executor:debug属于谷歌镜像仓库，可能存在无法拉取的情况

image: docker:19.03

variables:
 MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode -Dmaven.test.skip=true"
 MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
 DOCKER_DRIVER: overlay
 DOCKER_HOST: tcp://localhost:2375
 DOCKER_TLS_CERTDIR: ""
 SPRING_PROFILES_ACTIVE: docker
 IMAGE_VERSION: "1.8.6"
 DOCKER_REGISTRY_MIRROR: "https://XXX.mirror.aliyuncs.com"

cache:
 paths:
 - target/

stages:
 - test
 - package
 - review
 - deploy

maven-build:
 image: maven:3-jdk-8
 stage: test
 retry: 2
 script:
 - mvn $MAVEN_CLI_OPTS clean package -U -B -T 2C
 artifacts:
 expire_in: 1 week
 paths:
 - target/*.jar

maven-scan:
 stage: test
 retry: 2
 image: maven:3-jdk-8
 script:
 - mvn $MAVEN_CLI_OPTS verify sonar:sonar

maven-deploy:
 stage: deploy
 retry: 2
 image: maven:3-jdk-8
 script:
 - mvn $MAVEN_CLI_OPTS deploy


docker-harbor-build:
 stage: package
 retry: 2
 image:
 name: gcr.io/kaniko-project/executor:debug
 entrypoint: [""]
 script:
 - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
 - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$IMAGE_VERSION

deploy_live:
 image: fjy8018/kubectl:v1.14.0
 stage: deploy
 retry: 2
 environment:
 name: prod
 url: https://XXXX
 script:
 - kubectl version
 - kubectl get pods -n hmdt
 - cd manifests/
 - sed -i "s/__IMAGE_VERSION_SLUG__/${IMAGE_VERSION}/" deployment.yaml
 - kubectl apply -f deployment.yaml
 - kubectl rollout status -f deployment.yaml
 - kubectl get pods -n hmdt

执行流水线

runner自动扩缩容

Kubernetes中的runner会根据任务多少自动扩缩容，目前配置的上限为10个

Grafana也能监控到集群在构建过程中的资源使用情况

使用`DinD`构建`Dockerfile`结果

使用`Kaniko`构建`Dockerfile`的结果

部署结果

执行部署时gitlab会自动注入配置好的kubectl config

构建结果

部署完成后可在环境配置页中查看部署结果，只有成功的部署才会被记录

k8s共享存储以及Redis高可用集群快速搭建

Wed, 20 Nov 2019 15:55:00 +0800

介绍

StorageClass 为管理员提供了描述存储 "类" 的方法。不同的类型可能会映射到不同的服务质量等级或备份策略，通常由运维制定的任意策略

用户通过在 PersistentVolumeClaim 中包含存储类来请求动态供应的存储，动态卷供应的实现基于 storage.k8s.io API 组中的 StorageClass API 对象。集群管理员可以根据需要定义多个 StorageClass 对象，每个对象指定一个卷插件（又名 provisioner），卷插件向卷供应商提供在创建卷时需要的数据卷信息及相关参数。

环境

Kubernetes：v1.14.0

Docker：17.03.1-ce

机器数：$3 \times master + 2 \times worker$

其中，master3、worker1、worker2都挂载了2块磁盘，非系统盘为纯净未分区磁盘，用于提供给GlusterFs管理

安装GlusterFs

在所有可能被调度到pod的节点安装glusterfs

$ yum install -y glusterfs glusterfs-fuse net-tools

启动GlusterFs与heketi

使用yaml文件启动glusterfs及其操作工具heketi

glusterfs-daemonset.yaml

---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
 name: glusterfs
 labels:
 glusterfs: daemonset
 annotations:
 description: GlusterFS DaemonSet
 tags: glusterfs
spec:
 template:
 metadata:
 name: glusterfs
 labels:
 glusterfs: pod
 glusterfs-node: pod
 spec:
 nodeSelector:
 storagenode: glusterfs
 hostNetwork: true
 containers:
 - image: gluster/gluster-centos:latest
 imagePullPolicy: IfNotPresent
 name: glusterfs
 env:
 # alternative for /dev volumeMount to enable access to *all* devices
 - name: HOST_DEV_DIR
 value: "/mnt/host-dev"
 # set GLUSTER_BLOCKD_STATUS_PROBE_ENABLE to "1" so the
 # readiness/liveness probe validate gluster-blockd as well
 - name: GLUSTER_BLOCKD_STATUS_PROBE_ENABLE
 value: "1"
 - name: GB_GLFS_LRU_COUNT
 value: "15"
 - name: TCMU_LOGDIR
 value: "/var/log/glusterfs/gluster-block"
 resources:
 requests:
 memory: 100Mi
 cpu: 100m
 volumeMounts:
 - name: glusterfs-heketi
 mountPath: "/var/lib/heketi"
 - name: glusterfs-run
 mountPath: "/run"
 - name: glusterfs-lvm
 mountPath: "/run/lvm"
 - name: glusterfs-etc
 mountPath: "/etc/glusterfs"
 - name: glusterfs-logs
 mountPath: "/var/log/glusterfs"
 - name: glusterfs-config
 mountPath: "/var/lib/glusterd"
 - name: glusterfs-host-dev
 mountPath: "/mnt/host-dev"
 - name: glusterfs-misc
 mountPath: "/var/lib/misc/glusterfsd"
 - name: glusterfs-block-sys-class
 mountPath: "/sys/class"
 - name: glusterfs-block-sys-module
 mountPath: "/sys/module"
 - name: glusterfs-cgroup
 mountPath: "/sys/fs/cgroup"
 readOnly: true
 - name: glusterfs-ssl
 mountPath: "/etc/ssl"
 readOnly: true
 - name: kernel-modules
 mountPath: "/usr/lib/modules"
 readOnly: true
 securityContext:
 capabilities: {}
 privileged: true
 readinessProbe:
 timeoutSeconds: 3
 initialDelaySeconds: 40
 exec:
 command:
 - "/bin/bash"
 - "-c"
 - "if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh readiness; else systemctl status glusterd.service; fi"
 periodSeconds: 25
 successThreshold: 1
 failureThreshold: 50
 livenessProbe:
 timeoutSeconds: 3
 initialDelaySeconds: 40
 exec:
 command:
 - "/bin/bash"
 - "-c"
 - "if command -v /usr/local/bin/status-probe.sh; then /usr/local/bin/status-probe.sh liveness; else systemctl status glusterd.service; fi"
 periodSeconds: 25
 successThreshold: 1
 failureThreshold: 50
 volumes:
 - name: glusterfs-heketi
 hostPath:
 path: "/var/lib/heketi"
 - name: glusterfs-run
 - name: glusterfs-lvm
 hostPath:
 path: "/run/lvm"
 - name: glusterfs-etc
 hostPath:
 path: "/etc/glusterfs"
 - name: glusterfs-logs
 hostPath:
 path: "/var/log/glusterfs"
 - name: glusterfs-config
 hostPath:
 path: "/var/lib/glusterd"
 - name: glusterfs-host-dev
 hostPath:
 path: "/dev"
 - name: glusterfs-misc
 hostPath:
 path: "/var/lib/misc/glusterfsd"
 - name: glusterfs-block-sys-class
 hostPath:
 path: "/sys/class"
 - name: glusterfs-block-sys-module
 hostPath:
 path: "/sys/module"
 - name: glusterfs-cgroup
 hostPath:
 path: "/sys/fs/cgroup"
 - name: glusterfs-ssl
 hostPath:
 path: "/etc/ssl"
 - name: kernel-modules
 hostPath:
 path: "/usr/lib/modules"

heketi-security.yaml 用户与权限相关

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: heketi-clusterrolebinding
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: heketi-clusterrole
subjects:
- kind: ServiceAccount
 name: heketi-service-account
 namespace: default

---

apiVersion: v1
kind: ServiceAccount
metadata:
 name: heketi-service-account
 namespace: default

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: heketi-clusterrole
rules:
- apiGroups:
 - ""
 resources:
 - pods
 - pods/status
 - pods/exec
 verbs:
 - get
 - list
 - watch
 - create

heketi-deployment.yaml 启动服务并使用ingress四层反向代理

kind: Service
apiVersion: v1
metadata:
 name: heketi
 labels: 
 glusterfs: heketi-service
 deploy-heketi: support
 annotations: 
 description: Exposes Heketi Service
spec:
 selector: 
 name: heketi
 ports:
 - name: heketi
 port: 80
 targetPort: 8080

---

apiVersion: v1
kind: ConfigMap
metadata:
 name: tcp-services
 namespace: ingress-nginx
data:
 "30001": default/heketi:80

---

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
 name: heketi
 labels: 
 glusterfs: heketi-deployment
 annotations: 
 description: Defines how to deploy Heketi
spec:
 replicas: 1
 template:
 metadata:
 name: heketi
 labels: 
 name: heketi
 glusterfs: heketi-pod
 spec:
 serviceAccountName: heketi-service-account
 containers:
 - image: heketi/heketi:6
 imagePullPolicy: Always
 name: heketi
 env:
 - name: HEKETI_EXECUTOR
 value: "kubernetes"
 - name: HEKETI_DB_PATH
 value: "/var/lib/heketi/heketi.db"
 - name: HEKETI_FSTAB
 value: "/var/lib/heketi/fstab"
 - name: HEKETI_SNAPSHOT_LIMIT
 value: "14"
 - name: HEKETI_KUBE_GLUSTER_DAEMONSET
 value: "y"
 ports:
 - containerPort: 8080
 volumeMounts:
 - name: db
 mountPath: /var/lib/heketi
 readinessProbe:
 timeoutSeconds: 3
 initialDelaySeconds: 3
 httpGet: 
 path: /hello
 port: 8080
 livenessProbe:
 timeoutSeconds: 3
 initialDelaySeconds: 30
 httpGet: 
 path: /hello
 port: 8080
 volumes:
 - name: db
 hostPath:
 path: "/heketi-data"

GlusterFS初始化

创建

使用heketi-cli初始化GlusterFS节点，找到heketi容器并进入

$ docker ps | grep hek
9c92cf25e4bb heketi/heketi@sha256:44058f20a91211234abc533c9b4b5c42764b8cd740e21d03377bee14ebca4497 "/usr/bin/heketi-s..." About a minute ago Up 58 seconds k8s_he eti_heketi-68f9dfdfbf-w4qrd_default_c0566bf2-09ff-11ea-b6ce-000c298872e2_0
ebafb92c4dc1 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" About a minute ago Up About a minute k8s_POD_heketi-68f9dfdfbf-w4qrd_default_c0566bf2-09ff-11ea-b6ce-000c298872e2_0
$ docker exec -it 9c92 /bin/bash

配置容器本地环境变量，并配置节点json文件

$ export HEKETI_CLI_SERVER=http://localhost:8080
$ vi topology.json

主机名需要唯一并可以访问

{
 "clusters": [
 {
 "nodes": [
 {
 "node": {
 "hostnames": {
 "manage": [
 "k8smaster3"
 ],
 "storage": [
 "192.168.1.17"
 ]
 },
 "zone": 1
 },
 "devices": ["/dev/sdb"]
 },
 {
 "node": {
 "hostnames": {
 "manage": [
 "k8sworker1"
 ],
 "storage": [
 "192.168.1.18"
 ]
 },
 "zone": 1
 },
 "devices": ["/dev/sdb"]
 },
 {
 "node": {
 "hostnames": {
 "manage": [
 "k8sworker2"
 ],
 "storage": [
 "192.168.1.19"
 ]
 },
 "zone": 1
 },
 "devices": ["/dev/sdb"]
 }
 ]
 }
 ]
}

装载json文件

$ heketi-cli topology load --json topology.json
 Found node k8smaster3 on cluster 84d8a3a128c43a2a7d6c36aa6833823f
 Adding device /dev/sdb ... OK
 Found node k8sworker1 on cluster 84d8a3a128c43a2a7d6c36aa6833823f
 Adding device /dev/sdb ... OK
 Found node k8sworker2 on cluster 84d8a3a128c43a2a7d6c36aa6833823f
 Adding device /dev/sdb ... OK

可能出现的问题

1. 无法初始化

报错信息为

Creating cluster ... ID: 84d8a3a128c43a2a7d6c36aa6833823f
 Allowing file volumes on cluster.
 Allowing block volumes on cluster.
 Creating node k8smaster3 ... ID: b97a53e296850d59bbdd4043b13ebf78
 Adding device /dev/sdb ... Unable to add device: Unable to execute command on glusterfs-4vgh8: WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/centos/root not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/sda1 not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/centos/swap not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/sda2 not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds.
 Can't initialize physical volume "/dev/sdb" of volume group "vg_9bc3bdaff53104647cd24abe9ca60745" without -ff
 /dev/sdb: physical volume not initialized.
 Creating node k8sworker1 ... ID: 3a57327dbd24d9aedcae01f72857ef5e
 Adding device /dev/sdb ... Unable to add device: Unable to execute command on glusterfs-wzcf8: WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/centos/root not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/sda1 not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/centos/swap not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/sda2 not initialized in udev database even after waiting 10000000 microseconds.
 WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds.
 Can't initialize physical volume "/dev/sdb" of volume group "vg_c6a688843cd06e333c83e56814c95c55" without -ff
 /dev/sdb: physical volume not initialized.
 Creating node k8sworker2 ... ID: 141f497c0797b26a22a31288ca97acf1

通过查阅资料，发现如下issue，官方提供磁盘擦除命令，执行后重新初始化节点

deploy fails using k8s 1.7.1 when heketi tries to add devices to cluster #2 #279

Destroy the heketi and GlusterFS pods, then try the following on each of the nodes:
pvdisplay -C -o pv_name,vg_name # look for VGs on PVs on your target devices
lvremove -ff <any VGs found above>
vgremove -f <any VGs found above>
pvremove -f <any PVs found above>
wipefs -a <target device>
Then try gk-deploy from the beginning again. Naturally, double-check that you’re definitely removing something only on the target devices. :)

进入挂载磁盘节点，查看当前磁盘信息

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 40G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 39G 0 part
 ├─centos-root 253:0 0 35G 0 lvm /
 └─centos-swap 253:1 0 4G 0 lvm
sdb 8:16 0 40G 0 disk
sr0 11:0 1 1024M 0 rom

查看pvs名称

$ pvs
 PV VG Fmt Attr PSize PFree
 /dev/sda2 centos lvm2 a-- <39.00g 0
 /dev/sdb vg_c6a688843cd06e333c83e56814c95c55 lvm2 a-- 39.87g 39.87g

抹除磁盘

$ pvdisplay -C -o /dev/sdb,vg_c6a688843cd06e333c83e56814c95c55
$ lvremove -ff vg_c6a688843cd06e333c83e56814c95c55
$ vgremove -f vg_c6a688843cd06e333c83e56814c95c55
$ pvremove -f /dev/sdb
$ wipefs -a /dev/sdb

抹除后重新进入容器创建即可

验证

进入3个存储节点，检查创建情况，此处以worker1为例

$ netstat -ntlp | grep glusterd
tcp 0 0 0.0.0.0:24007 0.0.0.0:* LISTEN 25997/glusterd
$ docker ps | grep glus
eae6ef1d4a37 22a008905c35 "/usr/local/bin/up..." 31 minutes ago Up 31 minutes k8s_glusterfs_glusterfs-wzcf8_default_940d0f57-09ff-11ea-b6ce-000c298872e2_0
858fd84edbc2 registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 "/pause" 31 minutes ago Up 31 minutes k8s_POD_glusterfs-wzcf8_default_940d0f57-09ff-11ea-b6ce-000c298872e2_0
$ docker exec -it eae6 /bin/bash
$ gluster pool list
UUID Hostname State
6d4eda3d-a7c2-4e27-999c-972106c54c3d k8smaster3 Connected
56a6e1d4-46db-4ff2-83eb-39c313bc9197 192.168.1.19 Connected
a8ac96bd-b596-433c-8e2a-e3a97873886b localhost Connected

创建storage-class

定义StorageClass，通过访问ingress-nginx四层代理暴露的端口管理glusterfs

glusterfs-storage-class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: glusterfs-storage-class
provisioner: kubernetes.io/glusterfs
parameters:
 resturl: "http://192.168.1.16:30001"
 restauthenabled: "false"

创建PVC

创建PVC，可以被单个节点映射为ReadWriteOnce,或者多个节点映射为ReadOnlyMany,但不能同时使用这两种方式来映射。

glusterfs-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: glusterfs-pvc
spec:
 storageClassName: glusterfs-storage-class
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi

检查pvc，可见已经是已绑定(Bound)状态

$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
glusterfs-pvc Bound pvc-7c4f94cd-0a04-11ea-b6ce-000c298872e2 1Gi RWO glusterfs-storage-class 128m

基于Helm安装Redis-HA

访问charts官方仓库：https://github.com/helm/charts

在stable目录下找到redis-ha目录，修改values.yaml文件如下，主要配置存储类为storageClass: glusterfs-storage-class，然后修改用户名和密码等相关信息即可

image:
 repository: redis
 tag: 5.0.5-alpine
 pullPolicy: IfNotPresent
replicas: 3
labels: {}
configure-service-account/
serviceAccount:
 create: truetemplate
Automatically proxies to Redis master.
haproxy:
 enabled: false
 readOnly:
 enabled: false
 port: 6380
 replicas: 1
 image:
 repository: haproxy
 tag: 2.0.4
 pullPolicy: IfNotPresent
 annotations: {}
 resources: {}
 service:
 type: ClusterIP
 loadBalancerIP:
 annotations: {}
 serviceAccount:
 create: true
 exporter:
 image:
 repository: quay.azk8s.cn/prometheus/haproxy-exporter
 tag: v0.9.0
 enabled: false
 port: 9101
 init:
 resources: {}
 timeout:
 connect: 4s
 server: 30s
 client: 30s
 securityContext:
 runAsUser: 1000
 fsGroup: 1000
 runAsNonRoot: true
rbac:
 create: true
sysctlImage:
 enabled: false
 command: []
 registry: docker.io
 repository: busybox
 tag: 1.31.1
 pullPolicy: Always
 mountHostSys: false
configure-multiple-schedulers/
redis:
 port: 6379
 masterGroupName: mymaster
 config:
 min-replicas-to-write: 1unlimited.instance. Default is volatile-lru.with a new slave. The only way to prevent this is to enable diskless replication.
 save: "900 1"the disk as intermediate storage. Default is false.
 repl-diskless-sync: "yes"
 rdbcompression: "yes"
 rdbchecksum: "yes"
 resources: {}

sentinel:
 port: 26379
 quorum: 2
 config:
 down-after-milliseconds: 10000
 failover-timeout: 180000
 parallel-syncs: 5
 resources: {}

securityContext:
 runAsUser: 1000
 fsGroup: 1000
 runAsNonRoot: true
assign-pod-node/#nodeselectorassign-pod-node/#taints-and-tolerations-beta-featureassign-pod-node/#affinity-and-anti-affinity
nodeSelector: {}
requiredDuringSchedulingIgnoredDuringExecution as opposed to preferred.assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature
hardAntiAffinity: true
additionalAffinities: {}
affinity: |
exporter:
 enabled: true
 image: oliver006/redis_exporter
 tag: v0.31.0
 pullPolicy: IfNotPresent
 port: 9121
 scrapePath: /metrics
 resources: {}
 extraArgs: {}
 serviceMonitor:
 enabled: falsePrometheus Operator
podDisruptionBudget: {}
auth: true
redisPassword: password
authKey: auth
persistentVolume:
 enabled: true
 storageClass: glusterfs-storage-class
 accessModes:
 - ReadWriteOnce
 size: 100Mi
 annotations: {}
init:
 resources: {}

hostPath:
 chown: true

从修改后的本地文件安装redis-ha

$ helm install /root/redis-ha/ --name k8s-redis-ha --namespace env-prod
NAME: k8s-redis-ha
LAST DEPLOYED: Mon Nov 18 22:13:20 2019
NAMESPACE: env-prod
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
k8s-redis-ha-configmap 4 1s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
k8s-redis-ha-server-0 0/3 Pending 0 0s

==> v1/Role
NAME AGE
k8s-redis-ha 1s

==> v1/RoleBinding
NAME AGE
k8s-redis-ha 1s

==> v1/Secret
NAME TYPE DATA AGE
k8s-redis-ha Opaque 1 1s

==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
k8s-redis-ha ClusterIP None <none> 6379/TCP,26379/TCP,9121/TCP 1s
k8s-redis-ha-announce-0 ClusterIP 10.101.55.132 <none> 6379/TCP,26379/TCP,9121/TCP 1s
k8s-redis-ha-announce-1 ClusterIP 10.98.159.185 <none> 6379/TCP,26379/TCP,9121/TCP 1s
k8s-redis-ha-announce-2 ClusterIP 10.102.215.171 <none> 6379/TCP,26379/TCP,9121/TCP 1s

==> v1/ServiceAccount
NAME SECRETS AGE
k8s-redis-ha 1 1s

==> v1/StatefulSet
NAME READY AGE
k8s-redis-ha-server 0/3 1s


NOTES:
Redis can be accessed via port 6379 and Sentinel can be accessed via port 26379 on the following DNS name from within your cluster:
k8s-redis-ha.env-prod.svc.cluster.local

To connect to your Redis server:
1. To retrieve the redis password:
 echo $(kubectl get secret k8s-redis-ha -o "jsonpath={.data['auth']}" | base64 --decode)

2. Connect to the Redis master pod that you can use as a client. By default the k8s-redis-ha-server-0 pod is configured as the master:

 kubectl exec -it k8s-redis-ha-server-0 sh -n env-prod

3. Connect using the Redis CLI (inside container):

 redis-cli -a <REDIS-PASS-FROM-SECRET>

检查pvc创建情况

$ kubectl get pvc -n env-prod
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-k8s-redis-ha-server-0 Bound pvc-9348a01f-0a0d-11ea-b6ce-000c298872e2 1Gi RWO glusterfs-storage-class 62m
data-k8s-redis-ha-server-1 Bound pvc-9a8defcf-0a0d-11ea-b6ce-000c298872e2 1Gi RWO glusterfs-storage-class 62m
data-k8s-redis-ha-server-2 Bound pvc-a1975bf8-0a0d-11ea-b6ce-000c298872e2 1Gi RWO glusterfs-storage-class 62m

检查pv

$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-7c4f94cd-0a04-11ea-b6ce-000c298872e2 1Gi RWO Delete Bound default/glusterfs-pvc glusterfs-storage-class 128m
pvc-9348a01f-0a0d-11ea-b6ce-000c298872e2 1Gi RWO Delete Bound env-prod/data-k8s-redis-ha-server-0 glusterfs-storage-class 63m
pvc-9a8defcf-0a0d-11ea-b6ce-000c298872e2 1Gi RWO Delete Bound env-prod/data-k8s-redis-ha-server-1 glusterfs-storage-class 63m
pvc-a1975bf8-0a0d-11ea-b6ce-000c298872e2 1Gi RWO Delete Bound env-prod/data-k8s-redis-ha-server-2 glusterfs-storage-class 63m

配置ingress-nginx四层反向代理

apiVersion: v1
kind: ConfigMap
metadata:
 name: tcp-services
 namespace: ingress-nginx
data:
 "30001": default/heketi:80
 "30002": env-prod/k8s-redis-ha:6379
 "30003": env-prod/k8s-redis-ha:26379

使用redis客户端连接即可

k8s监控平台搭建

Tue, 12 Nov 2019 21:33:00 +0800

架构

目前主流容器化监控平台采用Prometheus+Grafana架构，下图为Prometheus架构图，可以主动发现各种指标和采集数据，实现

监控每个节点基础指标
监控每个容器基础指标
监控k8s集群组件基础指标

基础环境

Kubernetes：v1.14.0

Docker：17.03.1-ce

机器数：$3 \times master + 2 \times worker$

部署

Helm

Helm作为k8s平台软件包管理工具，结合prometheus-operator可以类似插件的方式快速搭建监控体系，Helm旗下包含charts项目，为软件包仓库

Helm官方仓库：https://github.com/helm/helm

charts官方仓库：https://github.com/helm/charts

安装Helm

通过官方仓库查看目前最新版为v2.16.0，但由于仍有不稳定的情况，故不推荐该版本，原因可通过其issue查看

此处选择安装v2.15.2的版本，下载页面

解压并放到可执行目录下

$ tar -zxvf helm-v2.15.2-linux-amd64.tar.gz
$ mv linux-amd64/helm /usr/local/bin/

Tiller安装

Tiller 是以 Deployment 方式部署在 Kubernetes 集群中的，由于国内网络原因，需要配置为阿里云镜像

$ helm init --client-only --stable-repo-url https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts/
$ helm repo add incubator https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts-incubator/
$ helm repo update

因为官方的镜像无法拉取，使用-i指定自己的镜像

$ helm init --service-account tiller --upgrade -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.15.2 --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

创建TLS认证服务端

$ helm init --service-account tiller --upgrade -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.15.2 --tiller-tls-cert /etc/kubernetes/ssl/tiller001.pem --tiller-tls-key /etc/kubernetes/ssl/tiller001-key.pem --tls-ca-cert /etc/kubernetes/ssl/ca.pem --tiller-namespace kube-system --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

Tiller授权

Tiller 需要访问ApiServer去对集群进行操作。目前的 Tiller 部署时默认没有定义授权的 ServiceAccount，这会导致访问 API Server 时被拒绝。所以我们需要明确为 Tiller 部署添加授权

创建ServiceAccount和创建角色绑定

$ kubectl create serviceaccount --namespace kube-system tiller
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

验证

查看Tiller的ServiceAccount，需要跟创建的名字一致：tiller

$ kubectl get deploy --namespace kube-system tiller-deploy -o yaml|grep serviceAccount
 serviceAccount: tiller
 serviceAccountName: tiller

验证pods

$ kubectl -n kube-system get pods|grep tiller
tiller-deploy-b6c45d5bc-jhntj 1/1 Running 0 2d8h

验证版本

$ helm version
Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}

Operator

可以自定义资源类型(CDR)结合自定义控制器实现

样例

apiVersion: app.example.com/v1
kind: AppService
metadata:
 name: nginx-app
spec:
 size: 2
 image: nginx:1.7.9
 ports:
 - port: 80
 targetPort: 80
 nodePort: 30002

Charts

charts项目下stable目录里存放所有软件包内容，由于其默认从多处镜像仓库中拉取镜像，可能存在网络问题，需结合实际情况修改镜像配置，手动安装软件包

推荐配置地址：

quay.io 替换为 quay.azk8s.cn
gcr.io/google-containers 替换为 registry.cn-hangzhou.aliyuncs.com/google-containers

本次需要安装的软件包为prometheus-operator，但其还包含多个依赖软件如下

grafana
kube-state-metrics
prometheus-node-exporter

若需要手动安装则需要按照整理目录如下

prometheus-operator/
├── charts
│   ├── grafana
│   ├── kube-state-metrics
│   └── prometheus-node-exporter
...

手动安装

Etcd监控配置

监控etcd需要额外配置，而且较为复杂，具体步骤如下

查看etcd在集群中的配置，由于本k8s集群基于kubeadm搭建，故etcd通过容器方式运行，查看方法

# 查看etcd所在pod名称
$ kubectl get pod --all-namespaces | grep etcd
kube-system etcd-k8smaster1 1/1 Running 0 7d1h
# 展开etcd详细配置
$ kubectl get pod -n kube-system etcd-k8smaster1 -o yaml

配置内容查看其健康检查端点，--listen-client-urls指明其监听在https端点，故需要配置安全证书才能访问

apiVersion: v1
kind: Pod
metadata:
 annotations:
 kubernetes.io/config.hash: 6119961323d801d05a7dd23e429cda3f
 kubernetes.io/config.mirror: 6119961323d801d05a7dd23e429cda3f
 kubernetes.io/config.seen: "2019-11-05T19:17:28.480984831+08:00"
 kubernetes.io/config.source: file
 creationTimestamp: "2019-11-05T11:18:53Z"
 labels:
 component: etcd
 tier: control-plane
 name: etcd-k8smaster1
 namespace: kube-system
 resourceVersion: "876368"
 selfLink: /api/v1/namespaces/kube-system/pods/etcd-k8smaster1
 uid: 0ccf5e91-ffbe-11e9-a5d2-000c298872e2
spec:
 containers:
 - command:
 - etcd
 - --advertise-client-urls=https://192.168.1.15:2379
 - --cert-file=/etc/kubernetes/pki/etcd/server.crt
 - --client-cert-auth=true
 - --data-dir=/var/lib/etcd
 - --initial-advertise-peer-urls=https://192.168.1.15:2380
 - --initial-cluster=k8smaster1=https://192.168.1.15:2380
 - --key-file=/etc/kubernetes/pki/etcd/server.key
 - --listen-client-urls=https://127.0.0.1:2379,https://192.168.1.15:2379
 - --listen-peer-urls=https://192.168.1.15:2380
 - --name=k8smaster1
 - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
 - --peer-client-cert-auth=true
 - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
 - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
 - --snapshot-count=10000
 - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
 image: registry.cn-hangzhou.aliyuncs.com/imooc/etcd:3.3.10
 imagePullPolicy: IfNotPresent
 livenessProbe:
...
hostIP: 192.168.1.15
...

配置安全证书

k8s创建etcd时会自动创建相关证书，当前集群版本为1.14，证书位于/etc/kubernetes/pki/etcd/中

$ ls /etc/kubernetes/pki/etcd/
ca.crt ca.key healthcheck-client.crt healthcheck-client.key peer.crt peer.key server.crt server.key

可以通过curl命令加载证书测试能否访问健康检查端点

$ curl -k --cert /etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key https://192.168.1.15:2379/metrics

若有如下输出则正常

# HELP etcd_debugging_mvcc_db_compaction_keys_total Total number of db keys compacted.
# TYPE etcd_debugging_mvcc_db_compaction_keys_total counter
etcd_debugging_mvcc_db_compaction_keys_total 1.030132e+06
# HELP etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds Bucketed histogram of db compaction pause duration.
# TYPE etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds histogram
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="1"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="2"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="4"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="8"} 0
...

创建证书秘钥

关联三个证书文件，创建k8s访问秘钥，秘钥名称为etcd-certs

$ kubectl create secret generic etcd-certs -n monitoring --from-file=/etc/kubernetes/pki/etcd/ca.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key

写入prometheus配置文件values.yaml

注意，此处路径前缀固定写法为/etc/prometheus/secrets/+秘钥名称+文件名称，若写错证书路径则无法在指标中查看到对应的target

... 
 serviceMonitor:
 ## Scrape interval. If not set, the Prometheus default scrape interval is used.
 ##
 interval: ""
 scheme: https
 insecureSkipVerify: true
 serverName: ""
 caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
 certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt
 keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key
...

创建

指定命名空间和文件目录，若命名空间不存在会自动创建

$ helm install /root/prometheus-operator/ --name k8s-prometheus --namespace monitoring

等待一段时间，输出如下则创建成功

NAME: k8s-prometheus
LAST DEPLOYED: Tue Nov 12 20:27:18 2019
NAMESPACE: monitoring
STATUS: DEPLOYED

RESOURCES:
==> v1/Alertmanager
NAME AGE
k8s-prometheus-prometheus-alertmanager 33s

==> v1/ClusterRole
NAME AGE
k8s-prometheus-grafana-clusterrole 34s
k8s-prometheus-prometheus-alertmanager 34s
k8s-prometheus-prometheus-operator 34s
k8s-prometheus-prometheus-operator-psp 34s
k8s-prometheus-prometheus-prometheus 34s
k8s-prometheus-prometheus-prometheus-psp 34s
psp-k8s-prometheus-kube-state-metrics 34s
psp-k8s-prometheus-prometheus-node-exporter 34s

==> v1/ClusterRoleBinding
NAME AGE
k8s-prometheus-grafana-clusterrolebinding 34s
k8s-prometheus-prometheus-alertmanager 34s
k8s-prometheus-prometheus-operator 34s
k8s-prometheus-prometheus-operator-psp 34s
k8s-prometheus-prometheus-prometheus 34s
k8s-prometheus-prometheus-prometheus-psp 34s
psp-k8s-prometheus-kube-state-metrics 34s
psp-k8s-prometheus-prometheus-node-exporter 34s

==> v1/ConfigMap
NAME DATA AGE
k8s-prometheus-grafana 1 34s
k8s-prometheus-grafana-config-dashboards 1 34s
k8s-prometheus-grafana-test 1 34s
k8s-prometheus-prometheus-apiserver 1 34s
k8s-prometheus-prometheus-cluster-total 1 34s
k8s-prometheus-prometheus-controller-manager 1 34s
k8s-prometheus-prometheus-etcd 1 34s
k8s-prometheus-prometheus-grafana-datasource 1 34s
k8s-prometheus-prometheus-k8s-coredns 1 34s
k8s-prometheus-prometheus-k8s-resources-cluster 1 34s
k8s-prometheus-prometheus-k8s-resources-namespace 1 34s
k8s-prometheus-prometheus-k8s-resources-node 1 34s
k8s-prometheus-prometheus-k8s-resources-pod 1 34s
k8s-prometheus-prometheus-k8s-resources-workload 1 34s
k8s-prometheus-prometheus-k8s-resources-workloads-namespace 1 34s
k8s-prometheus-prometheus-kubelet 1 34s
k8s-prometheus-prometheus-namespace-by-pod 1 34s
k8s-prometheus-prometheus-namespace-by-workload 1 34s
k8s-prometheus-prometheus-node-cluster-rsrc-use 1 34s
k8s-prometheus-prometheus-node-rsrc-use 1 34s
k8s-prometheus-prometheus-nodes 1 34s
k8s-prometheus-prometheus-persistentvolumesusage 1 34s
k8s-prometheus-prometheus-pod-total 1 34s
k8s-prometheus-prometheus-pods 1 34s
k8s-prometheus-prometheus-prometheus 1 34s
k8s-prometheus-prometheus-proxy 1 34s
k8s-prometheus-prometheus-scheduler 1 34s
k8s-prometheus-prometheus-statefulset 1 34s
k8s-prometheus-prometheus-workload-total 1 34s

==> v1/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
k8s-prometheus-prometheus-node-exporter 5 5 5 5 5 <none> 33s

==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
k8s-prometheus-grafana 1/1 1 1 33s
k8s-prometheus-kube-state-metrics 1/1 1 1 33s
k8s-prometheus-prometheus-operator 1/1 1 1 33s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
k8s-prometheus-grafana-769d6dfb69-mtbdg 2/2 Running 0 33s
k8s-prometheus-kube-state-metrics-76b8ff78cd-kjxvl 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-jqkvf 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-jxpzm 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-k8n6x 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-kpjzk 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-qc6kv 1/1 Running 0 33s
k8s-prometheus-prometheus-operator-5f894bc75d-kvkvs 2/2 Running 0 33s

==> v1/Prometheus
NAME AGE
k8s-prometheus-prometheus-prometheus 33s

==> v1/PrometheusRule
NAME AGE
k8s-prometheus-prometheus-alertmanager.rules 32s
k8s-prometheus-prometheus-etcd 32s
k8s-prometheus-prometheus-general.rules 32s
k8s-prometheus-prometheus-k8s.rules 32s
k8s-prometheus-prometheus-kube-apiserver.rules 32s
k8s-prometheus-prometheus-kube-prometheus-node-recording.rules 32s
k8s-prometheus-prometheus-kube-scheduler.rules 32s
k8s-prometheus-prometheus-kubernetes-absent 32s
k8s-prometheus-prometheus-kubernetes-apps 32s
k8s-prometheus-prometheus-kubernetes-resources 32s
k8s-prometheus-prometheus-kubernetes-storage 32s
k8s-prometheus-prometheus-kubernetes-system 32s
k8s-prometheus-prometheus-kubernetes-system-apiserver 32s
k8s-prometheus-prometheus-kubernetes-system-controller-manager 32s
k8s-prometheus-prometheus-kubernetes-system-kubelet 32s
k8s-prometheus-prometheus-kubernetes-system-scheduler 32s
k8s-prometheus-prometheus-node-exporter 32s
k8s-prometheus-prometheus-node-exporter.rules 32s
k8s-prometheus-prometheus-node-network 32s
k8s-prometheus-prometheus-node-time 32s
k8s-prometheus-prometheus-node.rules 32s
k8s-prometheus-prometheus-prometheus 32s
k8s-prometheus-prometheus-prometheus-operator 32s

==> v1/Role
NAME AGE
k8s-prometheus-grafana-test 34s

==> v1/RoleBinding
NAME AGE
k8s-prometheus-grafana-test 34s

==> v1/Secret
NAME TYPE DATA AGE
alertmanager-k8s-prometheus-prometheus-alertmanager Opaque 1 34s
k8s-prometheus-grafana Opaque 3 34s

==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
k8s-prometheus-grafana ClusterIP 10.100.203.67 <none> 80/TCP 33s
k8s-prometheus-kube-state-metrics ClusterIP 10.98.35.173 <none> 8080/TCP 33s
k8s-prometheus-prometheus-alertmanager ClusterIP 10.97.189.252 <none> 9093/TCP 33s
k8s-prometheus-prometheus-coredns ClusterIP None <none> 9153/TCP 34s
k8s-prometheus-prometheus-kube-controller-manager ClusterIP None <none> 10252/TCP 34s
k8s-prometheus-prometheus-kube-etcd ClusterIP None <none> 2379/TCP 34s
k8s-prometheus-prometheus-kube-proxy ClusterIP None <none> 10249/TCP 34s
k8s-prometheus-prometheus-kube-scheduler ClusterIP None <none> 10251/TCP 34s
k8s-prometheus-prometheus-node-exporter ClusterIP 10.100.146.95 <none> 9100/TCP 33s
k8s-prometheus-prometheus-operator ClusterIP 10.105.23.58 <none> 8080/TCP,443/TCP 33s
k8s-prometheus-prometheus-prometheus ClusterIP 10.99.7.136 <none> 9090/TCP 34s

==> v1/ServiceAccount
NAME SECRETS AGE
k8s-prometheus-grafana 1 34s
k8s-prometheus-grafana-test 1 34s
k8s-prometheus-kube-state-metrics 1 34s
k8s-prometheus-prometheus-alertmanager 1 34s
k8s-prometheus-prometheus-node-exporter 1 34s
k8s-prometheus-prometheus-operator 1 34s
k8s-prometheus-prometheus-prometheus 1 34s

==> v1/ServiceMonitor
NAME AGE
k8s-prometheus-prometheus-alertmanager 32s
k8s-prometheus-prometheus-apiserver 32s
k8s-prometheus-prometheus-coredns 32s
k8s-prometheus-prometheus-grafana 32s
k8s-prometheus-prometheus-kube-controller-manager 32s
k8s-prometheus-prometheus-kube-etcd 32s
k8s-prometheus-prometheus-kube-proxy 32s
k8s-prometheus-prometheus-kube-scheduler 32s
k8s-prometheus-prometheus-kube-state-metrics 32s
k8s-prometheus-prometheus-kubelet 32s
k8s-prometheus-prometheus-node-exporter 32s
k8s-prometheus-prometheus-operator 32s
k8s-prometheus-prometheus-prometheus 32s

==> v1beta1/ClusterRole
NAME AGE
k8s-prometheus-kube-state-metrics 34s

==> v1beta1/ClusterRoleBinding
NAME AGE
k8s-prometheus-kube-state-metrics 34s

==> v1beta1/MutatingWebhookConfiguration
NAME AGE
k8s-prometheus-prometheus-admission 33s

==> v1beta1/PodSecurityPolicy
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
k8s-prometheus-grafana false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
k8s-prometheus-grafana-test false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,downwardAPI,emptyDir,projected,secret
k8s-prometheus-kube-state-metrics false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false secret
k8s-prometheus-prometheus-alertmanager false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
k8s-prometheus-prometheus-node-exporter false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath
k8s-prometheus-prometheus-operator false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
k8s-prometheus-prometheus-prometheus false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim

==> v1beta1/Role
NAME AGE
k8s-prometheus-grafana 34s

==> v1beta1/RoleBinding
NAME AGE
k8s-prometheus-grafana 34s

==> v1beta1/ValidatingWebhookConfiguration
NAME AGE
k8s-prometheus-prometheus-admission 32s


NOTES:
The Prometheus Operator has been installed. Check its status by running:
 kubectl --namespace monitoring get pods -l "release=k8s-prometheus"

Visit https://github.com/coreos/prometheus-operator for instructions on how
to create & configure Alertmanager and Prometheus instances using the Operator.

升级软件包

一般用于修改yaml配置文件重载应用

$ helm upgrade k8s-prometheus /root/prometheus-operator/ -f /root/prometheus-operator/charts/grafana/values.yaml

删除

使用Helm的优势在于安装软件包和删除都非常方便，通过命令即可快速删除

普通删除，会放入回收站

$ helm delete k8s-prometheus

强制删除，不会放入回收站

$ helm delete k8s-prometheus --purge

对于当前监控组件来说，删除后同时要删除CRD

$ kubectl get crd | grep coreos

删除所有的CRD

$ kubectl delete crd $(kubectl get crd | grep coreos | awk '{print $1}')

访问指标面板

由于当前集群配置了ingress-nginx负载均衡，故新增两个配置文件即可访问Prometheus和Grafana

访问Grafana

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: prom-grafana
 namespace: monitoring
spec:
 rules:
 - host: your.domain
 http:
 paths:
 - backend:
 serviceName: k8s-prometheus-grafana
 servicePort: 80
 path: /

访问Prometheus

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: prometheus
 namespace: monitoring
spec:
 rules:
 - host: your.domain
 http:
 paths:
 - backend:
 serviceName: k8s-prometheus-prometheus-prometheus
 servicePort: web
 path: /

Prometheus

访问Prometheus web站点，首页支持执行PromeQL查询采集指标信息

访问target可见所有指标采集器都处于正常状态

访问监控与告警可查看所有告警规则

Grafana

首次访问会要求登陆

查看用户名和密码

在grafana软件包的values.yaml配置中有如下描述，声明密码被存储在k8s的secret中

# Administrator credentials when not using an existing secret (see below)
adminUser: admin
# adminPassword: strongpassword

# Use an existing secret for the admin user.
admin:
 existingSecret: ""
 userKey: admin-user
 passwordKey: admin-password

查看监控命名空间的秘钥

$ kubectl get secrets -n monitoring
NAME TYPE DATA AGE
alertmanager-k8s-prometheus-prometheus-alertmanager Opaque 1 34m
default-token-hrhgk kubernetes.io/service-account-token 3 2d10h
etcd-certs Opaque 3 2d8h
k8s-prometheus-grafana Opaque 3 34m
k8s-prometheus-grafana-test-token-nnnvk kubernetes.io/service-account-token 3 34m
k8s-prometheus-grafana-token-cdw2n kubernetes.io/service-account-token 3 34m
k8s-prometheus-kube-state-metrics-token-6rxp4 kubernetes.io/service-account-token 3 34m
k8s-prometheus-prometheus-admission Opaque 3 2d10h
k8s-prometheus-prometheus-alertmanager-token-hxqm9 kubernetes.io/service-account-token 3 34m
k8s-prometheus-prometheus-node-exporter-token-vjrcs kubernetes.io/service-account-token 3 34m
k8s-prometheus-prometheus-operator-token-bkhrb kubernetes.io/service-account-token 3 34m
k8s-prometheus-prometheus-prometheus-token-wf2v6 kubernetes.io/service-account-token 3 34m
prometheus-k8s-prometheus-prometheus-prometheus Opaque 1 37m
prometheus-k8s-prometheus-prometheus-prometheus-tls-assets Opaque 0 34m

k8s-prometheus-grafana 即目标秘钥，展示其详情

$ kubectl get secrets -n monitoring k8s-prometheus-grafana -o yaml
apiVersion: v1
data:
 admin-password: b3BlcmF0b3ItYWRtaW4=
 admin-user: cm9vdA==
 ldap-toml: ""
kind: Secret
metadata:
 creationTimestamp: "2019-11-12T12:27:55Z"
 labels:
 app: grafana
 chart: grafana-4.0.2
 heritage: Tiller
 release: k8s-prometheus
 name: k8s-prometheus-grafana
 namespace: monitoring
 resourceVersion: "1033230"
 selfLink: /api/v1/namespaces/monitoring/secrets/k8s-prometheus-grafana
 uid: da7e86d0-0547-11ea-a5d2-000c298872e2
type: Opaque

秘钥默认通过Base64 加密，通过在线网站对该Base64解密即可得到用户名和密码

登录后可查看Grafana已经被配置好的多种可视化面板提供指标展示

点击左上角即可查看已经配置好的指标展示面板

Etcd监控面板

节点监控

计算资源监控

Kubernetes基础命令学习

Sun, 20 Oct 2019 13:00:00 +0800

配置命令语法提示

执行命令查看指南

$ kubectl completion -h
Output shell completion code for the specified shell (bash or zsh). The shell code must be evaluated to provide
interactive completion of kubectl commands. This can be done by sourcing it from the .bash_profile.

 Detailed instructions on how to do this are available here:
https://kubernetes.io/docs/tasks/tools/install-kubectl/#enabling-shell-autocompletion

 Note for zsh users: [1] zsh completions are only supported in versions of zsh >= 5.2

Examples:
 # Installing bash completion on macOS using homebrew
 ## If running Bash 3.2 included with macOS
 brew install bash-completion
 ## or, if running Bash 4.1+
 brew install bash-completion@2
 ## If kubectl is installed via homebrew, this should start working immediately.
 ## If you've installed via other means, you may need add the completion to your completion directory
 kubectl completion bash > $(brew --prefix)/etc/bash_completion.d/kubectl


 # Installing bash completion on Linux
 ## If bash-completion is not installed on Linux, please install the 'bash-completion' package
 ## via your distribution's package manager.
 ## Load the kubectl completion code for bash into the current shell
 source <(kubectl completion bash)
 ## Write bash completion code to a file and source if from .bash_profile
 kubectl completion bash > ~/.kube/completion.bash.inc
 printf "
 # Kubectl shell completion
 source '$HOME/.kube/completion.bash.inc'
 " >> $HOME/.bash_profile
 source $HOME/.bash_profile

 # Load the kubectl completion code for zsh[1] into the current shell
 source <(kubectl completion zsh)
 # Set the kubectl completion code for zsh[1] to autoload on startup
 kubectl completion zsh > "${fpath[1]}/_kubectl"

Usage:
 kubectl completion SHELL [options]

Use "kubectl options" for a list of global command-line options (applies to all commands).

按照指南操作即可

配置多个上下文环境

查看当前上下文配置

$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* minikube minikube minikube

编辑~/.kube/config文件新增需要的上下文配置即可，配置完成后查看上下文信息

$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
 kubeadm kubeadm kubernetes-admin
* minikube minikube minikube

切换上下文

$ kubectl config use-context kubeadm
Switched to context "kubeadm".

再次查看，可见切换成功

$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* kubeadm kubeadm kubernetes-admin
 minikube minikube minikube

Pod相关操作

默认查看`default`命名空间的Pod

$ kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-deployment-54f57cf6bf-pznmh 1/1 Running 0 52s
nginx-deployment-54f57cf6bf-r9q89 1/1 Running 0 52s

查看所有命名空间的Pod

$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58cc8c89f4-c297b 1/1 Running 0 10m
kube-system coredns-58cc8c89f4-spzhj 1/1 Running 0 10m
kube-system etcd-k8s-master 1/1 Running 0 9m57s
kube-system kube-apiserver-k8s-master 1/1 Running 0 10m
kube-system kube-controller-manager-k8s-master 1/1 Running 0 9m58s
kube-system kube-flannel-ds-amd64-6fwg5 1/1 Running 0 5m
kube-system kube-flannel-ds-amd64-6rchg 1/1 Running 0 9m12s
kube-system kube-flannel-ds-amd64-z4plh 1/1 Running 0 4m54s
kube-system kube-proxy-cdcs4 1/1 Running 0 5m
kube-system kube-proxy-nlmbb 1/1 Running 0 4m54s
kube-system kube-proxy-xlb62 1/1 Running 0 10m
kube-system kube-scheduler-k8s-master 1/1 Running 0 10m

指定Pod对应的app名称

$ kubectl get pod -l app=nginx
NAME READY STATUS RESTARTS AGE
nginx-deployment-54f57cf6bf-pznmh 1/1 Running 0 58s
nginx-deployment-54f57cf6bf-r9q89 1/1 Running 0 58s

查看更多信息

$ kubectl get deployment -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
nginx-deployment 2/2 2 2 64s nginx nginx:1.7.9 app=nginx

查看Pod详细信息

$ kubectl describe pod
Name: nginx-deployment-54f57cf6bf-67llv
Namespace: default
Priority: 0
Node: k8s-node1/10.0.2.15
Start Time: Sun, 20 Oct 2019 10:19:42 +0800
Labels: app=nginx
 pod-template-hash=54f57cf6bf
Annotations: <none>
Status: Running
IP: 172.100.1.3
IPs:
 IP: 172.100.1.3
Controlled By: ReplicaSet/nginx-deployment-54f57cf6bf
Containers:
 nginx:
 Container ID: docker://ffd4b4fa3fd4678a61797cfdcba31ea2b0b36459b65a95100484eda4be286b9f
 Image: nginx:1.7.9
 Image ID: docker-pullable://nginx@sha256:e3456c851a152494c3e4ff5fcc26f240206abac0c9d794affb40e0714846c451
 Port: 80/TCP
 Host Port: 0/TCP
 State: Running
 Started: Sun, 20 Oct 2019 10:19:45 +0800
 Ready: True
 Restart Count: 0
 Environment: <none>
 Mounts:
 /var/run/secrets/kubernetes.io/serviceaccount from default-token-zjpq9 (ro)
Conditions:
 Type Status
 Initialized True
 Ready True
 ContainersReady True
 PodScheduled True
Volumes:
 default-token-zjpq9:
 Type: Secret (a volume populated by a Secret)
 SecretName: default-token-zjpq9
 Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Scheduled <unknown> default-scheduler Successfully assigned default/nginx-deployment-54f57cf6bf-67llv to k8s-node1
 Normal Pulled 46s kubelet, k8s-node1 Container image "nginx:1.7.9" already present on machine
 Normal Created 46s kubelet, k8s-node1 Created container nginx
 Normal Started 45s kubelet, k8s-node1 Started container nginx


Name: nginx-deployment-54f57cf6bf-r9q89
Namespace: default
Priority: 0
Node: k8s-node2/10.0.2.15
Start Time: Sun, 20 Oct 2019 10:17:24 +0800
Labels: app=nginx
 pod-template-hash=54f57cf6bf
Annotations: <none>
Status: Running
IP: 172.100.2.3
IPs:
 IP: 172.100.2.3
Controlled By: ReplicaSet/nginx-deployment-54f57cf6bf
Containers:
 nginx:
 Container ID: docker://61894b3762d723eabe2510284bb107098bdbcb8da4f841eea863a9c7a59c5806
 Image: nginx:1.7.9
 Image ID: docker-pullable://nginx@sha256:e3456c851a152494c3e4ff5fcc26f240206abac0c9d794affb40e0714846c451
 Port: 80/TCP
 Host Port: 0/TCP
 State: Running
 Started: Sun, 20 Oct 2019 10:18:08 +0800
 Ready: True
 Restart Count: 0
 Environment: <none>
 Mounts:
 /var/run/secrets/kubernetes.io/serviceaccount from default-token-zjpq9 (ro)
Conditions:
 Type Status
 Initialized True
 Ready True
 ContainersReady True
 PodScheduled True
Volumes:
 default-token-zjpq9:
 Type: Secret (a volume populated by a Secret)
 SecretName: default-token-zjpq9
 Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Scheduled <unknown> default-scheduler Successfully assigned default/nginx-deployment-54f57cf6bf-r9q89 to k8s-node2
 Normal Pulling 3m4s kubelet, k8s-node2 Pulling image "nginx:1.7.9"
 Normal Pulled 2m24s kubelet, k8s-node2 Successfully pulled image "nginx:1.7.9"
 Normal Created 2m22s kubelet, k8s-node2 Created container nginx
 Normal Started 2m22s kubelet, k8s-node2 Started container nginx

Deployment相关操作

部署Deployment

$ kubectl create -f nginx_deployment.yml
deployment.apps/nginx-deployment created

升级Deployment

apply兼有create的语义，故也可直接使用apply创建\部署deployment

$ kubectl apply -f nginx_deployment_scale.yml
deployment.apps/nginx-deployment configured

查看Deployment更多信息

$ kubectl get deployment -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
nginx-deployment 2/2 2 2 5m28s nginx nginx:1.8 app=nginx

查看详细信息

$ kubectl describe deployments.apps
Name: nginx-deployment
Namespace: default
CreationTimestamp: Sun, 20 Oct 2019 10:17:24 +0800
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 3
 kubectl.kubernetes.io/last-applied-configuration:
 {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},"spec":{"replica...
Selector: app=nginx
Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
 Labels: app=nginx
 Containers:
 nginx:
 Image: nginx:1.9.1
 Port: 80/TCP
 Host Port: 0/TCP
 Environment: <none>
 Mounts: <none>
 Volumes: <none>
Conditions:
 Type Status Reason
 ---- ------ ------
 Progressing True NewReplicaSetAvailable
 Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-56f8998dbc (2/2 replicas created)
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal ScalingReplicaSet 22m deployment-controller Scaled up replica set nginx-deployment-54f57cf6bf to 2
 Normal ScalingReplicaSet 18m deployment-controller Scaled up replica set nginx-deployment-9f46bb5 to 1
 Normal ScalingReplicaSet 17m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 1
 Normal ScalingReplicaSet 17m deployment-controller Scaled up replica set nginx-deployment-9f46bb5 to 2
 Normal ScalingReplicaSet 16m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 0
 Normal ScalingReplicaSet 15m deployment-controller Scaled up replica set nginx-deployment-9f46bb5 to 4
 Normal ScalingReplicaSet 12m deployment-controller Scaled down replica set nginx-deployment-9f46bb5 to 3
 Normal ScalingReplicaSet 6m25s deployment-controller Scaled up replica set nginx-deployment-56f8998dbc to 1
 Normal ScalingReplicaSet 5m43s deployment-controller Scaled down replica set nginx-deployment-9f46bb5 to 2
 Normal ScalingReplicaSet 63s (x6 over 5m43s) deployment-controller (combined from similar events): Scaled down replica set nginx-deployment-56f8998dbc to 2

Node相关

查看Node

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 24m v1.16.2
k8s-node1 Ready <none> 18m v1.16.2
k8s-node2 Ready <none> 18m v1.16.2

查看更多信息

$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master Ready master 24m v1.16.2 10.0.2.15 <none> CentOS Linux 7 (Core) 3.10.0-957.12.2.el7.x86_64 docker://18.9.8
k8s-node1 Ready <none> 18m v1.16.2 10.0.2.15 <none> CentOS Linux 7 (Core) 3.10.0-957.12.2.el7.x86_64 docker://18.9.8
k8s-node2 Ready <none> 18m v1.16.2 10.0.2.15 <none> CentOS Linux 7 (Core) 3.10.0-957.12.2.el7.x86_64 docker://18.9.8

查看标签

$ kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
k8s-master Ready master 24m v1.16.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master,kubernetes.io/os=linux,node-role.kubernetes.io/master=
k8s-node1 Ready <none> 18m v1.16.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node1,kubernetes.io/os=linux
k8s-node2 Ready <none> 18m v1.16.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node2,kubernetes.io/os=linux

设置角色标签

$ kubectl label nodes k8s-node1 node-role.kubernetes.io/worker=
node/k8s-node1 labeled

查看详细信息

$ kubectl describe nodes
Name: k8s-master
Roles: master
Labels: beta.kubernetes.io/arch=amd64
 beta.kubernetes.io/os=linux
 kubernetes.io/arch=amd64
 kubernetes.io/hostname=k8s-master
 kubernetes.io/os=linux
 node-role.kubernetes.io/master=
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"32:38:c5:18:f0:ed"}
 flannel.alpha.coreos.com/backend-type: vxlan
 flannel.alpha.coreos.com/kube-subnet-manager: true
 flannel.alpha.coreos.com/public-ip: 10.0.2.15
 kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
 node.alpha.kubernetes.io/ttl: 0
 volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sun, 20 Oct 2019 10:05:02 +0800
Taints: node-role.kubernetes.io/master:NoSchedule
Unschedulable: false
Conditions:
 Type Status LastHeartbeatTime LastTransitionTime Reason Message
 ---- ------ ----------------- ------------------ ------ -------
 MemoryPressure False Sun, 20 Oct 2019 10:27:46 +0800 Sun, 20 Oct 2019 10:05:02 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
 DiskPressure False Sun, 20 Oct 2019 10:27:46 +0800 Sun, 20 Oct 2019 10:05:02 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
 PIDPressure False Sun, 20 Oct 2019 10:27:46 +0800 Sun, 20 Oct 2019 10:05:02 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
 Ready True Sun, 20 Oct 2019 10:27:46 +0800 Sun, 20 Oct 2019 10:07:42 +0800 KubeletReady kubelet is posting ready status
Addresses:
 InternalIP: 10.0.2.15
 Hostname: k8s-master
Capacity:
 cpu: 2
 ephemeral-storage: 41921540Ki
 hugepages-2Mi: 0
 memory: 1882144Ki
 pods: 110
Allocatable:
 cpu: 2
 ephemeral-storage: 38634891201
 hugepages-2Mi: 0
 memory: 1779744Ki
 pods: 110
System Info:
 Machine ID: d74c50274729456aad2a6198de9863b3
 System UUID: D74C5027-4729-456A-AD2A-6198DE9863B3
 Boot ID: fb2ea7bd-afb2-4051-afb7-78d9f59ec6f6
 Kernel Version: 3.10.0-957.12.2.el7.x86_64
 OS Image: CentOS Linux 7 (Core)
 Operating System: linux
 Architecture: amd64
 Container Runtime Version: docker://18.9.8
 Kubelet Version: v1.16.2
 Kube-Proxy Version: v1.16.2
PodCIDR: 172.100.0.0/24
PodCIDRs: 172.100.0.0/24
Non-terminated Pods: (8 in total)
 Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
 --------- ---- ------------ ---------- --------------- ------------- ---
 kube-system coredns-58cc8c89f4-c297b 100m (5%) 0 (0%) 70Mi (4%) 170Mi (9%) 23m
 kube-system coredns-58cc8c89f4-spzhj 100m (5%) 0 (0%) 70Mi (4%) 170Mi (9%) 23m
 kube-system etcd-k8s-master 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22m
 kube-system kube-apiserver-k8s-master 250m (12%) 0 (0%) 0 (0%) 0 (0%) 22m
 kube-system kube-controller-manager-k8s-master 200m (10%) 0 (0%) 0 (0%) 0 (0%) 22m
 kube-system kube-flannel-ds-amd64-6rchg 100m (5%) 100m (5%) 50Mi (2%) 50Mi (2%) 21m
 kube-system kube-proxy-xlb62 0 (0%) 0 (0%) 0 (0%) 0 (0%) 23m
 kube-system kube-scheduler-k8s-master 100m (5%) 0 (0%) 0 (0%) 0 (0%) 22m
Allocated resources:
 (Total limits may be over 100 percent, i.e., overcommitted.)
 Resource Requests Limits
 -------- -------- ------
 cpu 850m (42%) 100m (5%)
 memory 190Mi (10%) 390Mi (22%)
 ephemeral-storage 0 (0%) 0 (0%)
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal NodeHasSufficientMemory 23m (x8 over 23m) kubelet, k8s-master Node k8s-master status is now: NodeHasSufficientMemory
 Normal NodeHasNoDiskPressure 23m (x8 over 23m) kubelet, k8s-master Node k8s-master status is now: NodeHasNoDiskPressure
 Normal NodeHasSufficientPID 23m (x7 over 23m) kubelet, k8s-master Node k8s-master status is now: NodeHasSufficientPID
 Normal Starting 23m kube-proxy, k8s-master Starting kube-proxy.


Name: k8s-node1
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
 beta.kubernetes.io/os=linux
 kubernetes.io/arch=amd64
 kubernetes.io/hostname=k8s-node1
 kubernetes.io/os=linux
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"16:5c:40:b1:14:b6"}
 flannel.alpha.coreos.com/backend-type: vxlan
 flannel.alpha.coreos.com/kube-subnet-manager: true
 flannel.alpha.coreos.com/public-ip: 10.0.2.15
 kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
 node.alpha.kubernetes.io/ttl: 0
 volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sun, 20 Oct 2019 10:11:03 +0800
Taints: <none>
Unschedulable: false
Conditions:
 Type Status LastHeartbeatTime LastTransitionTime Reason Message
 ---- ------ ----------------- ------------------ ------ -------
 MemoryPressure False Sun, 20 Oct 2019 10:28:08 +0800 Sun, 20 Oct 2019 10:11:03 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
 DiskPressure False Sun, 20 Oct 2019 10:28:08 +0800 Sun, 20 Oct 2019 10:11:03 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
 PIDPressure False Sun, 20 Oct 2019 10:28:08 +0800 Sun, 20 Oct 2019 10:11:03 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
 Ready True Sun, 20 Oct 2019 10:28:08 +0800 Sun, 20 Oct 2019 10:11:34 +0800 KubeletReady kubelet is posting ready status
Addresses:
 InternalIP: 10.0.2.15
 Hostname: k8s-node1
Capacity:
 cpu: 1
 ephemeral-storage: 41921540Ki
 hugepages-2Mi: 0
 memory: 1882296Ki
 pods: 110
Allocatable:
 cpu: 1
 ephemeral-storage: 38634891201
 hugepages-2Mi: 0
 memory: 1779896Ki
 pods: 110
System Info:
 Machine ID: 5d806dde9b0444bdb5c9800a228267d8
 System UUID: 5D806DDE-9B04-44BD-B5C9-800A228267D8
 Boot ID: e6b7222c-686c-49cb-8e56-9bb263852d36
 Kernel Version: 3.10.0-957.12.2.el7.x86_64
 OS Image: CentOS Linux 7 (Core)
 Operating System: linux
 Architecture: amd64
 Container Runtime Version: docker://18.9.8
 Kubelet Version: v1.16.2
 Kube-Proxy Version: v1.16.2
PodCIDR: 172.100.1.0/24
PodCIDRs: 172.100.1.0/24
Non-terminated Pods: (4 in total)
 Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
 --------- ---- ------------ ---------- --------------- ------------- ---
 default nginx-deployment-9f46bb5-7vlb7 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m53s
 default nginx-deployment-9f46bb5-c7czx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6m35s
 kube-system kube-flannel-ds-amd64-6fwg5 100m (10%) 100m (10%) 50Mi (2%) 50Mi (2%) 17m
 kube-system kube-proxy-cdcs4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 17m
Allocated resources:
 (Total limits may be over 100 percent, i.e., overcommitted.)
 Resource Requests Limits
 -------- -------- ------
 cpu 100m (10%) 100m (10%)
 memory 50Mi (2%) 50Mi (2%)
 ephemeral-storage 0 (0%) 0 (0%)
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Starting 17m kubelet, k8s-node1 Starting kubelet.
 Normal NodeHasSufficientMemory 17m kubelet, k8s-node1 Node k8s-node1 status is now: NodeHasSufficientMemory
 Normal NodeHasNoDiskPressure 17m kubelet, k8s-node1 Node k8s-node1 status is now: NodeHasNoDiskPressure
 Normal NodeHasSufficientPID 17m kubelet, k8s-node1 Node k8s-node1 status is now: NodeHasSufficientPID
 Normal NodeAllocatableEnforced 17m kubelet, k8s-node1 Updated Node Allocatable limit across pods
 Normal Starting 17m kube-proxy, k8s-node1 Starting kube-proxy.
 Normal NodeReady 16m kubelet, k8s-node1 Node k8s-node1 status is now: NodeReady


Name: k8s-node2
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
 beta.kubernetes.io/os=linux
 kubernetes.io/arch=amd64
 kubernetes.io/hostname=k8s-node2
 kubernetes.io/os=linux
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"76:49:24:1f:e5:d6"}
 flannel.alpha.coreos.com/backend-type: vxlan
 flannel.alpha.coreos.com/kube-subnet-manager: true
 flannel.alpha.coreos.com/public-ip: 10.0.2.15
 kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
 node.alpha.kubernetes.io/ttl: 0
 volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sun, 20 Oct 2019 10:11:09 +0800
Taints: <none>
Unschedulable: false
Conditions:
 Type Status LastHeartbeatTime LastTransitionTime Reason Message
 ---- ------ ----------------- ------------------ ------ -------
 MemoryPressure False Sun, 20 Oct 2019 10:27:42 +0800 Sun, 20 Oct 2019 10:11:09 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
 DiskPressure False Sun, 20 Oct 2019 10:27:42 +0800 Sun, 20 Oct 2019 10:11:09 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
 PIDPressure False Sun, 20 Oct 2019 10:27:42 +0800 Sun, 20 Oct 2019 10:11:09 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
 Ready True Sun, 20 Oct 2019 10:27:42 +0800 Sun, 20 Oct 2019 10:11:40 +0800 KubeletReady kubelet is posting ready status
Addresses:
 InternalIP: 10.0.2.15
 Hostname: k8s-node2
Capacity:
 cpu: 1
 ephemeral-storage: 41921540Ki
 hugepages-2Mi: 0
 memory: 1882296Ki
 pods: 110
Allocatable:
 cpu: 1
 ephemeral-storage: 38634891201
 hugepages-2Mi: 0
 memory: 1779896Ki
 pods: 110
System Info:
 Machine ID: 7c59ba12638e4fdfa20cfeda0aa2fda1
 System UUID: 7C59BA12-638E-4FDF-A20C-FEDA0AA2FDA1
 Boot ID: e402f75c-42e7-454d-aaca-f96853091c15
 Kernel Version: 3.10.0-957.12.2.el7.x86_64
 OS Image: CentOS Linux 7 (Core)
 Operating System: linux
 Architecture: amd64
 Container Runtime Version: docker://18.9.8
 Kubelet Version: v1.16.2
 Kube-Proxy Version: v1.16.2
PodCIDR: 172.100.2.0/24
PodCIDRs: 172.100.2.0/24
Non-terminated Pods: (3 in total)
 Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
 --------- ---- ------------ ---------- --------------- ------------- ---
 default nginx-deployment-9f46bb5-mv4b7 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5m52s
 kube-system kube-flannel-ds-amd64-z4plh 100m (10%) 100m (10%) 50Mi (2%) 50Mi (2%) 17m
 kube-system kube-proxy-nlmbb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 17m
Allocated resources:
 (Total limits may be over 100 percent, i.e., overcommitted.)
 Resource Requests Limits
 -------- -------- ------
 cpu 100m (10%) 100m (10%)
 memory 50Mi (2%) 50Mi (2%)
 ephemeral-storage 0 (0%) 0 (0%)
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Starting 17m kubelet, k8s-node2 Starting kubelet.
 Normal NodeHasSufficientMemory 17m kubelet, k8s-node2 Node k8s-node2 status is now: NodeHasSufficientMemory
 Normal NodeHasNoDiskPressure 17m kubelet, k8s-node2 Node k8s-node2 status is now: NodeHasNoDiskPressure
 Normal NodeHasSufficientPID 17m kubelet, k8s-node2 Node k8s-node2 status is now: NodeHasSufficientPID
 Normal NodeAllocatableEnforced 17m kubelet, k8s-node2 Updated Node Allocatable limit across pods
 Normal Starting 17m kube-proxy, k8s-node2 Starting kube-proxy.
 Normal NodeReady 16m kubelet, k8s-node2 Node k8s-node2 status is now: NodeReady

服务伸缩

`scale`命令方式

$ kubectl scale --replicas=6 deployment/nginx-deployment
deployment.apps/nginx-deployment scaled

查看副本数

$ kubectl get replicaset
NAME DESIRED CURRENT READY AGE
nginx-deployment-54f57cf6bf 0 0 0 21m
nginx-deployment-56f8998dbc 6 6 3 5m7s
nginx-deployment-9f46bb5 0 0 0 16m

Kubernetes 集群搭建

Sun, 20 Oct 2019 12:31:00 +0800

搭建方法分类

如果是搭建Kubernetes的学习环境，则可以直接使用minikube快速搭建单节点的Kubernetes环境，官方推荐使用kubeadm搭建Kubernetes集群生产环境

但kubeadm需要连接谷歌容器仓库获取镜像，在网络受限的情况下无法搭建成功，故有两种搭建方法：

服务器使用代理，代理所有的http和https连接，可以自由访问任意地址
使用阿里云镜像地址

本次采用第二种方式进行搭建

环境准备

本次宿主机为CentOS 7，结合3台virtual box虚拟机模拟集群环境，虚拟机使用Vagrant进行管理，vagrant安装比较简单，此处不再赘述

宿主机配置：

CPU：4C
RAM：8G

虚拟机配置

由于网络受限，vagrant默认拉取的操作系统文件地址不可访问，需要自行从官方下载镜像或者使用代理下载镜像，此处对此操作也不再赘述

三台虚拟机配置文件

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.require_version ">= 1.6.0"

boxes = [
 {
 :name => "k8s-master",
 :eth1 => "192.168.205.120",
 :mem => "2048",
 :cpu => "2"
 },
 {
 :name => "k8s-node1",
 :eth1 => "192.168.205.121",
 :mem => "2048",
 :cpu => "1"
 },
 {
 :name => "k8s-node2",
 :eth1 => "192.168.205.122",
 :mem => "2048",
 :cpu => "1"
 }

]

Vagrant.configure(2) do |config|

 config.vm.box = "centos/7"
 boxes.each do |opts|
 config.vm.define opts[:name] do |config|
 config.vm.hostname = opts[:name]
 config.vm.provider "vmware_fusion" do |v|
 v.vmx["memsize"] = opts[:mem]
 v.vmx["numvcpus"] = opts[:cpu]
 end
 config.vm.provider "virtualbox" do |v|
 v.customize ["modifyvm", :id, "--memory", opts[:mem]]
 v.customize ["modifyvm", :id, "--cpus", opts[:cpu]]
 end
 config.vm.network :private_network, ip: opts[:eth1]
 end
 end
 config.vm.provision "shell", privileged: true, path: "./setup.sh"
end

所有虚拟机启动后都会执行./setup.sh脚本配置基础环境，脚本内容如下

#/bin/sh

sudo yum install -y vim telnet bind-utils wget yum-utils

sudo yum-config-manager \
 --add-repo \
 https://download.docker.com/linux/centos/docker-ce.repo
sudo yum -y install docker-ce-18.09.8 docker-ce-cli-18.09.8 containerd.io

if [ ! $(getent group docker) ];
then 
 sudo groupadd docker;
else
 echo "docker user group already exists"
fi

sudo gpasswd -a $USER docker
sudo systemctl restart docker

# open password auth for backup if ssh key doesn't work, bydefault, username=vagrant password=vagrant
sudo sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/g' /etc/ssh/sshd_config
sudo systemctl restart sshd

sudomkdir /etc/yum.repos.d/bak && mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/bak
sudowget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.cloud.tencent.com/repo/centos7_base.repo
sudowget -O /etc/yum.repos.d/epel.repo http://mirrors.cloud.tencent.com/repo/epel-7.repo

yum clean all && yum makecache

sudo bash -c 'cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF'

sudo wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
 "registry-mirrors": ["https://clayphwh.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

sudo setenforce 0

sudo yum install -y kubelet kubeadm kubectl

sudo bash -c 'cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
EOF'
sudo sysctl --system

sudo systemctl stop firewalld
sudo systemctl disable firewalld
sudo swapoff -a

sudo systemctl enable docker.service
sudo systemctl enable kubelet.service

脚本内已将K8S镜像均配置为阿里云镜像仓库，并配置了阿里云容器加速服务地址，更便于在国内拉取镜像

启动虚拟机

使用命令启动虚拟机

$ vagrant up

启动需要较长时间，若出现大量超时也请耐心等待

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.163.com
 * updates: mirrors.163.com
http://vault.centos.org/7.1.1503/os/x86_64/repodata/0e6e90965f55146ba5025ea450f822d1bb0267d0299ef64dd4365825e6bad995-c7-x86_64-comps.xml.gz: [Errno 12] Timeout on http://vault.centos.org/7.1.1503/os/x86_64/repodata/0e6e90965f55146ba5025ea450f822d1bb0267d0299ef64dd4365825e6bad995-c7-x86_64-comps.xml.gz: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
http://vault.centos.org/7.1.1503/updates/x86_64/repodata/93b71f445d2ec2138d28152612f4fb29c8e76ee31f2666b964d88249b4e0a955-primary.sqlite.bz2: [Errno 12] Timeout on http://vault.centos.org/7.1.1503/updates/x86_64/repodata/93b71f445d2ec2138d28152612f4fb29c8e76ee31f2666b964d88249b4e0a955-primary.sqlite.bz2: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
http://vault.centos.org/7.2.1511/os/x86_64/repodata/c6411f1cc8a000ed2b651b49134631d279abba1ec1f78e5dcca79a52d8c1eada-primary.sqlite.bz2: [Errno 12] Timeout on http://vault.centos.org/7.2.1511/os/x86_64/repodata/c6411f1cc8a000ed2b651b49134631d279abba1ec1f78e5dcca79a52d8c1eada-primary.sqlite.bz2: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
...

启动完成后通过命令查看虚拟机状态

$ vagrant status
Current machine states:

k8s-master running (virtualbox)
k8s-node1 running (virtualbox)
k8s-node2 running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

K8S master节点初始化

进入主节点，查看环境是否已经配置完成

$ vagrant ssh k8s-master

执行以下三条语句查看是否有预期输出

[vagrant@k8s-master ~]$ sudo which kubeadm
/bin/kubeadm
[vagrant@k8s-master ~]$ sudo which kubelet
/bin/kubelet
[vagrant@k8s-master ~]$ sudo which kubectl
/bin/kubectl
[vagrant@k8s-master ~]$ sudo docker version
Client:
 Version: 18.09.8
 API version: 1.39
 Go version: go1.10.8
 Git commit: 0dd43dd87f
 Built: Wed Jul 17 17:40:31 2019
 OS/Arch: linux/amd64
 Experimental: false

Server: Docker Engine - Community
 Engine:
 Version: 18.09.8
 API version: 1.39 (minimum version 1.12)
 Go version: go1.10.8
 Git commit: 0dd43dd
 Built: Wed Jul 17 17:10:42 2019
 OS/Arch: linux/amd64
 Experimental: false

使用初始化命令，并指定拉取镜像地址为阿里云地址，同时指定容器网络地址和注册中心广播地址

$ sudo kubeadm init --pod-network-cidr 172.100.0.0/16 --image-repository registry.aliyuncs.com/google_containers --apiserver-advertise-address 192.168.205.120

输出如下即初始化完成

$ sudo kubeadm init --pod-network-cidr 172.100.0.0/16 --image-repository registry.aliyuncs.com/google_containers --apiserver-advertise-address 192.168.205.120
W1020 01:31:27.560129 20473 version.go:101] could not fetch a `Kubernetes`version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1020 01:31:27.560713 20473 version.go:102] falling back to the local client version: v1.16.2
[init] Using `Kubernetes`version: v1.16.2
[preflight] Running pre-flight checks
 [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a `Kubernetes`cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master `kubernetes`kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.205.120]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.205.120 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.205.120 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 59.511578 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: rod570.p67pymzbil4m6u8d
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your `Kubernetes`control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
 https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.205.120:6443 --token rod570.p67pymzbil4m6u8d \
 --discovery-token-ca-cert-hash sha256:8f4cb9d555c78d58befeb3cfa3f7537989aa599e53e4f1bae929d8cc7afd1476

kubectl 配置

拷贝kubectl配置

$ rm -rf ~/.kube
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

若不配置直接使用则报无法连接错误

[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces
The connection to the server localhost:8080 was refused - did you specify the right host or port?

此时使用kubectl可查看k8s容器运行情况，输出如下则为初始化成功，启动由于未配置网络插件，故coredns一直处于Pending状态

[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58cc8c89f4-c297b 0/1 Pending 0 81s
kube-system coredns-58cc8c89f4-spzhj 0/1 Pending 0 81s
kube-system etcd-k8s-master 1/1 Running 0 39s
kube-system kube-apiserver-k8s-master 1/1 Running 0 49s
kube-system kube-controller-manager-k8s-master 1/1 Running 0 40s
kube-system kube-proxy-xlb62 1/1 Running 0 81s
kube-system kube-scheduler-k8s-master 1/1 Running 0 51s

安装网络插件

网络插件有多个可选，可通过官网查看，此处选择安装flannel CNI网络插件

For flannel to work correctly, you must pass --pod-network-cidr=10.244.0.0/16 to kubeadm init.

Set /proc/sys/net/bridge/bridge-nf-call-iptables to 1 by running sysctl net.bridge.bridge-nf-call-iptables=1 to pass bridged IPv4 traffic to iptables’ chains. This is a requirement for some CNI plugins to work, for more information please see here.

Make sure that your firewall rules allow UDP ports 8285 and 8472 traffic for all hosts participating in the overlay network. see here .

Note that flannel works on amd64, arm, arm64, ppc64le and s390x under Linux. Windows (amd64) is claimed as supported in v0.11.0 but the usage is undocumented.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml
For more information about flannel, see the CoreOS flannel repository on GitHub .

相关的网络配置已在脚本中写入，此时直接执行yml脚本即可

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

再次查看容器运行情况，所有容器均已运行，则master节点配置完成

[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58cc8c89f4-c297b 1/1 Running 0 5m17s
kube-system coredns-58cc8c89f4-spzhj 1/1 Running 0 5m17s
kube-system etcd-k8s-master 1/1 Running 0 4m35s
kube-system kube-apiserver-k8s-master 1/1 Running 0 4m45s
kube-system kube-controller-manager-k8s-master 1/1 Running 0 4m36s
kube-system kube-flannel-ds-amd64-6rchg 1/1 Running 0 3m50s
kube-system kube-proxy-xlb62 1/1 Running 0 5m17s
kube-system kube-scheduler-k8s-master 1/1 Running 0 4m47s

配置worker节点

以k8s-node1为例，执行master初始化时提供的脚本即可加入k8s集群，输出如下则加入成功

[vagrant@k8s-node1 ~]$ sudo kubeadm join 192.168.205.120:6443 --token 8ry5oo.y48ksgurn103zq4h \
> --discovery-token-ca-cert-hash sha256:ead07352591500c2cfe3321bf87d2e068790e16f2a7e0cc23541c864d24006d4
[preflight] Running pre-flight checks
 [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

稍等片刻进入master节点查看容器运行情况，会发现多了几个网络容器，属于预期情况

[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58cc8c89f4-c297b 1/1 Running 0 6m28s
kube-system coredns-58cc8c89f4-spzhj 1/1 Running 0 6m28s
kube-system etcd-k8s-master 1/1 Running 0 5m46s
kube-system kube-apiserver-k8s-master 1/1 Running 0 5m56s
kube-system kube-controller-manager-k8s-master 1/1 Running 0 5m47s
kube-system kube-flannel-ds-amd64-6fwg5 1/1 Running 0 49s
kube-system kube-flannel-ds-amd64-6rchg 1/1 Running 0 5m1s
kube-system kube-flannel-ds-amd64-z4plh 1/1 Running 0 43s
kube-system kube-proxy-cdcs4 1/1 Running 0 49s
kube-system kube-proxy-nlmbb 1/1 Running 0 43s
kube-system kube-proxy-xlb62 1/1 Running 0 6m28s
kube-system kube-scheduler-k8s-master 1/1 Running 0 5m58s

集群验证

在master节点中启动nginx容器测试集群情况

$ kubectl create deployment nginx --image=nginx

稍等片刻，输出如下则集群正常运行，k8s集群配置完成

[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-86c57db685-2gkxm 1/1 Running 0 58s
kube-system coredns-58cc8c89f4-c297b 1/1 Running 0 7m46s
kube-system coredns-58cc8c89f4-spzhj 1/1 Running 0 7m46s
kube-system etcd-k8s-master 1/1 Running 0 7m4s
kube-system kube-apiserver-k8s-master 1/1 Running 0 7m14s
kube-system kube-controller-manager-k8s-master 1/1 Running 0 7m5s
kube-system kube-flannel-ds-amd64-6fwg5 1/1 Running 0 2m7s
kube-system kube-flannel-ds-amd64-6rchg 1/1 Running 0 6m19s
kube-system kube-flannel-ds-amd64-z4plh 1/1 Running 0 2m1s
kube-system kube-proxy-cdcs4 1/1 Running 0 2m7s
kube-system kube-proxy-nlmbb 1/1 Running 0 2m1s
kube-system kube-proxy-xlb62 1/1 Running 0 7m46s
kube-system kube-scheduler-k8s-master 1/1 Running 0 7m16s

Kubernetes on F嘉阳

ArgoCD落地GitOps

背景

技术选型

原理

实践

环境

安装ArgoCD

设计代码库结构

环境配置迁移

部署ingress

部署Prometheus+Grafana

Kubesphere落地DevOps

创建企业空间

安装helm

找不到环境变量报错

安装SonarQube

配置 SonarQube 服务器

创建 SonarQube 管理员令牌 (Token)

创建 Webhook 服务器

将 SonarQube 配置添加到 ks-installer

将 SonarQube 服务器添加至 Jenkins

将 sonarqubeURL 添加到 KubeSphere 控制台

重启服务

为新工程创建 SonarQube Token

创建流水线工程

创建凭证

创建流水线

在 KubeSphere 控制台查看结果

Kubesphere运维Kubernetes集群

背景

解决方案

安装步骤

依赖包安装

配置DNS解析

按照官方教程安装多节点kubernetes

使用

利用Docker自建DNS服务器

背景

步骤

搭建DNS服务器

服务器指向新DNS服务器

补充

DevOps实践心得二

背景

Mesos+Marathon的DevOps分布式环境探索

全面容器化服务探索

遇到的坑

后续发展展望

k8s证书失效问题排查

问题背景

处理过程

1. 排查服务是否启动

2. 检查APIServer

3. 重启kubelet

4.检查API Server日志

解决方案

外部推荐方案

采用方案

重新签发证书

证书自动续签

k8s环境下GitLab+Helm+GitLab Runner Java项目CICD落地实践

背景

优势

环境

GitLab-Runner部署

配置介绍

使用DinD方式构建已经不再推荐

Use docker-in-docker workflow with Docker executor

Using docker:dind

使用Kaniko构建Docker镜像

使用Helm部署

GitLab-Runner选择可能导致的坑

安装

工程配置

DinD方式构建所需配置

Kubectl和Kubernetes权限配置

Kubernetes关联GitLab配置

GitLab环境配置

CI脚本配置

Using `docker:dind`

使用`DinD`构建`Dockerfile`结果

使用`Kaniko`构建`Dockerfile`的结果

默认查看`default`命名空间的Pod

`scale`命令方式