Grafana on F嘉阳

ArgoCD落地GitOps

Sat, 23 Aug 2025 22:49:50 +0800

背景

在过去的DevOps实践中，跨环境的基础设施部署和迁移经常存在遗漏，导致环境差异，部署一个新环境往往需要几天。传统CICD流水线采用“推”模式，将变更推送到不同的环境，推送后也不对目标环境状态进行管理和监控，容易导致推送后部署失败。随着云原生容器化已经成为行业事实标准，GitOps成为一种更可靠的应用部署和基础设置运维的范式。

技术选型

ArgoCD遵循GitOps的核心原则，已经成为落地GitOps的默认技术选型，因此此处不探讨其他选型方案。

原理

Git 作为唯一事实来源（GitOps思想）
自动化和持续同步：Argo CD会持续监控 Git 仓库的状态，并自动将集群的实际状态与 Git 中声明的目标状态进行同步。
- 利用K8s的Kustomize能力完成对集群apply操作

实践

环境

安装ArgoCD

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

检查安装结果

查看argo cd登录密码

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

设计代码库结构

落地ArgoCD通常有以下几种仓库设计范式

单仓库模式
多仓库模式
App of Apps模式

本次采用App of Apps的单仓库模式，仓库结构如下

.
├── apps
│ ├── app-one
│ ├── app-two
├── cluster-infra
│ ├── argocd
│ │ ├── base
│ │ └── overlays
│ ├── ingress
│ │ ├── base
│ │ └── overlays
│ │ └── dev
│ └── prometheus-stack
│ ├── base
│ └── overlays
│ └── dev
└── environments
 └── dev
 └── root.yaml

该结构特点如下

基础设施和应用部署分离（apps和cluster-infra）
环境基础配置和环境定制配置分离（base和overlays）
环境根节点配置（environments控制git仓库配置、应用apps注册）

本次将该代码库存放到阿里云代码仓库中。

环境配置迁移

目前的k3s环境中已经手动运维安装了argocd，现在需要将已安装的配置迁移到git库中

将官方安装配置https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml写入到cluster-infra/argocd/base/argocd.yaml中

配置``cluster-infra/argocd/base/kustomization.yaml`文件，注册同目录的安装文件

resources:
 - argocd.yaml

本次依赖阿里云代码仓库，通过ssh连接，需要覆盖默认的argocd配置，编写cluster-infra/argocd/overlays/with-known-hosts/ssh-known-hosts.patch.yaml

apiVersion: v1 
kind: ConfigMap 
metadata: 
 name: argocd-ssh-known-hosts-cm 
data: 
 ssh_known_hosts: |- 
 # BEGIN >>> 自定义条目（全局生效） 
 codeup.aliyun.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6WbValczSDKC4ktjRzDNC1qINDAViw/oGA8ud7qH5F9XFZ4QAWhZmFBsp+cppBIOM63exjc4O9vbzMrd7AUItXigymGfTqrGt5VykIiaNhjttxSwbZf3N9eDcAvX4gGm3fcm6D1ACvPxvxMsCwOpdz5tDHqDxGOLPAXN+FSAvjdcN4uMz0MXghGh9Q3zSpy8z1TLkAKT2QBnr4q2tsdZKwWON8bBHIPwP3RSAdSUL6bAyotyncMDAvG53kwRKJ3wJ5ZezXqZNNV7EzHds+pvwThtuWtWsBuuRpL/oZiviWlXy6hILoWbkRfgONPwT0faaOn1GLzjHLuqHHfALGOen 
 # END <<< 自定义条目 
 
 # This file was automatically generated by hack/update-ssh-known-hosts.sh. DO NOT EDIT 
 [ssh.github.com]:443 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg= 
 [ssh.github.com]:443 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl 
 [ssh.github.com]:443 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk= 
 bitbucket.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPIQmuzMBuKdWeF4+a2sjSSpBK0iqitSQ+5BM9KhpexuGt20JpTVM7u5BDZngncgrqDMbWdxMWWOGtZ9UgbqgZE= 
 bitbucket.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO 
 bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQeJzhupRu0u0cdegZIa8e86EG2qOCsIsD1Xw0xSeiPDlCr7kq97NLmMbpKTX6Esc30NuoqEEHCuc7yWtwp8dI76EEEB1VqY9QJq6vk+aySyboD5QF61I/1WeTwu+deCbgKMGbUijeXhtfbxSxm6JwGrXrhBdofTsbKRUsrN1WoNgUa8uqN1Vx6WAJw1JHPhglEGGHea6QICwJOAr/6mrui/oB7pkaWKHj3z7d1IC4KWLtY47elvjbaTlkN04Kc/5LFEirorGYVbt15kAUlqGM65pk6ZBxtaO3+30LVlORZkxOh+LKL/BvbZ/iRNhItLqNyieoQj/uh/7Iv4uyH/cV/0b4WDSd3DptigWq84lJubb9t/DnZlrJazxyDCulTmKdOR7vs9gMTo+uoIrPSb8ScTtvw65+odKAlBj59dhnVp9zd7QUojOpXlL62Aw56U4oO+FALuevvMjiWeavKhJqlR7i5n9srYcrNV7ttmDw7kf/97P5zauIhxcjX+xHv4M= 
 github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg= 
 github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl 
 github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk= 
 gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY= 
 gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf 
 gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9 
 ssh.dev.azure.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H 
 vs-ssh.visualstudio.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

配置``cluster-infra/argocd/overlays/with-known-hosts/kustomization.yaml`文件，注册hosts patch文件

resources: 
 - ../../base 
 
patchesStrategicMerge: 
 - ssh-known-hosts.patch.yaml

按这样配置后，argocd会先执行base的安装文件apply基础环境，再使用ssh-known-hosts.patch.yaml覆盖ssh的配置，保证环境与git配置一致。

接下来，继续在apps根目录注册应用，配置environments/dev/kustomization.yaml

resources:
 - ../../cluster-infra/argocd/overlays/with-known-hosts

编写根目录git配置

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: dev-root 
 namespace: argocd 
spec: 
 project: default 
 source: 
 repoURL: git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git 
 targetRevision: master 
 path: environments/dev 
 kustomize: 
 enableHelm: true 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: argocd 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

到此，所有配置迁移完成，将git仓库提交到远程后，在master节点添加私钥

argocd repo add git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git --ssh-private-key-path /root/.ssh/gitops_ed25519

同步dev根应用

kubectl apply -f dev-root.yaml

本地使用kubectl暴露临时端口，再使用argocd cli连接集群查看状态

kubectl port-forward svc/argocd-server -n argocd --address 0.0.0.0 30443:443
argocd login xxxx:30443 --username admin
argocd app list

部署ingress

一直使用临时端口+命令行查看应用状态也不方便，官方提供webui界面查看部署状态，考虑使用ingress暴露固定域名访问

ingress的部署也遵循相同步骤

配置官方安装文件
配置环境个性化
1. 端口暴露
根应用添加配置

将官方https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml安装文件配置到cluster-infra/ingress/base/ingress-nginx.yaml

配置端口暴露cluster-infra/ingress/overlays/dev/ingress-nginx-controller-nodeport.yaml

apiVersion: v1 
kind: Service 
metadata: 
 name: ingress-nginx-controller 
 namespace: ingress-nginx 
spec: 
 type: NodePort 
 ports: 
 - name: http 
 port: 80 
 targetPort: http 
 protocol: TCP 
 nodePort: 30080 
 - name: https 
 port: 443 
 targetPort: https 
 protocol: TCP 
 nodePort: 30443

根应用添加配置

resources: 
 - ../../cluster-infra/ingress/overlays/dev 
 - ../../cluster-infra/argocd/overlays/with-known-hosts

在argocd环境目录下添加ingress域名配置

apiVersion: networking.k8s.io/v1 
kind: Ingress 
metadata: 
 name: argocd-ingress 
 namespace: argocd 
 annotations: 
 nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" 
spec: 
 ingressClassName: nginx 
 rules: 
 - host: xxxx 
 http: 
 paths: 
 - path: / 
 pathType: Prefix 
 backend: 
 service: 
 name: argocd-server 
 port: 
 number: 443

git提交后即可自动同步并创建ingress

访问域名查看argocd webui

部署Prometheus+Grafana

为了完善集群监控，argocd 支持通过helm charts部署应用

配置cluster-infra/prometheus-stack/base/prometheus-stack.yaml

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: prometheus-stack 
 namespace: argocd 
 annotations: 
 argocd.argoproj.io/sync-options: ServerSideApply=true 
spec: 
 project: default 
 source: 
 repoURL: https://prometheus-community.github.io/helm-charts 
 chart: kube-prometheus-stack 
 targetRevision: 75.16.1 
 helm: 
 valueFiles: 
 - values.yaml 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: monitoring 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

注意，由于Helm仓库部署prometheus时伴随大量CRD资源文件apply，需要开启ServerSideApply模式避免通过接口apply导致字符超长报错。参考：https://argocd.devops.gold/user-guide/sync-options/#_9

使用helm模式安装的应用无法使用patchesStrategicMerge策略覆盖，目前覆盖个性化配置只能将所有内容写到一个资源文件内，因此dev环境的个性化配置values如下

prometheus: 
 prometheusSpec: 
 retention: 3d 
 retentionSize: "2Gi" 
 resources: 
 requests: 
 memory: 128Mi 
 limits: 
 memory: 1Gi 
 storageSpec: 
 volumeClaimTemplate: 
 spec: 
 resources: 
 requests: 
 storage: 5Gi 
 ingress: 
 enabled: true 
 ingressClassName: nginx 
 annotations: {} 
 hosts: 
 - prometheus.fjy8018.top 
 paths: 
 - / 
 tls: 
 - hosts: 
 - xxx.xxx
 secretName: prometheus-tls 
 prometheusSpec: 
 externalUrl: https://xxx.xxx 
 routePrefix: / 
 
grafana: 
 adminPassword: "2EvwJJ0NOLzhYa" 
 resources: 
 requests: 
 memory: 64Mi 
 limits: 
 memory: 512Mi 
 ingress: 
 enabled: true 
 ingressClassName: nginx 
 annotations: {} 
 hosts: 
 - xxx.xxx 
 tls: 
 - hosts: 
 - xxx.xxx
 secretName: grafana-tls

需要在dev中覆盖base的资源文件，通过multi repo模式合并helm配置

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: prometheus-stack 
 namespace: argocd 
 annotations: 
 argocd.argoproj.io/sync-options: ServerSideApply=true 
spec: 
 project: default 
 sources: 
 - repoURL: https://prometheus-community.github.io/helm-charts 
 chart: kube-prometheus-stack 
 targetRevision: 75.16.1 
 helm: 
 valueFiles: 
 - $values/cluster-infra/prometheus-stack/overlays/dev/values-dev.yaml 
 - repoURL: 'git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git' 
 targetRevision: master 
 ref: values 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: monitoring 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

prometheus依赖CRD提前部署的namespace需要注册为独立应用提前配置好

apiVersion: v1 
kind: Namespace 
metadata: 
 name: monitoring

命名空间的配置和普通应用一致

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: common-ns 
 namespace: argocd 
spec: 
 project: default 
 source: 
 repoURL: git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git 
 targetRevision: master 
 path: cluster-infra/common-ns 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: argocd 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

最终根目录应用配置如下

resources: 
 - ../../cluster-infra/ingress/overlays/dev 
 - ../../cluster-infra/argocd/overlays/with-known-hosts 
 - ../../cluster-infra/prometheus-stack/overlays/dev 
 - common-ns-app.yaml

同步后效果如下

查看监控部署拓扑

通过域名访问grafana

k8s监控平台搭建

Tue, 12 Nov 2019 21:33:00 +0800

架构

目前主流容器化监控平台采用Prometheus+Grafana架构，下图为Prometheus架构图，可以主动发现各种指标和采集数据，实现

监控每个节点基础指标
监控每个容器基础指标
监控k8s集群组件基础指标

基础环境

Kubernetes：v1.14.0

Docker：17.03.1-ce

机器数：$3 \times master + 2 \times worker$

部署

Helm

Helm作为k8s平台软件包管理工具，结合prometheus-operator可以类似插件的方式快速搭建监控体系，Helm旗下包含charts项目，为软件包仓库

Helm官方仓库：https://github.com/helm/helm

charts官方仓库：https://github.com/helm/charts

安装Helm

通过官方仓库查看目前最新版为v2.16.0，但由于仍有不稳定的情况，故不推荐该版本，原因可通过其issue查看

此处选择安装v2.15.2的版本，下载页面

解压并放到可执行目录下

$ tar -zxvf helm-v2.15.2-linux-amd64.tar.gz
$ mv linux-amd64/helm /usr/local/bin/

Tiller安装

Tiller 是以 Deployment 方式部署在 Kubernetes 集群中的，由于国内网络原因，需要配置为阿里云镜像

$ helm init --client-only --stable-repo-url https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts/
$ helm repo add incubator https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts-incubator/
$ helm repo update

因为官方的镜像无法拉取，使用-i指定自己的镜像

$ helm init --service-account tiller --upgrade -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.15.2 --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

创建TLS认证服务端

$ helm init --service-account tiller --upgrade -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.15.2 --tiller-tls-cert /etc/kubernetes/ssl/tiller001.pem --tiller-tls-key /etc/kubernetes/ssl/tiller001-key.pem --tls-ca-cert /etc/kubernetes/ssl/ca.pem --tiller-namespace kube-system --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

Tiller授权

Tiller 需要访问ApiServer去对集群进行操作。目前的 Tiller 部署时默认没有定义授权的 ServiceAccount，这会导致访问 API Server 时被拒绝。所以我们需要明确为 Tiller 部署添加授权

创建ServiceAccount和创建角色绑定

$ kubectl create serviceaccount --namespace kube-system tiller
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

验证

查看Tiller的ServiceAccount，需要跟创建的名字一致：tiller

$ kubectl get deploy --namespace kube-system tiller-deploy -o yaml|grep serviceAccount
 serviceAccount: tiller
 serviceAccountName: tiller

验证pods

$ kubectl -n kube-system get pods|grep tiller
tiller-deploy-b6c45d5bc-jhntj 1/1 Running 0 2d8h

验证版本

$ helm version
Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}

Operator

可以自定义资源类型(CDR)结合自定义控制器实现

样例

apiVersion: app.example.com/v1
kind: AppService
metadata:
 name: nginx-app
spec:
 size: 2
 image: nginx:1.7.9
 ports:
 - port: 80
 targetPort: 80
 nodePort: 30002

Charts

charts项目下stable目录里存放所有软件包内容，由于其默认从多处镜像仓库中拉取镜像，可能存在网络问题，需结合实际情况修改镜像配置，手动安装软件包

推荐配置地址：

quay.io 替换为 quay.azk8s.cn
gcr.io/google-containers 替换为 registry.cn-hangzhou.aliyuncs.com/google-containers

本次需要安装的软件包为prometheus-operator，但其还包含多个依赖软件如下

grafana
kube-state-metrics
prometheus-node-exporter

若需要手动安装则需要按照整理目录如下

prometheus-operator/
├── charts
│   ├── grafana
│   ├── kube-state-metrics
│   └── prometheus-node-exporter
...

手动安装

Etcd监控配置

监控etcd需要额外配置，而且较为复杂，具体步骤如下

查看etcd在集群中的配置，由于本k8s集群基于kubeadm搭建，故etcd通过容器方式运行，查看方法

# 查看etcd所在pod名称
$ kubectl get pod --all-namespaces | grep etcd
kube-system etcd-k8smaster1 1/1 Running 0 7d1h
# 展开etcd详细配置
$ kubectl get pod -n kube-system etcd-k8smaster1 -o yaml

配置内容查看其健康检查端点，--listen-client-urls指明其监听在https端点，故需要配置安全证书才能访问

apiVersion: v1
kind: Pod
metadata:
 annotations:
 kubernetes.io/config.hash: 6119961323d801d05a7dd23e429cda3f
 kubernetes.io/config.mirror: 6119961323d801d05a7dd23e429cda3f
 kubernetes.io/config.seen: "2019-11-05T19:17:28.480984831+08:00"
 kubernetes.io/config.source: file
 creationTimestamp: "2019-11-05T11:18:53Z"
 labels:
 component: etcd
 tier: control-plane
 name: etcd-k8smaster1
 namespace: kube-system
 resourceVersion: "876368"
 selfLink: /api/v1/namespaces/kube-system/pods/etcd-k8smaster1
 uid: 0ccf5e91-ffbe-11e9-a5d2-000c298872e2
spec:
 containers:
 - command:
 - etcd
 - --advertise-client-urls=https://192.168.1.15:2379
 - --cert-file=/etc/kubernetes/pki/etcd/server.crt
 - --client-cert-auth=true
 - --data-dir=/var/lib/etcd
 - --initial-advertise-peer-urls=https://192.168.1.15:2380
 - --initial-cluster=k8smaster1=https://192.168.1.15:2380
 - --key-file=/etc/kubernetes/pki/etcd/server.key
 - --listen-client-urls=https://127.0.0.1:2379,https://192.168.1.15:2379
 - --listen-peer-urls=https://192.168.1.15:2380
 - --name=k8smaster1
 - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
 - --peer-client-cert-auth=true
 - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
 - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
 - --snapshot-count=10000
 - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
 image: registry.cn-hangzhou.aliyuncs.com/imooc/etcd:3.3.10
 imagePullPolicy: IfNotPresent
 livenessProbe:
...
hostIP: 192.168.1.15
...

配置安全证书

k8s创建etcd时会自动创建相关证书，当前集群版本为1.14，证书位于/etc/kubernetes/pki/etcd/中

$ ls /etc/kubernetes/pki/etcd/
ca.crt ca.key healthcheck-client.crt healthcheck-client.key peer.crt peer.key server.crt server.key

可以通过curl命令加载证书测试能否访问健康检查端点

$ curl -k --cert /etc/kubernetes/pki/etcd/healthcheck-client.crt --key /etc/kubernetes/pki/etcd/healthcheck-client.key https://192.168.1.15:2379/metrics

若有如下输出则正常

# HELP etcd_debugging_mvcc_db_compaction_keys_total Total number of db keys compacted.
# TYPE etcd_debugging_mvcc_db_compaction_keys_total counter
etcd_debugging_mvcc_db_compaction_keys_total 1.030132e+06
# HELP etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds Bucketed histogram of db compaction pause duration.
# TYPE etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds histogram
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="1"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="2"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="4"} 0
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="8"} 0
...

创建证书秘钥

关联三个证书文件，创建k8s访问秘钥，秘钥名称为etcd-certs

$ kubectl create secret generic etcd-certs -n monitoring --from-file=/etc/kubernetes/pki/etcd/ca.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key

写入prometheus配置文件values.yaml

注意，此处路径前缀固定写法为/etc/prometheus/secrets/+秘钥名称+文件名称，若写错证书路径则无法在指标中查看到对应的target

... 
 serviceMonitor:
 ## Scrape interval. If not set, the Prometheus default scrape interval is used.
 ##
 interval: ""
 scheme: https
 insecureSkipVerify: true
 serverName: ""
 caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
 certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt
 keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key
...

创建

指定命名空间和文件目录，若命名空间不存在会自动创建

$ helm install /root/prometheus-operator/ --name k8s-prometheus --namespace monitoring

等待一段时间，输出如下则创建成功

NAME: k8s-prometheus
LAST DEPLOYED: Tue Nov 12 20:27:18 2019
NAMESPACE: monitoring
STATUS: DEPLOYED

RESOURCES:
==> v1/Alertmanager
NAME AGE
k8s-prometheus-prometheus-alertmanager 33s

==> v1/ClusterRole
NAME AGE
k8s-prometheus-grafana-clusterrole 34s
k8s-prometheus-prometheus-alertmanager 34s
k8s-prometheus-prometheus-operator 34s
k8s-prometheus-prometheus-operator-psp 34s
k8s-prometheus-prometheus-prometheus 34s
k8s-prometheus-prometheus-prometheus-psp 34s
psp-k8s-prometheus-kube-state-metrics 34s
psp-k8s-prometheus-prometheus-node-exporter 34s

==> v1/ClusterRoleBinding
NAME AGE
k8s-prometheus-grafana-clusterrolebinding 34s
k8s-prometheus-prometheus-alertmanager 34s
k8s-prometheus-prometheus-operator 34s
k8s-prometheus-prometheus-operator-psp 34s
k8s-prometheus-prometheus-prometheus 34s
k8s-prometheus-prometheus-prometheus-psp 34s
psp-k8s-prometheus-kube-state-metrics 34s
psp-k8s-prometheus-prometheus-node-exporter 34s

==> v1/ConfigMap
NAME DATA AGE
k8s-prometheus-grafana 1 34s
k8s-prometheus-grafana-config-dashboards 1 34s
k8s-prometheus-grafana-test 1 34s
k8s-prometheus-prometheus-apiserver 1 34s
k8s-prometheus-prometheus-cluster-total 1 34s
k8s-prometheus-prometheus-controller-manager 1 34s
k8s-prometheus-prometheus-etcd 1 34s
k8s-prometheus-prometheus-grafana-datasource 1 34s
k8s-prometheus-prometheus-k8s-coredns 1 34s
k8s-prometheus-prometheus-k8s-resources-cluster 1 34s
k8s-prometheus-prometheus-k8s-resources-namespace 1 34s
k8s-prometheus-prometheus-k8s-resources-node 1 34s
k8s-prometheus-prometheus-k8s-resources-pod 1 34s
k8s-prometheus-prometheus-k8s-resources-workload 1 34s
k8s-prometheus-prometheus-k8s-resources-workloads-namespace 1 34s
k8s-prometheus-prometheus-kubelet 1 34s
k8s-prometheus-prometheus-namespace-by-pod 1 34s
k8s-prometheus-prometheus-namespace-by-workload 1 34s
k8s-prometheus-prometheus-node-cluster-rsrc-use 1 34s
k8s-prometheus-prometheus-node-rsrc-use 1 34s
k8s-prometheus-prometheus-nodes 1 34s
k8s-prometheus-prometheus-persistentvolumesusage 1 34s
k8s-prometheus-prometheus-pod-total 1 34s
k8s-prometheus-prometheus-pods 1 34s
k8s-prometheus-prometheus-prometheus 1 34s
k8s-prometheus-prometheus-proxy 1 34s
k8s-prometheus-prometheus-scheduler 1 34s
k8s-prometheus-prometheus-statefulset 1 34s
k8s-prometheus-prometheus-workload-total 1 34s

==> v1/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
k8s-prometheus-prometheus-node-exporter 5 5 5 5 5 <none> 33s

==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
k8s-prometheus-grafana 1/1 1 1 33s
k8s-prometheus-kube-state-metrics 1/1 1 1 33s
k8s-prometheus-prometheus-operator 1/1 1 1 33s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
k8s-prometheus-grafana-769d6dfb69-mtbdg 2/2 Running 0 33s
k8s-prometheus-kube-state-metrics-76b8ff78cd-kjxvl 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-jqkvf 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-jxpzm 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-k8n6x 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-kpjzk 1/1 Running 0 33s
k8s-prometheus-prometheus-node-exporter-qc6kv 1/1 Running 0 33s
k8s-prometheus-prometheus-operator-5f894bc75d-kvkvs 2/2 Running 0 33s

==> v1/Prometheus
NAME AGE
k8s-prometheus-prometheus-prometheus 33s

==> v1/PrometheusRule
NAME AGE
k8s-prometheus-prometheus-alertmanager.rules 32s
k8s-prometheus-prometheus-etcd 32s
k8s-prometheus-prometheus-general.rules 32s
k8s-prometheus-prometheus-k8s.rules 32s
k8s-prometheus-prometheus-kube-apiserver.rules 32s
k8s-prometheus-prometheus-kube-prometheus-node-recording.rules 32s
k8s-prometheus-prometheus-kube-scheduler.rules 32s
k8s-prometheus-prometheus-kubernetes-absent 32s
k8s-prometheus-prometheus-kubernetes-apps 32s
k8s-prometheus-prometheus-kubernetes-resources 32s
k8s-prometheus-prometheus-kubernetes-storage 32s
k8s-prometheus-prometheus-kubernetes-system 32s
k8s-prometheus-prometheus-kubernetes-system-apiserver 32s
k8s-prometheus-prometheus-kubernetes-system-controller-manager 32s
k8s-prometheus-prometheus-kubernetes-system-kubelet 32s
k8s-prometheus-prometheus-kubernetes-system-scheduler 32s
k8s-prometheus-prometheus-node-exporter 32s
k8s-prometheus-prometheus-node-exporter.rules 32s
k8s-prometheus-prometheus-node-network 32s
k8s-prometheus-prometheus-node-time 32s
k8s-prometheus-prometheus-node.rules 32s
k8s-prometheus-prometheus-prometheus 32s
k8s-prometheus-prometheus-prometheus-operator 32s

==> v1/Role
NAME AGE
k8s-prometheus-grafana-test 34s

==> v1/RoleBinding
NAME AGE
k8s-prometheus-grafana-test 34s

==> v1/Secret
NAME TYPE DATA AGE
alertmanager-k8s-prometheus-prometheus-alertmanager Opaque 1 34s
k8s-prometheus-grafana Opaque 3 34s

==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
k8s-prometheus-grafana ClusterIP 10.100.203.67 <none> 80/TCP 33s
k8s-prometheus-kube-state-metrics ClusterIP 10.98.35.173 <none> 8080/TCP 33s
k8s-prometheus-prometheus-alertmanager ClusterIP 10.97.189.252 <none> 9093/TCP 33s
k8s-prometheus-prometheus-coredns ClusterIP None <none> 9153/TCP 34s
k8s-prometheus-prometheus-kube-controller-manager ClusterIP None <none> 10252/TCP 34s
k8s-prometheus-prometheus-kube-etcd ClusterIP None <none> 2379/TCP 34s
k8s-prometheus-prometheus-kube-proxy ClusterIP None <none> 10249/TCP 34s
k8s-prometheus-prometheus-kube-scheduler ClusterIP None <none> 10251/TCP 34s
k8s-prometheus-prometheus-node-exporter ClusterIP 10.100.146.95 <none> 9100/TCP 33s
k8s-prometheus-prometheus-operator ClusterIP 10.105.23.58 <none> 8080/TCP,443/TCP 33s
k8s-prometheus-prometheus-prometheus ClusterIP 10.99.7.136 <none> 9090/TCP 34s

==> v1/ServiceAccount
NAME SECRETS AGE
k8s-prometheus-grafana 1 34s
k8s-prometheus-grafana-test 1 34s
k8s-prometheus-kube-state-metrics 1 34s
k8s-prometheus-prometheus-alertmanager 1 34s
k8s-prometheus-prometheus-node-exporter 1 34s
k8s-prometheus-prometheus-operator 1 34s
k8s-prometheus-prometheus-prometheus 1 34s

==> v1/ServiceMonitor
NAME AGE
k8s-prometheus-prometheus-alertmanager 32s
k8s-prometheus-prometheus-apiserver 32s
k8s-prometheus-prometheus-coredns 32s
k8s-prometheus-prometheus-grafana 32s
k8s-prometheus-prometheus-kube-controller-manager 32s
k8s-prometheus-prometheus-kube-etcd 32s
k8s-prometheus-prometheus-kube-proxy 32s
k8s-prometheus-prometheus-kube-scheduler 32s
k8s-prometheus-prometheus-kube-state-metrics 32s
k8s-prometheus-prometheus-kubelet 32s
k8s-prometheus-prometheus-node-exporter 32s
k8s-prometheus-prometheus-operator 32s
k8s-prometheus-prometheus-prometheus 32s

==> v1beta1/ClusterRole
NAME AGE
k8s-prometheus-kube-state-metrics 34s

==> v1beta1/ClusterRoleBinding
NAME AGE
k8s-prometheus-kube-state-metrics 34s

==> v1beta1/MutatingWebhookConfiguration
NAME AGE
k8s-prometheus-prometheus-admission 33s

==> v1beta1/PodSecurityPolicy
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
k8s-prometheus-grafana false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
k8s-prometheus-grafana-test false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,downwardAPI,emptyDir,projected,secret
k8s-prometheus-kube-state-metrics false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false secret
k8s-prometheus-prometheus-alertmanager false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
k8s-prometheus-prometheus-node-exporter false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath
k8s-prometheus-prometheus-operator false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
k8s-prometheus-prometheus-prometheus false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim

==> v1beta1/Role
NAME AGE
k8s-prometheus-grafana 34s

==> v1beta1/RoleBinding
NAME AGE
k8s-prometheus-grafana 34s

==> v1beta1/ValidatingWebhookConfiguration
NAME AGE
k8s-prometheus-prometheus-admission 32s


NOTES:
The Prometheus Operator has been installed. Check its status by running:
 kubectl --namespace monitoring get pods -l "release=k8s-prometheus"

Visit https://github.com/coreos/prometheus-operator for instructions on how
to create & configure Alertmanager and Prometheus instances using the Operator.

升级软件包

一般用于修改yaml配置文件重载应用

$ helm upgrade k8s-prometheus /root/prometheus-operator/ -f /root/prometheus-operator/charts/grafana/values.yaml

删除

使用Helm的优势在于安装软件包和删除都非常方便，通过命令即可快速删除

普通删除，会放入回收站

$ helm delete k8s-prometheus

强制删除，不会放入回收站

$ helm delete k8s-prometheus --purge

对于当前监控组件来说，删除后同时要删除CRD

$ kubectl get crd | grep coreos

删除所有的CRD

$ kubectl delete crd $(kubectl get crd | grep coreos | awk '{print $1}')

访问指标面板

由于当前集群配置了ingress-nginx负载均衡，故新增两个配置文件即可访问Prometheus和Grafana

访问Grafana

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: prom-grafana
 namespace: monitoring
spec:
 rules:
 - host: your.domain
 http:
 paths:
 - backend:
 serviceName: k8s-prometheus-grafana
 servicePort: 80
 path: /

访问Prometheus

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: prometheus
 namespace: monitoring
spec:
 rules:
 - host: your.domain
 http:
 paths:
 - backend:
 serviceName: k8s-prometheus-prometheus-prometheus
 servicePort: web
 path: /

Prometheus

访问Prometheus web站点，首页支持执行PromeQL查询采集指标信息

访问target可见所有指标采集器都处于正常状态

访问监控与告警可查看所有告警规则

Grafana

首次访问会要求登陆

查看用户名和密码

在grafana软件包的values.yaml配置中有如下描述，声明密码被存储在k8s的secret中

# Administrator credentials when not using an existing secret (see below)
adminUser: admin
# adminPassword: strongpassword

# Use an existing secret for the admin user.
admin:
 existingSecret: ""
 userKey: admin-user
 passwordKey: admin-password

查看监控命名空间的秘钥

$ kubectl get secrets -n monitoring
NAME TYPE DATA AGE
alertmanager-k8s-prometheus-prometheus-alertmanager Opaque 1 34m
default-token-hrhgk kubernetes.io/service-account-token 3 2d10h
etcd-certs Opaque 3 2d8h
k8s-prometheus-grafana Opaque 3 34m
k8s-prometheus-grafana-test-token-nnnvk kubernetes.io/service-account-token 3 34m
k8s-prometheus-grafana-token-cdw2n kubernetes.io/service-account-token 3 34m
k8s-prometheus-kube-state-metrics-token-6rxp4 kubernetes.io/service-account-token 3 34m
k8s-prometheus-prometheus-admission Opaque 3 2d10h
k8s-prometheus-prometheus-alertmanager-token-hxqm9 kubernetes.io/service-account-token 3 34m
k8s-prometheus-prometheus-node-exporter-token-vjrcs kubernetes.io/service-account-token 3 34m
k8s-prometheus-prometheus-operator-token-bkhrb kubernetes.io/service-account-token 3 34m
k8s-prometheus-prometheus-prometheus-token-wf2v6 kubernetes.io/service-account-token 3 34m
prometheus-k8s-prometheus-prometheus-prometheus Opaque 1 37m
prometheus-k8s-prometheus-prometheus-prometheus-tls-assets Opaque 0 34m

k8s-prometheus-grafana 即目标秘钥，展示其详情

$ kubectl get secrets -n monitoring k8s-prometheus-grafana -o yaml
apiVersion: v1
data:
 admin-password: b3BlcmF0b3ItYWRtaW4=
 admin-user: cm9vdA==
 ldap-toml: ""
kind: Secret
metadata:
 creationTimestamp: "2019-11-12T12:27:55Z"
 labels:
 app: grafana
 chart: grafana-4.0.2
 heritage: Tiller
 release: k8s-prometheus
 name: k8s-prometheus-grafana
 namespace: monitoring
 resourceVersion: "1033230"
 selfLink: /api/v1/namespaces/monitoring/secrets/k8s-prometheus-grafana
 uid: da7e86d0-0547-11ea-a5d2-000c298872e2
type: Opaque

秘钥默认通过Base64 加密，通过在线网站对该Base64解密即可得到用户名和密码

登录后可查看Grafana已经被配置好的多种可视化面板提供指标展示

点击左上角即可查看已经配置好的指标展示面板

Etcd监控面板

节点监控

计算资源监控