ArgoCD落地GitOps

Sat, 23 Aug 2025 22:49:50 +0800

背景

在过去的DevOps实践中，跨环境的基础设施部署和迁移经常存在遗漏，导致环境差异，部署一个新环境往往需要几天。传统CICD流水线采用“推”模式，将变更推送到不同的环境，推送后也不对目标环境状态进行管理和监控，容易导致推送后部署失败。随着云原生容器化已经成为行业事实标准，GitOps成为一种更可靠的应用部署和基础设置运维的范式。

技术选型

ArgoCD遵循GitOps的核心原则，已经成为落地GitOps的默认技术选型，因此此处不探讨其他选型方案。

原理

Git 作为唯一事实来源（GitOps思想）
自动化和持续同步：Argo CD会持续监控 Git 仓库的状态，并自动将集群的实际状态与 Git 中声明的目标状态进行同步。
- 利用K8s的Kustomize能力完成对集群apply操作

实践

环境

安装ArgoCD

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

检查安装结果

查看argo cd登录密码

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

设计代码库结构

落地ArgoCD通常有以下几种仓库设计范式

单仓库模式
多仓库模式
App of Apps模式

本次采用App of Apps的单仓库模式，仓库结构如下

.
├── apps
│ ├── app-one
│ ├── app-two
├── cluster-infra
│ ├── argocd
│ │ ├── base
│ │ └── overlays
│ ├── ingress
│ │ ├── base
│ │ └── overlays
│ │ └── dev
│ └── prometheus-stack
│ ├── base
│ └── overlays
│ └── dev
└── environments
 └── dev
 └── root.yaml

该结构特点如下

基础设施和应用部署分离（apps和cluster-infra）
环境基础配置和环境定制配置分离（base和overlays）
环境根节点配置（environments控制git仓库配置、应用apps注册）

本次将该代码库存放到阿里云代码仓库中。

环境配置迁移

目前的k3s环境中已经手动运维安装了argocd，现在需要将已安装的配置迁移到git库中

将官方安装配置https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml写入到cluster-infra/argocd/base/argocd.yaml中

配置``cluster-infra/argocd/base/kustomization.yaml`文件，注册同目录的安装文件

resources:
 - argocd.yaml

本次依赖阿里云代码仓库，通过ssh连接，需要覆盖默认的argocd配置，编写cluster-infra/argocd/overlays/with-known-hosts/ssh-known-hosts.patch.yaml

apiVersion: v1 
kind: ConfigMap 
metadata: 
 name: argocd-ssh-known-hosts-cm 
data: 
 ssh_known_hosts: |- 
 # BEGIN >>> 自定义条目（全局生效） 
 codeup.aliyun.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6WbValczSDKC4ktjRzDNC1qINDAViw/oGA8ud7qH5F9XFZ4QAWhZmFBsp+cppBIOM63exjc4O9vbzMrd7AUItXigymGfTqrGt5VykIiaNhjttxSwbZf3N9eDcAvX4gGm3fcm6D1ACvPxvxMsCwOpdz5tDHqDxGOLPAXN+FSAvjdcN4uMz0MXghGh9Q3zSpy8z1TLkAKT2QBnr4q2tsdZKwWON8bBHIPwP3RSAdSUL6bAyotyncMDAvG53kwRKJ3wJ5ZezXqZNNV7EzHds+pvwThtuWtWsBuuRpL/oZiviWlXy6hILoWbkRfgONPwT0faaOn1GLzjHLuqHHfALGOen 
 # END <<< 自定义条目 
 
 # This file was automatically generated by hack/update-ssh-known-hosts.sh. DO NOT EDIT 
 [ssh.github.com]:443 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg= 
 [ssh.github.com]:443 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl 
 [ssh.github.com]:443 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk= 
 bitbucket.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPIQmuzMBuKdWeF4+a2sjSSpBK0iqitSQ+5BM9KhpexuGt20JpTVM7u5BDZngncgrqDMbWdxMWWOGtZ9UgbqgZE= 
 bitbucket.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO 
 bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQeJzhupRu0u0cdegZIa8e86EG2qOCsIsD1Xw0xSeiPDlCr7kq97NLmMbpKTX6Esc30NuoqEEHCuc7yWtwp8dI76EEEB1VqY9QJq6vk+aySyboD5QF61I/1WeTwu+deCbgKMGbUijeXhtfbxSxm6JwGrXrhBdofTsbKRUsrN1WoNgUa8uqN1Vx6WAJw1JHPhglEGGHea6QICwJOAr/6mrui/oB7pkaWKHj3z7d1IC4KWLtY47elvjbaTlkN04Kc/5LFEirorGYVbt15kAUlqGM65pk6ZBxtaO3+30LVlORZkxOh+LKL/BvbZ/iRNhItLqNyieoQj/uh/7Iv4uyH/cV/0b4WDSd3DptigWq84lJubb9t/DnZlrJazxyDCulTmKdOR7vs9gMTo+uoIrPSb8ScTtvw65+odKAlBj59dhnVp9zd7QUojOpXlL62Aw56U4oO+FALuevvMjiWeavKhJqlR7i5n9srYcrNV7ttmDw7kf/97P5zauIhxcjX+xHv4M= 
 github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg= 
 github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl 
 github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk= 
 gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY= 
 gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf 
 gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9 
 ssh.dev.azure.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H 
 vs-ssh.visualstudio.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

配置``cluster-infra/argocd/overlays/with-known-hosts/kustomization.yaml`文件，注册hosts patch文件

resources: 
 - ../../base 
 
patchesStrategicMerge: 
 - ssh-known-hosts.patch.yaml

按这样配置后，argocd会先执行base的安装文件apply基础环境，再使用ssh-known-hosts.patch.yaml覆盖ssh的配置，保证环境与git配置一致。

接下来，继续在apps根目录注册应用，配置environments/dev/kustomization.yaml

resources:
 - ../../cluster-infra/argocd/overlays/with-known-hosts

编写根目录git配置

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: dev-root 
 namespace: argocd 
spec: 
 project: default 
 source: 
 repoURL: git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git 
 targetRevision: master 
 path: environments/dev 
 kustomize: 
 enableHelm: true 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: argocd 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

到此，所有配置迁移完成，将git仓库提交到远程后，在master节点添加私钥

argocd repo add git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git --ssh-private-key-path /root/.ssh/gitops_ed25519

同步dev根应用

kubectl apply -f dev-root.yaml

本地使用kubectl暴露临时端口，再使用argocd cli连接集群查看状态

kubectl port-forward svc/argocd-server -n argocd --address 0.0.0.0 30443:443
argocd login xxxx:30443 --username admin
argocd app list

部署ingress

一直使用临时端口+命令行查看应用状态也不方便，官方提供webui界面查看部署状态，考虑使用ingress暴露固定域名访问

ingress的部署也遵循相同步骤

配置官方安装文件
配置环境个性化
1. 端口暴露
根应用添加配置

将官方https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml安装文件配置到cluster-infra/ingress/base/ingress-nginx.yaml

配置端口暴露cluster-infra/ingress/overlays/dev/ingress-nginx-controller-nodeport.yaml

apiVersion: v1 
kind: Service 
metadata: 
 name: ingress-nginx-controller 
 namespace: ingress-nginx 
spec: 
 type: NodePort 
 ports: 
 - name: http 
 port: 80 
 targetPort: http 
 protocol: TCP 
 nodePort: 30080 
 - name: https 
 port: 443 
 targetPort: https 
 protocol: TCP 
 nodePort: 30443

根应用添加配置

resources: 
 - ../../cluster-infra/ingress/overlays/dev 
 - ../../cluster-infra/argocd/overlays/with-known-hosts

在argocd环境目录下添加ingress域名配置

apiVersion: networking.k8s.io/v1 
kind: Ingress 
metadata: 
 name: argocd-ingress 
 namespace: argocd 
 annotations: 
 nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" 
spec: 
 ingressClassName: nginx 
 rules: 
 - host: xxxx 
 http: 
 paths: 
 - path: / 
 pathType: Prefix 
 backend: 
 service: 
 name: argocd-server 
 port: 
 number: 443

git提交后即可自动同步并创建ingress

访问域名查看argocd webui

部署Prometheus+Grafana

为了完善集群监控，argocd 支持通过helm charts部署应用

配置cluster-infra/prometheus-stack/base/prometheus-stack.yaml

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: prometheus-stack 
 namespace: argocd 
 annotations: 
 argocd.argoproj.io/sync-options: ServerSideApply=true 
spec: 
 project: default 
 source: 
 repoURL: https://prometheus-community.github.io/helm-charts 
 chart: kube-prometheus-stack 
 targetRevision: 75.16.1 
 helm: 
 valueFiles: 
 - values.yaml 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: monitoring 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

注意，由于Helm仓库部署prometheus时伴随大量CRD资源文件apply，需要开启ServerSideApply模式避免通过接口apply导致字符超长报错。参考：https://argocd.devops.gold/user-guide/sync-options/#_9

使用helm模式安装的应用无法使用patchesStrategicMerge策略覆盖，目前覆盖个性化配置只能将所有内容写到一个资源文件内，因此dev环境的个性化配置values如下

prometheus: 
 prometheusSpec: 
 retention: 3d 
 retentionSize: "2Gi" 
 resources: 
 requests: 
 memory: 128Mi 
 limits: 
 memory: 1Gi 
 storageSpec: 
 volumeClaimTemplate: 
 spec: 
 resources: 
 requests: 
 storage: 5Gi 
 ingress: 
 enabled: true 
 ingressClassName: nginx 
 annotations: {} 
 hosts: 
 - prometheus.fjy8018.top 
 paths: 
 - / 
 tls: 
 - hosts: 
 - xxx.xxx
 secretName: prometheus-tls 
 prometheusSpec: 
 externalUrl: https://xxx.xxx 
 routePrefix: / 
 
grafana: 
 adminPassword: "2EvwJJ0NOLzhYa" 
 resources: 
 requests: 
 memory: 64Mi 
 limits: 
 memory: 512Mi 
 ingress: 
 enabled: true 
 ingressClassName: nginx 
 annotations: {} 
 hosts: 
 - xxx.xxx 
 tls: 
 - hosts: 
 - xxx.xxx
 secretName: grafana-tls

需要在dev中覆盖base的资源文件，通过multi repo模式合并helm配置

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: prometheus-stack 
 namespace: argocd 
 annotations: 
 argocd.argoproj.io/sync-options: ServerSideApply=true 
spec: 
 project: default 
 sources: 
 - repoURL: https://prometheus-community.github.io/helm-charts 
 chart: kube-prometheus-stack 
 targetRevision: 75.16.1 
 helm: 
 valueFiles: 
 - $values/cluster-infra/prometheus-stack/overlays/dev/values-dev.yaml 
 - repoURL: 'git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git' 
 targetRevision: master 
 ref: values 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: monitoring 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

prometheus依赖CRD提前部署的namespace需要注册为独立应用提前配置好

apiVersion: v1 
kind: Namespace 
metadata: 
 name: monitoring

命名空间的配置和普通应用一致

apiVersion: argoproj.io/v1alpha1 
kind: Application 
metadata: 
 name: common-ns 
 namespace: argocd 
spec: 
 project: default 
 source: 
 repoURL: git@codeup.aliyun.com:xxxx/fjy8018/pve-k3s-gitops.git 
 targetRevision: master 
 path: cluster-infra/common-ns 
 destination: 
 server: https://kubernetes.default.svc 
 namespace: argocd 
 syncPolicy: 
 automated: 
 prune: true 
 selfHeal: true

最终根目录应用配置如下

resources: 
 - ../../cluster-infra/ingress/overlays/dev 
 - ../../cluster-infra/argocd/overlays/with-known-hosts 
 - ../../cluster-infra/prometheus-stack/overlays/dev 
 - common-ns-app.yaml

同步后效果如下

查看监控部署拓扑

通过域名访问grafana

ArgoCD on F嘉阳