GitLab on F嘉阳

k8s环境下GitLab+Helm+GitLab Runner Java项目CICD落地实践

Wed, 27 Nov 2019 10:24:00 +0800

背景

目前使用5台服务器搭建了Kubernetes集群环境，监控、日志采集均已落地，业务也手工迁移到集群中顺利运行，故需要将原本基于原生docker环境的CICD流程迁移到Kubernetes集群中

优势

Kubernetes集群实现CICD有几个显著优势

Deployment 天然支持滚动部署、结合其他Kubernetes特性还能实现蓝绿部署、金丝雀部署等
新版本的GitLab与GitLab Runner天然支持Kubernetes集群，支持runner自动伸缩，减小资源占用

环境

Kubernetes版本：1.14

GitLab版本：12.2.5

GitLab-Runner版本：12.1.0

Docker环境版本：17.03.1

GitLab-Runner部署

配置介绍

原始环境的gitlab runner通过手动执行官网提供的注册命令和启动命令，分成两部部署，需要较多的手工操作，而在Kubernetes中，其支持使用Helm一键部署，官方文档如下

GitLab Runner Helm Chart

其实官方文档的指引并不清晰，许多配置在文档中没有介绍用法，推荐去其源码仓库查看详细的参数使用文档

The Kubernetes executor

其中介绍了几个关键配置，在后面修改工程的ci配置文件时会用到

使用DinD方式构建已经不再推荐

官方文档介绍

Use docker-in-docker workflow with Docker executor
The second approach is to use the special docker-in-docker (dind) Docker image with all tools installed (docker) and run the job script in context of that image in privileged mode.

Note: docker-compose is not part of docker-in-docker (dind). To use docker-compose in your CI builds, follow the docker-compose installation instructions.

Danger: By enabling --docker-privileged, you are effectively disabling all of the security mechanisms of containers and exposing your host to privilege escalation which can lead to container breakout. For more information, check out the official Docker documentation on Runtime privilege and Linux capabilities.

Docker-in-Docker works well, and is the recommended configuration, but it is not without its own challenges:

When using docker-in-docker, each job is in a clean environment without the past history. Concurrent jobs work fine because every build gets it’s own instance of Docker engine so they won’t conflict with each other. But this also means jobs can be slower because there’s no caching of layers.

By default, Docker 17.09 and higher uses --storage-driver overlay2 which is the recommended storage driver. See Using the overlayfs driver for details.

Since the docker:19.03.1-dind container and the Runner container don’t share their root filesystem, the job’s working directory can be used as a mount point for child containers. For example, if you have files you want to share with a child container, you may create a subdirectory under /builds/$CI_PROJECT_PATH and use it as your mount point (for a more thorough explanation, check issue #41227):

总之使用DinD进行容器构建并非不可行，但面临许多问题，例如使用overlay2网络需要Docker版本高于 17.09

Using docker:dind
Running the docker:dind also known as the docker-in-docker image is also possible but sadly needs the containers to be run in privileged mode. If you’re willing to take that risk other problems will arise that might not seem as straight forward at first glance. Because the docker daemon is started as a service usually in your .gitlab-ci.yaml it will be run as a separate container in your Pod. Basically containers in Pods only share volumes assigned to them and an IP address by which they can reach each other using localhost. /var/run/docker.sock is not shared by the docker:dind container and the docker binary tries to use it by default.

To overwrite this and make the client use TCP to contact the Docker daemon, in the other container, be sure to include the environment variables of the build container:

DOCKER_HOST=tcp://localhost:2375 for no TLS connection.

DOCKER_HOST=tcp://localhost:2376 for TLS connection.

Make sure to configure those properly. As of Docker 19.03, TLS is enabled by default but it requires mapping certificates to your client. You can enable non-TLS connection for DIND or mount certificates as described in Use Docker In Docker Workflow wiht Docker executor

在Docker 19.03.1版本之后默认开启了TLS配置，在构建的环境变量中需要声明，否则报连接不上docker的错误，并且使用DinD构建需要runner开启特权模式，以访问主机的资源，并且由于使用了特权模式，在Pod中对runner需要使用的资源限制将失效

使用Kaniko构建Docker镜像

目前官方提供另一种方式在docker容器中构建并推送镜像，实现更加优雅，可以实现无缝迁移，那就是kaniko

Building a Docker image with kaniko

其优势官网描述如下

在Kubernetes集群中构建Docker映像的另一种方法是使用kaniko。iko子

允许您构建没有特权访问权限的映像。

无需Docker守护程序即可工作。

在后面的实践中会使用两种方式构建Docker镜像，可根据实际情况选择

使用Helm部署

拉取Helm Gitlab-Runner仓库到本地，修改配置

GitLab Runner

将原有的gitlab-runner配置迁移到Helm中，迁移后如下

image: alpine-v12.1.0
imagePullPolicy: IfNotPresent
gitlabUrl: https://gitlab.fjy8018.top/
runnerRegistrationToken: "***"
unregisterRunners: true
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
rbac:
 create: true
 clusterWideAccess: false
metrics:
 enabled: true
 listenPort: 9090
runners:
 image: ubuntu:16.04
 imagePullSecrets:
 - name: registry-secret
 locked: false
 tags: "k8s"
 runUntagged: true
 privileged: true
 pollTimeout: 180
 outputLimit: 4096
 cache: {}
 builds: {}
 services: {}
 helpers: {}
resources:
 limits:
 memory: 2048Mi
 cpu: 1500m
 requests:
 memory: 128Mi
 cpu: 200m
affinity: {}
nodeSelector: {}
tolerations: []
hostAliases:
 - ip: "192.168.1.13"
 hostnames:
 - "gitlab.fjy8018.top"
 - ip: "192.168.1.30"
 hostnames:
 - "harbor.fjy8018.top"
podAnnotations: {}

其中配置了私钥、内网harbor地址、harbor拉取资源私钥，资源限制策略

GitLab-Runner选择可能导致的坑

选择runner镜像为alpine-v12.1.0，这一点单独说一下，目前最新的runner版本为12.5.0，但其有许多问题，alpine新版镜像在Kubernetes中间断发生无法解析DNS的问题，反映到GitLab-Runner中就是Could not resolve host和server misbehaving

查阅解决方法

通过查询发现，其官方仓库还有多个相关issue没有关闭

官方gitlab：Kubernetes runner: Could not resolve host

stackoverflow：Gitlab Runner is not able to resolve DNS of Gitlab Server

给出的解决方案无一例外都是降级到alpine-v12.1.0

We had same issue for couple of days. We tried change CoreDNS config, move runners to different k8s cluster and so on. Finally today i checked my personal runner and found that i’m using different version. Runners in cluster had gitlab/gitlab-runner:alpine-v12.3.0, when mine had gitlab/gitlab-runner:alpine-v12.0.1. We added line
image: gitlab/gitlab-runner:alpine-v12.1.0
in values.yaml and this solved problem for us

其问题的根源应该在于alpine基础镜像对Kubernetes 集群支持有问题，

ndots breaks DNS resolving #64924

docker-alpine仓库对应也有未关闭的issue，其中就提到了关于DNS解析超时和异常的问题

DNS Issue #255

安装

一行命令安装即可

$ helm install /root/gitlab-runner/ --name k8s-gitlab-runner --namespace gitlab-runner

输出如下

NAME: k8s-gitlab-runner
LAST DEPLOYED: Tue Nov 26 21:51:57 2019
NAMESPACE: gitlab-runner
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
k8s-gitlab-runner-gitlab-runner 5 0s

==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
k8s-gitlab-runner-gitlab-runner 0/1 1 0 0s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
k8s-gitlab-runner-gitlab-runner-744d598997-xwh92 0/1 Pending 0 0s

==> v1/Role
NAME AGE
k8s-gitlab-runner-gitlab-runner 0s

==> v1/RoleBinding
NAME AGE
k8s-gitlab-runner-gitlab-runner 0s

==> v1/Secret
NAME TYPE DATA AGE
k8s-gitlab-runner-gitlab-runner Opaque 2 0s

==> v1/ServiceAccount
NAME SECRETS AGE
k8s-gitlab-runner-gitlab-runner 1 0s


NOTES:

Your GitLab Runner should now be registered against the GitLab instance reachable at: "https://gitlab.fjy8018.top/"

查看gitlab admin页面，发现已经有一个runner成功注册

工程配置

DinD方式构建所需配置

如果原本的ci文件是基于19.03 DinD镜像构建的则需要加上TLS相关配置

image: docker:19.03

variables:
 DOCKER_DRIVER: overlay
 DOCKER_HOST: tcp://localhost:2375
 DOCKER_TLS_CERTDIR: ""
...

其余配置保持不变，使用DinD构建

Kubectl和Kubernetes权限配置

由于使用k8s集群，而通过集群部署需要使用kubectl客户端，故手动创建了一个kubectl docker镜像，使用gitlab触发dockerhub构建，构建内容公开透明，可放心使用，如有其它版本的构建需求也可提pull request，会在后面补充，目前用到的只有1.14.0

fjy8018/kubectl

有kubectl客户端，还需要配置连接TLS和连接账户

为了保障安全，新建一个专门访问该工程命名空间的ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
 name: hmdt-gitlab-ci
 namespace: hmdt

利用集群提供的RBAC机制，为该账户授予该命名空间的admin权限

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
 name: hmdt-gitlab-role
 namespace: hmdt
subjects:
 - kind: ServiceAccount
 name: hmdt-gitlab-ci
 namespace: hmdt
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: admin

创建后在查询其在k8s集群中生成的唯一名称，此处为hmdt-gitlab-ci-token-86n89

$ kubectl describe sa hmdt-gitlab-ci -n hmdt
Name: hmdt-gitlab-ci
Namespace: hmdt
Labels: <none>
Annotations: kubectl.`Kubernetes`.io/last-applied-configuration:
 {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"hmdt-gitlab-ci","namespace":"hmdt"}}
Image pull secrets: <none>
Mountable secrets: hmdt-gitlab-ci-token-86n89
Tokens: hmdt-gitlab-ci-token-86n89
Events: <none>

然后根据上面的Secret找到CA证书

$ kubectl get secret hmdt-gitlab-ci-token-86n89 -n hmdt -o json | jq -r '.data["ca.crt"]' | base64 -d

再找到对应的 Token

$ kubectl get secret hmdt-gitlab-ci-token-86n89 -n hmdt -o json | jq -r '.data.token' | base64 -d

Kubernetes关联GitLab配置

进入gitlab Kubernetes集群配置页面，填写相关信息，让gitlab自动连接上集群环境

注意，需要将此处取消勾选，否则gitlab会自动创建新的用户账户，而不使用已经创建好的用户账户，在运行过程中会报无权限错误

不取消导致的报错如下，gitlab创建了新的用户账户hmdt-prod-service-account，但没有操作指定命名空间的权限

GitLab环境配置

创建环境

名称和url可以按需自定义

CI脚本配置

最终配置CI文件如下，该文件使用DinD方式构建Dockerfile

image: docker:19.03

variables:
 MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode -Dmaven.test.skip=true"
 MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
 DOCKER_DRIVER: overlay
 DOCKER_HOST: tcp://localhost:2375
 DOCKER_TLS_CERTDIR: ""
 SPRING_PROFILES_ACTIVE: docker
 IMAGE_VERSION: "1.8.6"
 DOCKER_REGISTRY_MIRROR: "https://XXX.mirror.aliyuncs.com"

stages:
 - test
 - package
 - review
 - deploy

maven-build:
 image: maven:3-jdk-8
 stage: test
 retry: 2
 script:
 - mvn $MAVEN_CLI_OPTS clean package -U -B -T 2C
 artifacts:
 expire_in: 1 week
 paths:
 - target/*.jar

maven-scan:
 stage: test
 retry: 2
 image: maven:3-jdk-8
 script:
 - mvn $MAVEN_CLI_OPTS verify sonar:sonar

maven-deploy:
 stage: deploy
 retry: 2
 image: maven:3-jdk-8
 script:
 - mvn $MAVEN_CLI_OPTS deploy

docker-harbor-build:
 image: docker:19.03
 stage: package
 retry: 2
 services:
 - name: docker:19.03-dind
 alias: docker
 before_script:
 - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
 script:
 - docker build --pull -t "$CI_REGISTRY_IMAGE:$IMAGE_VERSION" .
 - docker push "$CI_REGISTRY_IMAGE:$IMAGE_VERSION"
 - docker logout $CI_REGISTRY

deploy_live:
 image: fjy8018/kubectl:v1.14.0
 stage: deploy
 retry: 2
 environment:
 name: prod
 url: https://XXXX
 script:
 - kubectl version
 - kubectl get pods -n hmdt
 - cd manifests/
 - sed -i "s/__IMAGE_VERSION_SLUG__/${IMAGE_VERSION}/" deployment.yaml
 - kubectl apply -f deployment.yaml
 - kubectl rollout status -f deployment.yaml
 - kubectl get pods -n hmdt

若需要使用Kaniko构建Dockerfile，则配置如下

注意，其中依赖的镜像gcr.io/kaniko-project/executor:debug属于谷歌镜像仓库，可能存在无法拉取的情况

image: docker:19.03

variables:
 MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode -Dmaven.test.skip=true"
 MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
 DOCKER_DRIVER: overlay
 DOCKER_HOST: tcp://localhost:2375
 DOCKER_TLS_CERTDIR: ""
 SPRING_PROFILES_ACTIVE: docker
 IMAGE_VERSION: "1.8.6"
 DOCKER_REGISTRY_MIRROR: "https://XXX.mirror.aliyuncs.com"

cache:
 paths:
 - target/

stages:
 - test
 - package
 - review
 - deploy

maven-build:
 image: maven:3-jdk-8
 stage: test
 retry: 2
 script:
 - mvn $MAVEN_CLI_OPTS clean package -U -B -T 2C
 artifacts:
 expire_in: 1 week
 paths:
 - target/*.jar

maven-scan:
 stage: test
 retry: 2
 image: maven:3-jdk-8
 script:
 - mvn $MAVEN_CLI_OPTS verify sonar:sonar

maven-deploy:
 stage: deploy
 retry: 2
 image: maven:3-jdk-8
 script:
 - mvn $MAVEN_CLI_OPTS deploy


docker-harbor-build:
 stage: package
 retry: 2
 image:
 name: gcr.io/kaniko-project/executor:debug
 entrypoint: [""]
 script:
 - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
 - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$IMAGE_VERSION

deploy_live:
 image: fjy8018/kubectl:v1.14.0
 stage: deploy
 retry: 2
 environment:
 name: prod
 url: https://XXXX
 script:
 - kubectl version
 - kubectl get pods -n hmdt
 - cd manifests/
 - sed -i "s/__IMAGE_VERSION_SLUG__/${IMAGE_VERSION}/" deployment.yaml
 - kubectl apply -f deployment.yaml
 - kubectl rollout status -f deployment.yaml
 - kubectl get pods -n hmdt

执行流水线

runner自动扩缩容

Kubernetes中的runner会根据任务多少自动扩缩容，目前配置的上限为10个

Grafana也能监控到集群在构建过程中的资源使用情况

使用`DinD`构建`Dockerfile`结果

使用`Kaniko`构建`Dockerfile`的结果

部署结果

执行部署时gitlab会自动注入配置好的kubectl config

构建结果

部署完成后可在环境配置页中查看部署结果，只有成功的部署才会被记录

GitLab Api使用

Wed, 06 Nov 2019 15:01:04 +0800

使用场景

为了进行规范化配置管理工作，基于gitlab规范分支管理流程，经常需要新建分支、设置保护分支和操作分支封板等，若只有几个工程则直接在gitlab web界面上操作即可，一旦工程数量增多，则会消耗大量时间，故考虑可通过gitlab 提供的web api编写脚本进行自动化操作

Api指南

此处摘要部分官网API文档指南，所有api操作都需要在gitlab上申请访问令牌，该令牌应当至少包含api访问权限

获取工程列表

支持多种参数过滤，非必传

Get a list of all visible projects across GitLab for the authenticated user. When accessed without authentication, only public projects with “simple” fields are returned.
GET /projects
Attribute Type Required Description

archived boolean no Limit by archived status

visibility string no Limit by visibility public, internal, or private

order_by string no Return projects ordered by id, name, path, created_at, updated_at, or last_activity_at fields. Default is created_at

sort string no Return projects sorted in asc or desc order. Default is desc

search string no Return list of projects matching the search criteria

simple boolean no Return only limited fields for each project. This is a no-op without authentication as then only simple fields are returned.

owned boolean no Limit by projects explicitly owned by the current user

membership boolean no Limit by projects that the current user is a member of

starred boolean no Limit by projects starred by the current user

statistics boolean no Include project statistics

with_custom_attributes boolean no Include custom attributes in response (admins only)

with_issues_enabled boolean no Limit by enabled issues feature

with_merge_requests_enabled boolean no Limit by enabled merge requests feature

with_programming_language string no Limit by projects which use the given programming language

wiki_checksum_failed boolean no Limit projects where the wiki checksum calculation has failed (Introduced in GitLab Premium 11.2)

repository_checksum_failed boolean no Limit projects where the repository checksum calculation has failed (Introduced in GitLab Premium 11.2)

min_access_level integer no Limit by current user minimal access level

Attribute	Type	Required	Description
`archived`	boolean	no	Limit by archived status
`visibility`	string	no	Limit by visibility `public`, `internal`, or `private`
`order_by`	string	no	Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`
`sort`	string	no	Return projects sorted in `asc` or `desc` order. Default is `desc`
`search`	string	no	Return list of projects matching the search criteria
`simple`	boolean	no	Return only limited fields for each project. This is a no-op without authentication as then only simple fields are returned.
`owned`	boolean	no	Limit by projects explicitly owned by the current user
`membership`	boolean	no	Limit by projects that the current user is a member of
`starred`	boolean	no	Limit by projects starred by the current user
`statistics`	boolean	no	Include project statistics
`with_custom_attributes`	boolean	no	Include custom attributes in response (admins only)
`with_issues_enabled`	boolean	no	Limit by enabled issues feature
`with_merge_requests_enabled`	boolean	no	Limit by enabled merge requests feature
`with_programming_language`	string	no	Limit by projects which use the given programming language
`wiki_checksum_failed`	boolean	no	Limit projects where the wiki checksum calculation has failed (Introduced in GitLab Premium 11.2)
`repository_checksum_failed`	boolean	no	Limit projects where the repository checksum calculation has failed (Introduced in GitLab Premium 11.2)
`min_access_level`	integer	no	Limit by current user minimal access level

分支操作

文档地址

创建分支

需要传递仓库ID、源分支名、新分支名称

Create a new branch in the repository.
POST /projects/:id/repository/branches
Parameters:

Attribute Type Required Description

id integer yes ID or URL-encoded path of the project owned by the authenticated user.

branch string yes Name of the branch.

ref string yes Branch name or commit SHA to create branch from.

Example request:
curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/repository/branches?branch=newbranch&ref=master

Attribute	Type	Required	Description
`id`	integer	yes	ID or URL-encoded path of the project owned by the authenticated user.
`branch`	string	yes	Name of the branch.
`ref`	string	yes	Branch name or commit SHA to create branch from.

保护分支

分支保护等级说明

Valid access levels

The access levels are defined in the ProtectedRefAccess.allowed_access_levels method. Currently, these levels are recognized:
0 => No access
30 => Developer access
40 => Maintainer access
60 => Admin access

需要传递分支名称、访问级别等信息

Protects a single repository branch or several project repository branches using a wildcard protected branch.
POST /projects/:id/protected_branches
curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/projects/5/protected_branches?name=*-stable&push_access_level=30&merge_access_level=30&unprotect_access_level=40'
Attribute Type Required Description

id integer/string yes The ID or URL-encoded path of the project owned by the authenticated user

name string yes The name of the branch or wildcard

push_access_level string no Access levels allowed to push (defaults: 40, maintainer access level)

merge_access_level string no Access levels allowed to merge (defaults: 40, maintainer access level)

unprotect_access_level string no Access levels allowed to unprotect (defaults: 40, maintainer access level)

allowed_to_push array no Array of access levels allowed to push, with each described by a hash

allowed_to_merge array no Array of access levels allowed to merge, with each described by a hash

allowed_to_unprotect array no Array of access levels allowed to unprotect, with each described by a hash

code_owner_approval_required boolean no Prevent pushes to this branch if it matches an item in the CODEOWNERS file. (defaults: false)

Attribute	Type	Required	Description
`id`	integer/string	yes	The ID or URL-encoded path of the project owned by the authenticated user
`name`	string	yes	The name of the branch or wildcard
`push_access_level`	string	no	Access levels allowed to push (defaults: `40`, maintainer access level)
`merge_access_level`	string	no	Access levels allowed to merge (defaults: `40`, maintainer access level)
`unprotect_access_level`	string	no	Access levels allowed to unprotect (defaults: `40`, maintainer access level)
`allowed_to_push`	array	no	Array of access levels allowed to push, with each described by a hash
`allowed_to_merge`	array	no	Array of access levels allowed to merge, with each described by a hash
`allowed_to_unprotect`	array	no	Array of access levels allowed to unprotect, with each described by a hash
`code_owner_approval_required`	boolean	no	Prevent pushes to this branch if it matches an item in the `CODEOWNERS` file. (defaults: false)

取消保护分支

需要传递仓库ID、分支名称

Unprotects the given protected branch or wildcard protected branch.
DELETE /projects/:id/protected_branches/:name
curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/projects/5/protected_branches/*-stable'
Attribute Type Required Description

id integer/string yes The ID or URL-encoded path of the project owned by the authenticated user

name string yes The name of the branch

Attribute	Type	Required	Description
`id`	integer/string	yes	The ID or URL-encoded path of the project owned by the authenticated user
`name`	string	yes	The name of the branch

删除分支

需要传递仓库ID和分支名称

Delete a branch from the repository.

Note: In the case of an error, an explanation message is provided.
DELETE /projects/:id/repository/branches/:branch
Parameters:

Attribute Type Required Description

id integer/string yes ID or URL-encoded path of the project owned by the authenticated user.

branch string yes Name of the branch.

Example request:
curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/repository/branches/newbranch

Attribute	Type	Required	Description
`id`	integer/string	yes	ID or URL-encoded path of the project owned by the authenticated user.
`branch`	string	yes	Name of the branch.

接口测试

使用postman对接口进行测试

获取工程列表

通过该接口可以查询所有工程的ID，注意，该接口默认使用了分页

脚本编写

批量创建分支

定义工程ID数组，对数组进行遍历，并设置基础分支变量，方便调整，此处默认写死基础分支，有需要也可以通过变量传递

#!/usr/bin/env bash

# 访问令牌
PRIVATE_TOKEN="XXXX"
# 分支名称
branch_name=$1

base_branch="BasicBranch"

# 项目ID
projectids=(47 48 50 51 52 54 55 56 58 59 60 62 63 64 65 67 68 69 70 71 74 75 89 90 91 92 93 95 103 104 113 114 115 143)

for id in "${projectids[@]}" ; do
	curl --request POST --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://10.139.6.26:7077/api/v4/projects/$id/repository/branches?branch=$branch_name&ref=$base_branch"
done

批量保护分支

此处写死了分支访问级别，可参照上文的参数进行配置

#!/usr/bin/env bash

# 访问令牌
PRIVATE_TOKEN="XXXX"
# 分支名称
branch_name=$1

# 项目ID
projectids=(47 48 50 51 52 54 55 56 58 59 60 62 63 64 65 67 68 69 70 71 74 75 89 90 91 92 93 95 103 104 113 114 115 143)

for id in "${projectids[@]}" ; do
 # 访问权限均为 Maintainer access
 curl --request POST --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://10.139.6.26:7077/api/v4/projects/$id/protected_branches?name=$branch_name&push_access_level=40&merge_access_level=40&unprotect_access_level=40"
done

批量取消保护分支

调用对应的删除接口即可

#!/usr/bin/env bash

# 访问令牌
PRIVATE_TOKEN="XXXX"
# 分支名称
branch_name=$1

# 项目ID
projectids=(47 48 50 51 52 54 55 56 58 59 60 62 63 64 65 67 68 69 70 71 74 75 89 90 91 92 93 95 103 104 113 114 115 143)

for id in "${projectids[@]}" ; do
 # 访问权限均为 Maintainer access
 curl --request DELETE --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://10.139.6.26:7077/api/v4/projects/$id/protected_branches/$branch_name"
done

批量分支封板

分支封板即设置访问级别为no one

#!/usr/bin/env bash

# 访问令牌
PRIVATE_TOKEN="XXXX"
# 分支名称
branch_name=$1

# 项目ID
projectids=(47 48 50 51 52 54 55 56 58 59 60 62 63 64 65 67 68 69 70 71 74 75 89 90 91 92 93 95 103 104 113 114 115 143)

for id in "${projectids[@]}" ; do
 # 分支封板
 curl --request POST --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://10.139.6.26:7077/api/v4/projects/$id/protected_branches?name=$branch_name&push_access_level=0&merge_access_level=0&unprotect_access_level=40"
done

批量删除分支

删除分支属于危险操作，操作需谨慎，因为不管是否为保护分支，都可以直接删除，而且通过api操作不会弹出确认操作

#!/usr/bin/env bash

# 访问令牌
PRIVATE_TOKEN="XXXX"
# 分支名称
branch_name=$1

# 项目ID
projectids=(47 48 50 51 52 54 55 56 58 59 60 62 63 64 65 67 68 69 70 71 74 75 89 90 91 92 93 95 103 104 113 114 115 143)

for id in "${projectids[@]}" ; do
 echo "删除项目ID：$id 的 $branch_name 分支"
 curl --request DELETE --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://10.139.6.26:7077/api/v4/projects/$id/repository/branches/$branch_name"
done

DevOps实践心得一

Fri, 16 Aug 2019 19:32:00 +0800

初遇DevOps

最早接触到DevOps是在2018年8月26日，那时首先接触到的就是大名鼎鼎的Jenkins，当时看了多篇关于持续集成和持续部署的文章，蠢蠢欲动，便开始了第一套DevOps环境的搭建

Jenkins

第一套DevOps是用Jenkins+Ansible+GitLab搭建的，那时只是了解Docker的好处，并没有使用Docker搭建整套环境，还是基于传统虚拟服务器搭建的整套流程。

Jenkins不仅可以编写groove脚本、pipeline 流水线等，还能与maven集成自动构建，深深感受到了Jenkins的强大，同时大量的用户为其贡献了许多插件，有很多实践问题都可以在网上搜索到解决方案

搭建过程：

最后顺利实现了静态网站持续集成、持续部署，详见：相关文章

Docker+GitLab

随着容器化技术在国内越来越流行，容器普及度越来越高，其带来的环境隔离极大推动了DevOps的落地，使得无需使用python环境隔离，可以直接利用容器技术实现跨平台和环境隔离，并且利于迁移和部署

于是在2019年4月开始研究GitLab官方文档，发现其早已原生支持流水线，并依托其GitLab-runner可以直接实现整套DevOps环境

因此，经过不断摸索，完整的搭建的基于Docker和GitLab的DevOps环境，搭建过程见：

由于国内当时对GitLab流水线使用的并不多，因此遇到的很多坑都需要翻查官方文档和国外的实践记录，比如：

DevOps 带来的好处

2019年7月将手上的一个小型项目尝试使用完整的DevOps流程开发，开发小组选择敏捷开发方法，所有的开发工具都尽量的适应快速的需求变化，同时投入非常少量的开发人员，分工明确，从一开始一周发布一次到最后一天可以发布多次

开发人员已经完全从运维中脱离，全团队没有专门的运维人员，极大的提高了开发效率，精简了人员构成，一旦构建失败，开发人员可以根据邮件通知的内容排查上线失败问题，调整代码和配置后重新提交

面板提供了流水线统计图，如果出现频繁的失败还是需要人工介入，查看问题原因

当前的问题

尽管使用了基于Docker+GitLab的流水线实现了完整的DevOps流程，但对于监控和日志部分仍然处在很多问题

尽管上线的是单体应用，服务器和应用的健康状态都无法得知，也没有相应的告警
应用的日志没有被外部服务收集，无法进行快速分析
没有实现完整的测试流程，生成测试覆盖率报告等
没有实现0停机上线，在容器切换时仍需停机1-2分钟
没有实现灰度上线、蓝绿部署等，上线失败需要手动回滚

未来探索方向

全面容器化服务治理方向
- 国内对容器化有非常多的实践
- kubernetes作为容器调度成为大多数服务治理的选择
- 容器隔离、迁移和伸缩带来的巨大优势
容器与传统应用部署结合方式
- 以半全自动方式落地DevOps，采用Mesos+marathon方式同时兼容传统服务部署和容器化部署
- 兼容分布式服务治理
- 国外有大量的实践案例
- 日志收集体系完备

Docker 19.03 版本对原Gitlab-Runner的影响

Sun, 28 Jul 2019 13:09:00 +0800

版本变化始末

在 2019.07.24 Docker社区发布了19.03的新版本，在此前版本号为18.0x

新版本带来的坑

原本没太在意docker版本的变化，因为多数情况都是兼容旧版的所有特性和功能的

结果在近期的CICD构建中，在ci脚本没有变化的情况下突然所有runner的构建都失败了

随后立即采取以下措施尝试修复

工程版本回退到上一次可用状态
更换runner服务器和容器
更改ci脚本
重启docker服务

均无效

执行构建步骤出现了以往没有出现过的警告信息

Running with gitlab-runner 11.8.0 (4745a6f3)
 on docker-in-docker-runner 4iAG2sYM
Using Docker executor with image docker:stable ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:fd0c64832f7e46b63a180e6000dbba7ad7a63542c5764841cba73429ba74a39e for docker:dind ...
Waiting for services to be up and running...

*** WARNING: Service runner-4iAG2sYM-project-89-concurrent-0-docker-0 probably didn't start properly.

Health check error:
service "runner-4iAG2sYM-project-89-concurrent-0-docker-0-wait-for-service" timeout

Health check container logs:

...

2019-07-24T03:11:35.365911823Z time="2019-07-24T03:11:35.365537741Z" level=info msg="API listen on [::]:2376"
2019-07-24T03:11:35.366107674Z time="2019-07-24T03:11:35.365626706Z" level=info msg="API listen on /var/run/docker.sock"

*********

Pulling docker image docker:stable ...
Using docker image sha256:c4154a2b47a18fe9437956ab981bd5924b19e7ae3eb3ed60c42cf8dfa394d550 for docker:stable ...

最终报错信息

Downloading artifacts for maven-build (280)...
Downloading artifacts from coordinator... ok id=280 responseStatus=200 OK token=L9iuC5Ss
$ docker build -t $CONTAINER_IMAGE:$IMAGE_VERSION .
Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?
ERROR: Job failed: exit code 1

寻找解决方法

该错误在早期版本就出现过，但与本次情况均不符合，以往通过挂载容器路径即可，而本次情况是runner服务和构建脚本均不变的情况下突然出现构建失败的情况

最终在社区ISSUS讨论中发现了相关讨论，并且问题发生时间相近

Jobs failing with “Cannot connect to the Docker daemon”

官方开发者解释当天几个小时前发布的新版本 19.03默认开启了TLS确实会影响已有的Docker in Docker 模式的构建环境

同时官方开发人员建议自行部署的runner建议开启TLS功能

该开发者在另一个讨论中提供了19.03环境下的runner启动和配置样板，同时还有样例ci脚本

Update DinD example for 19.03

解决方法

最好可以卸载旧版本的Docker，尽管运行runner和dind环境都基于docker容器中
加入TLS相关runner注册配置

此处采用更新Docker版本并使用最新版的runner镜像进行注册

更新docker版本

官方文档

卸载旧docker

$ sudo yum remove docker \
 docker-client \
 docker-client-latest \
 docker-common \
 docker-latest \
 docker-latest-logrotate \
 docker-logrotate \
 docker-engine

配置生产环境的依赖仓库

$ sudo yum-config-manager \
 --add-repo \
 https://download.docker.com/linux/centos/docker-ce.repo

官方建议的安装新版Docker命令

$ sudo yum install docker-ce docker-ce-cli containerd.io

但经过这次事件后个人不建议使用该命令，而是应该手动指定所有Docker环境版本号，防止以后版本更新导致不兼容问题

查看当前可以安装的所有版本

$ sudo yum list docker-ce --showduplicates | sort -r
 * updates: repos.lax.quadranet.com
Loaded plugins: fastestmirror
 * extras: mirror.hostduplex.com
 * epel: d2lzkl7pfhq30w.cloudfront.net
 * elrepo-kernel: repos.lax-noc.com
docker-ce.x86_64 3:19.03.1-3.el7 docker-ce-stable
docker-ce.x86_64 3:19.03.0-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.8-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.7-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.6-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.5-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.4-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.3-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.2-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.1-3.el7 docker-ce-stable
docker-ce.x86_64 3:18.09.0-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.3.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.2.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.1.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.0.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.12.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.12.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.3.ce-1.el7 docker-ce-stable
docker-ce.x86_64 17.03.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.0.ce-1.el7.centos docker-ce-stable
Determining fastest mirrors
 * base: mirror.hostduplex.com
Available Packages

指定最新版本

$ sudo yum install docker-ce-19.03.1 docker-ce-cli-19.03.1 containerd.io

启动docker

$ sudo systemctl start docker

设置开机自启

$ sudo systemctl enable docker

安装Gitlab Runner

官方对新版启用TLS的注册样例

sudo gitlab-runner register -n \
 --url https://gitlab.com/ \
 --registration-token REGISTRATION_TOKEN \
 --executor docker \
 --description "My Docker Runner" \
 --docker-image "docker:stable" \
 --docker-privileged \
 --docker-volumes "/certs/client"

此处采用容器化注册runner命令为

$ docker run --rm -t -i \
-v /etc/gitlab-runner/config:/etc/gitlab-runner \
gitlab/gitlab-runner:alpine-v12.1.0 register \
--url "https://gitlab.com/" \
--registration-token REGISTRATION_TOKEN \
--executor docker \
--non-interactive \
--run-untagged \
--tag-list "Runner,Docker" \
--run-untagged="true" \
--locked="false" \
--description "My Docker Runner" \
--docker-image "docker:stable" \
--docker-privileged \
--docker-volumes "/certs/client"

主要的变化至于增加了证书挂载--docker-volumes "/certs/client"配置，同时指定了runner的版本号

启动runner

$ docker run --name gitlab-runner --restart always -d \
-v /etc/gitlab-runner/config:/etc/gitlab-runner \
-v /var/run/docker.sock:/var/run/docker.sock \
gitlab/gitlab-runner:alpine-v12.1.0

修改构建脚本

官方对新版启用TLS的构建脚本样例，指定了构建时使用的docker版本与外部一直，防止不兼容的问题

同时加入了TLS环境变量配置

 image: docker:19.03

 variables:
 # When using dind service we need to instruct docker, to talk with
 # the daemon started inside of the service. The daemon is
 # available with a network connection instead of the default
 # /var/run/docker.sock socket. docker:19.03-dind does this
 # automatically by setting the DOCKER_HOST in
 # https://github.com/docker-library/docker/blob/d45051476babc297257df490d22cbd806f1b11e4/19.03/docker-entrypoint.sh#L23-L29
 #
 # The 'docker' hostname is the alias of the service container as described at
 # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services.
 #
 # Note that if you're using the Kubernetes executor, the variable should be set to
 # tcp://localhost:2376/ because of how the Kubernetes executor connects services
 # to the job container
 # DOCKER_HOST: tcp://localhost:2376/
 #
 # When using dind, it's wise to use the overlayfs driver for
 # improved performance.
 DOCKER_DRIVER: overlay2
 # Specify to Docker where to create the certificates, Docker will
 # create them automatically on boot, and will create
 # `/certs/client` that will be shared between the service and
 # build container.
 DOCKER_TLS_CERTDIR: "/certs"

 services:
 - docker:19.03-dind

 before_script:
 - echo $DOCKER_HOST
 - docker info

 build:
 stage: build
 script:
 - docker build -t my-docker-image .

在此前自己的构建脚本是按照官网提供的样板，没有指定版本，因此会导致每次构建都会使用当前最新的版本作为基础，最终导致外部Docker和内部Docker容器版本不一致

官网示例DinD构建脚本

image: docker:stable
services:
 - docker:dind
...

按照社区配置修改后重新构建则成功

总结

对于Docker等云服务平台仍然需要特别注意版本变化，为了防止不兼容的情况，在生产环境尽量不要使用latest版本标签，而是应该手动指定所有参与构建的版本环境

DevOps GitLab CICD 实践3——CI文件编写

Wed, 03 Apr 2019 16:32:00 +0800

前置步骤：

Docker 版 GitLab CICD 实践1——GitLab 部署 [[Docker 版 GitLab CICD 实践]]
Docker 版 GitLab CICD 实践2——Runner 部署 [[Docker 版 GitLab CICD 实践——Runner 部署]]

官方文档

编写结构类似Jenkins pineline流水线

GitLab CI/CD Pipeline Configuration Reference

GitLab CI/CD Pipeline Configuration Reference
GitLab CI/CD pipelines are configured using a YAML file called .gitlab-ci.yml within each project.

The .gitlab-ci.yml file defines the structure and order of the pipelines and determines:

What to execute using GitLab Runner.

What decisions to make when specific conditions are encountered. For example, when a process succeeds or fails.

This topic covers CI/CD pipeline configuration. For other CI/CD configuration information, see:

GitLab CI/CD Variables, for configuring the environment the pipelines run in.

GitLab Runner advanced configuration, for configuring GitLab Runner.

We have complete examples of configuring pipelines:

For a quick introduction to GitLab CI, follow our quick start guide.

For a collection of examples, see GitLab CI/CD Examples.

To see a large .gitlab-ci.yml file used in an enterprise, see the .gitlab-ci.yml file for gitlab-ce.

目标

maven项目自动打包
~~自动测试~~
自动镜像构建并上传

官方参数表列出了作业的可用参数：

关键词	描述
`script`	由Runner执行的Shell脚本。
`image`	使用泊坞窗图像。也可用：`image:name`和`image:entrypoint`。
`services`	使用docker services图像。也可用：`services:name`，`services:alias`，`services:entrypoint`，和`services:command`。
`before_script`	覆盖在作业之前执行的一组命令。
`after_script`	覆盖作业后执行的一组命令。
`stages`	定义管道中的阶段。
`stage`	定义作业阶段（默认值:) `test`。
`only`	创建作业时限制。也可用：`only:refs`，`only:kubernetes`，`only:variables`，和`only:changes`。
`except`	在未创建作业时限制。也可用：`except:refs`，`except:kubernetes`，`except:variables`，和`except:changes`。
`tags`	用于选择Runner的标签列表。
`allow_failure`	让工作失败。失败的作业无助于提交状态。
`when`	什么时候开始工作。也可用：`when:manual`和`when:delayed`。
`environment`	作业部署到的环境的名称。也可用：`environment:name`，`environment:url`，`environment:on_stop`，和`environment:action`。
`cache`	后续运行之间应缓存的文件列表。也可用：`cache:paths`，`cache:key`，`cache:untracked`，和`cache:policy`。
`artifacts`	成功附加到作业的文件和目录列表。也可用：`artifacts:paths`，`artifacts:name`，`artifacts:untracked`，`artifacts:when`，`artifacts:expire_in`，`artifacts:reports`，和`artifacts:reports:junit`。在GitLab 企业版，这些都是可供选择：`artifacts:reports:codequality`，`artifacts:reports:sast`，`artifacts:reports:dependency_scanning`，`artifacts:reports:container_scanning`，`artifacts:reports:dast`，`artifacts:reports:license_management`，和`artifacts:reports:performance`。
`dependencies`	作业所依赖的其他作业，以便您可以在它们之间传递工件。
`coverage`	给定作业的代码覆盖率设置。
`retry`	在发生故障的情况下，可以自动重试作业的次数和次数。
`parallel`	应该并行运行多少个作业实例。
`trigger`	定义下游管道触发器。
`include`	允许此作业包含外部YAML文件。也可用：`include:local`，`include:file`，`include:template`，和`include:remote`。
`extends`	此作业将继承的配置条目。
`pages`	上传作业结果以用于GitLab Pages。
`variables`	在作业级别定义作业变量。

注：参数types和type被弃用。

参考

官方构建案例

Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD

参考其脚本

Configure GitLab CI/CD to deploy your application
Now we need to add the GitLab CI/CD configuration file (.gitlab-ci.yml) to our project’s root. This is how GitLab figures out what commands need to be run whenever code is pushed to our repository. We will add the following .gitlab-ci.yml file to the root directory of the repository, GitLab will detect it automatically and run the steps defined once we push our code:
image: java:8

stages:
 - build
 - deploy

build:
 stage: build
 script: ./mvnw package
 artifacts:
 paths:
 - target/demo-0.0.1-SNAPSHOT.jar

production:
 stage: deploy
 script:
 - curl --location "https://cli.run.pivotal.io/stable?release=linux64-binary&source=github" | tar zx
 - ./cf login -u $CF_USERNAME -p $CF_PASSWORD -a api.run.pivotal.io
 - ./cf push
 only:
 - master

配置

全局变量

进入工程CI设置配置全局变量脚本，包含镜像仓库登陆名称、密码、打包名称等

配置全局变量目的在于配置脚本中不应该包含密码等敏感信息

若希望使用GitLab内置环境变量，可参考官方表格

GitLab CI/CD environment variables

CI脚本

结合可用参数和样例配置，根据已经存在的SpringBoot项目编写响应的CI脚本.gitlab-ci.yml

image: docker:stable
services:
 - docker:dind

variables:
 DOCKER_DRIVER: overlay
 SPRING_PROFILES_ACTIVE: gitlab-ci

stages:
 - build
 - package

maven-build:
 image: maven:3-jdk-8
 stage: build
 script: "mvn package -B"
 artifacts:
 paths:
 - target/*.jar

docker-build:
 stage: package
 script:
 - docker build -t $CONTAINER_IMAGE:latest .
 - docker login -u $DOCKER_HUB_USER -p $DOCKER_HUB_PASS
 - docker push $CONTAINER_IMAGE:latest

该脚本对于满足脚本目标的工程都可复用，若需要测试，则增加相关的测试步骤即可

此处由于工程包含外部依赖关系，不在构建时测试

自动构建

默认的触发构建事件为commit等

触发后会自动执行任务

Maven自动触发构建，自动镜像打包

自动上传（推送）镜像

执行结果

若执行失败则会发送邮件给开发人员

CICD全套流程目标实现！

总结

不管使用GitLab CICD或者是使用Jenkins+Ansible的方式，目标都在于践行DevOps，打通开发与运维流程，一旦配置好CICD流程，往后开发人员便不再操心打包、构建等操作，可以专注开发

DevOps GitLab CICD 实践2——Runner 部署

Wed, 03 Apr 2019 16:31:00 +0800

前置步骤：DevOps GitLab CICD 实践1——GitLab 部署

官方文档

文档地址

虽然有官方指引，但个人感觉指引的不够清晰，导致初次配置可能频繁失败

Run GitLab Runner in a container
This is how you can run GitLab Runner inside a Docker container.

General GitLab Runner Docker image usage
GitLab Runner Docker images (based on Ubuntu or Alpine Linux) are designed as wrappers around the standard gitlab-runner command, like if GitLab Runner was installed directly on the host.

The general rule is that every GitLab Runner command that normally would be executed as:
gitlab-runner [Runner command and options...]
can be executed with:
docker run [chosen docker options...] gitlab/gitlab-runner [Runner command and options...]
For example, getting the top-level help information for GitLab Runner command could be executed as:
docker run --rm -t -i gitlab/gitlab-runner --help

NAME:
 gitlab-runner - a GitLab Runner

USAGE:
 gitlab-runner [global options] command [command options] [arguments...]

VERSION:
 10.7.0 (7c273476)

(...)
In short, the gitlab-runner part of the command is replaced with docker run [docker options] gitlab/gitlab-runner, while the rest of Runner’s command stays as it is described in the register documentation. The only difference is that the gitlab-runner command is executed inside of a Docker container.

Docker image installation and configuration
Install Docker first:
 curl -sSL https://get.docker.com/ | sh
You need to mount a config volume into the gitlab-runner container to be used for configs and other resources:
 docker run -d --name gitlab-runner --restart always \
 -v /srv/gitlab-runner/config:/etc/gitlab-runner \
 -v /var/run/docker.sock:/var/run/docker.sock \
 gitlab/gitlab-runner:latest
Tip: On macOS, use /Users/Shared instead of /srv.

Or, you can use a config container to mount your custom data volume:
 docker run -d --name gitlab-runner-config \
 -v /etc/gitlab-runner \
 busybox:latest \
 /bin/true
And then, run the Runner:
 docker run -d --name gitlab-runner --restart always \
 -v /var/run/docker.sock:/var/run/docker.sock \
 --volumes-from gitlab-runner-config \
 gitlab/gitlab-runner:latest
Register the runner you just launched by following the instructions in the Docker section of Registering Runners. The runner won’t pick up any jobs until it’s registered.

安装步骤

获取Gitlab Runner秘钥

可以通过Gitlab管理员账号获取，也可以让每一个用户自行配置

普通用户查看秘钥

进入任意一个仓库的设置中，查看CICD配置

准备注册专用Runner令牌

管理员查看令牌

进入总设置页面，配置全局Runner令牌

Runner注册

由于Runner一般运行复杂构建、打包任务，推荐配置在性能、带宽更大的机房

准备Docker环境

$ curl -sSL https://get.docker.com/ | sh
$ systemctl start docker

注册

可以根据需要选择注册Runner类型

同时，为了方便配置，使用单行注册并且关闭交互

单行注册官方文档

命令说明：

-v挂载的目的是为了将注册后的配置文件持久化，用于运行容器
--rm指定容器运行结束后自动删除停止的容器
-it指定使用命令行交互方式运行，便于查看注册结果

$ docker run --rm -it \
 -v /www/wwwroot/gitlab/srv/gitlab-runner/config:/etc/gitlab-runner \
 gitlab/gitlab-runner:alpine-v11.8.0 register \
 --non-interactive \
 --executor "docker" \
 --docker-image docker:stable \
 --url "Gitlab URL" \
 --registration-token "令牌" \
 --description "描述" \
 --tag-list "标签1,标签2" \
 --run-untagged \
 --docker-privileged \
 --locked="false"

注册成功提示

此时管理面板显示新的Runner已经注册

Runner 启动

挂载本地配置信息并启动Runner

$ docker run -d --name gitlab-runner --restart always \
 	-v /www/wwwroot/gitlab/srv/gitlab-runner/config:/etc/gitlab-runner \
 -v /var/run/docker.sock:/var/run/docker.sock \
 gitlab/gitlab-runner:alpine-v11.8.0

此时可见Runner已经保持和Gitlab的联系

DevOps GitLab CICD 实践1——GitLab 部署

Wed, 03 Apr 2019 16:30:00 +0800

配置目标

邮件提示
GitHub第三方授权登陆
GitLab Runner
Docker私服注册

官方介绍

目前微服务盛行环境下，服务部署优先考虑Docker方式，便于迁移和弹性伸缩

官方镜像介绍 GitLab Docker images

GitLab Docker images
Both GitLab CE and EE are in Docker Hub:

GitLab CE Docker image

GitLab EE Docker image

The GitLab Docker images are monolithic images of GitLab running all the necessary services on a single container.

In the following examples we are using the image of GitLab CE. To use GitLab EE instead of GitLab CE, replace the image name to gitlab/gitlab-ee:latest.

If you want to use the latest RC image, use gitlab/gitlab-ce:rc or gitlab/gitlab-ee:rc for GitLab CE and GitLab EE respectively.

The GitLab Docker images can be run in multiple ways:

Run the image in Docker Engine

Install GitLab into a cluster

Install GitLab using docker-compose

docker-compose 脚本

此处选择社区版(CE)安装，同时为了便于参数配置，使用docker-compose方式编写脚本文件

Install GitLab using docker-compose
With Docker compose you can easily configure, install, and upgrade your Docker-based GitLab installation.
Install Docker Compose
Create a docker-compose.yml file (or download an example):
 web:
 image: 'gitlab/gitlab-ce:latest'
 restart: always
 hostname: 'gitlab.example.com'
 environment:
 GITLAB_OMNIBUS_CONFIG: |
 external_url 'https://gitlab.example.com'
 # Add any other gitlab.rb configuration here, each on its own line
 ports:
 - '80:80'
 - '443:443'
 - '22:22'
 volumes:
 - '/srv/gitlab/config:/etc/gitlab'
 - '/srv/gitlab/logs:/var/log/gitlab'
 - '/srv/gitlab/data:/var/opt/gitlab'
Make sure you are in the same directory as docker-compose.yml and run docker-compose up -d to start GitLab
Read “Pre-configure Docker container” to see how the GITLAB_OMNIBUS_CONFIG variable works.

Below is another docker-compose.yml example with GitLab running on a custom HTTP and SSH port. Notice how the GITLAB_OMNIBUS_CONFIG variables match the ports section:
web:
 image: 'gitlab/gitlab-ce:latest'
 restart: always
 hostname: 'gitlab.example.com'
 environment:
 GITLAB_OMNIBUS_CONFIG: |
 external_url 'http://gitlab.example.com:9090'
 gitlab_rails['gitlab_shell_ssh_port'] = 2224
 ports:
 - '9090:9090'
 - '2224:22'
 volumes:
 - '/srv/gitlab/config:/etc/gitlab'
 - '/srv/gitlab/logs:/var/log/gitlab'
 - '/srv/gitlab/data:/var/opt/gitlab'
This is the same as using --publish 9090:9090 --publish 2224:22.

官方提示说明Docker CE版基于Omnibus版本，故环境配置也可参考相关文档

Omnibus文档目录

Installation and Configuration using omnibus package
Note: This section describes the commonly used configuration settings. Check configuration section of the documentation for complete configuration settings.

Installing GitLab

Manually downloading and installing a GitLab package

Setting up a domain name/URL for the GitLab Instance so that it can be accessed easily

Enabling HTTPS

Enabling notification EMails

Enabling replying via email

Installing and configuring postfix

Enabling container registry on GitLab

You will require SSL certificates for the domain used for container registry

Enabling GitLab Pages

If you want HTTPS enabled, you will have to get wildcard certificates

Enabling Elasticsearch

GitLab Mattermost Set up the Mattermost messaging app that ships with Omnibus GitLab package.

GitLab Prometheus Set up the Prometheus monitoring included in the Omnibus GitLab package.

GitLab High Availability Roles

结合配置目标编写yaml文件

注意：

此处邮件使用163邮箱（官方没有提供163邮箱支持案例）
Docker私服公钥执行从私服上获取
由于特殊原因，目标配置未启动SSL安全连接，但GitLab可以通过简单配置支持SSL并自动更新证书

配置文档

Let’s Encrypt Integration
Primary GitLab Instance

Note: Introduced in GitLab version 10.5 and disabled by default. Enabled by default in GitLab version 10.7 and later if external_url is set with the httpsprotocol and no certificates are configured.

Note: In order for Let’s Encrypt verification to work correctly, ports 80 and 443 will need to be accessible to the Let’s Encrypt servers that run the validation. Also note that the validation currently does not work with non-standard ports.
Caution Administrators installing or upgrading to GitLab version 10.7 or later and do not plan on using Let’s Encrypt should set the following in /etc/gitlab/gitlab.rb to disable:
letsencrypt['enable'] = false
Add the following entries to /etc/gitlab/gitlab.rb to enable Let’s Encrypt support for the primary domain:
letsencrypt['enable'] = true # GitLab 10.5 and 10.6 require this option
external_url "https://gitlab.example.com"	 # Must use https protocol
letsencrypt['contact_emails'] = ['foo@email.com'] # Optional

生成163邮箱授权密码

生成GitHub授权秘钥

最终配置

version: '3.1'

services:

 gitlab:
 environment:
 GITLAB_OMNIBUS_CONFIG: |
 external_url '外部访问地址'
 gitlab_rails['gitlab_shell_ssh_port'] = 外部SSH端口
 registry_external_url 'Docker私服地址'
 registry_nginx['ssl_certificate'] = "Docker 私服CA证书 crt文件"
 registry_nginx['ssl_certificate_key'] = "Docker 私服公钥 pem文件"
 gitlab_rails['smtp_enable'] = true
 gitlab_rails['smtp_address'] = "smtp.163.com"
 gitlab_rails['smtp_port'] = 465
 gitlab_rails['smtp_user_name'] = "邮件发送者名称"
 gitlab_rails['gitlab_email_from'] = '邮件发送地址'
 gitlab_rails['smtp_password'] = "授权密码"
 gitlab_rails['smtp_domain'] = "163.com"
 gitlab_rails['smtp_authentication'] = "login"
 gitlab_rails['smtp_enable_starttls_auto'] = true
 gitlab_rails['smtp_tls'] = true
 gitlab_rails['omniauth_enabled'] = true
 gitlab_rails['omniauth_allow_single_sign_on'] = true
 gitlab_rails['omniauth_block_auto_created_users'] = true
 gitlab_rails['omniauth_providers'] = [
 {
 "name" => "github",
 "app_id" => "Client ID",
 "app_secret" => "Client Secret",
 "url" => "https://github.com/",
 "args" => { "scope" => "user:email" }
 }
 ]
 image: gitlab/gitlab-ce:latest
 hostname: gitlab.fjy8018.top
 restart: always
 networks:
 - devops-service-bridge
 ports:
 - '8443:443'
 - '8099:8099'
 - '8022:22'
 volumes:
 - ./srv/gitlab/config:/etc/gitlab
 - ./srv/gitlab/logs:/var/log/gitlab
 - ./srv/gitlab/data:/var/opt/gitlab
 - /etc/docker/certs.d:/etc/docker/certs.d


networks:
 devops-service-bridge:
 driver: bridge

Jenkins+GitLab+Ansible实现静态网站自动部署

Sat, 01 Sep 2018 12:45:00 +0800

环境准备

参考前几篇文章配置后实现以下环境

部署好Jenkins服务器一台(192.168.163.133),并在Jenkins服务器上部署ansible，且与目标主机(192.168.163.132)做了免密登录
可以访问的GitLab服务器一台，并注册了Jenkins相关账号且在Jenkins系统设置中配置完毕
一台工作环境主机，用于代码编写，该主机可以通过域名访问目标主机(192.168.163.132)

编写ansible playbook

搭建基础工程目录结构

从仓库下载之前编写的ansible工程目录结构

若没有也可自行创建，最终文件目录结构如下

└─ansible-playbook-repo
 │ deploy.yml
 │
 ├─inventory
 │ dev
 │ prod
 │
 └─roles
 └─nginx
 ├─files
 │ │ health_check.sh
 │ │ index.html
 │ │
 │ └─website
 │
 ├─tasks
 │ main.yml
 │
 └─templates
 nginx.conf.j2

文件编写

工程入口deploy.yml编写

- hosts: "nginx"
 gather_facts: true
 remote_user: root
 roles:
 - nginx

编写开发环境配置文件inventory/dev，此处dev与prod配置一致，若实际需求可自行修改，用于最后jenkins参数化构建的选项 inventory/dev文件内容

[nginx]
test.fjy8018.top

[nginx:vars]
server_name=test.fjy8018.top
port=80
user=ansible
worker_processes=4
max_open_file=65505
root=/www

inventory/prod文件内容

[nginx]
test.fjy8018.top

[nginx:vars]
server_name=test.fjy8018.top
port=80
user=ansible
worker_processes=4
max_open_file=65505
root=/www

编写健康检查shell脚本health_check.sh roles/nginx/files/health_check.sh文件内容

#!/bin/bash

# 接受命令行参数
URL=$1

# 访问目标地址，判断是否能正常访问并输出
curl -Is http://$URL > /dev/null && echo "远程主机状态正常" || echo "远程主机已停止"

添加静态网站文件

/roles/nginx/files/website为静态网页模板目录，里面存放从网上下载的静态网站模板配置nginx指向该资源 /roles/nginx/templates/nginx.conf.j2文件内容

# For more information on configuration, see: 
user {{ user }}; 
worker_processes {{ worker_processes }}; 
 
error_log /var/log/nginx/error.log; 
 
pid /var/run/nginx.pid; 
 
events { 
 worker_connections {{ max_open_file }}; 
} 
 
 
http { 
 include /etc/nginx/mime.types; 
 default_type application/octet-stream; 
 
 log_format main '$remote_addr - $remote_user [$time_local] "$request" ' 
 '$status $body_bytes_sent "$http_referer" ' 
 '"$http_user_agent" "$http_x_forwarded_for"'; 
 
 access_log /var/log/nginx/access.log main; 
 
 sendfile on; 
 #tcp_nopush on; 
 
 #keepalive_timeout 0; 
 keepalive_timeout 65; 
 
 #gzip on; 
 
 # Load config files from the /etc/nginx/conf.d directory 
 # The default server is in conf.d/default.conf 
 #include /etc/nginx/conf.d/*.conf; 
 server { 
 listen {{ port }} default_server; 
 server_name {{ server_name }}; 
 
 #charset koi8-r; 
 
 #access_log logs/host.access.log main; 
 
 location / { 
 root {{ root }}/website; 
 index index.html index.htm; 
 } 
 
 error_page 404 /404.html; 
 location = /404.html { 
 root /usr/share/nginx/html; 
 } 
 
 # redirect server error pages to the static page /50x.html 
 # 
 error_page 500 502 503 504 /50x.html; 
 location = /50x.html { 
 root /usr/share/nginx/html; 
 } 
 
 } 
 
}

编写主任务文件

主任务文件main.yml内容如下

- name: 关闭系统防火墙
 service: name=firewalld state=stopped

- name: 关闭强制访问控制策略
 selinux: state=disabled

- name: 设置yum源
 yum: pkg=epel-release state=latest

- name: 写入nginx配置文件
 template: src=roles/nginx/templates/nginx.conf.j2 dest=/etc/nginx/nginx.conf

- name: 创建nginx根目录文件夹
 file: 'path={{ root }} state=directory owner={{ user }} group={{ user }} mode=0755'

- name: 文件拷贝
 copy: 'remote_src=no src=roles/nginx/files/website dest=/www/website owner={{ user }} mode=0755'

- name: 启动nginx
 service: name=nginx state=restarted

- name: 运行健康检查
 shell: "sh roles/nginx/files/health_check.sh {{ server_name }}"
 # 本地执行
 delegate_to: localhost
 # 参数传递
 register: health_status

- debug: msg="{{ health_status.stdout }}"

Git提交

提交变更到版本仓库GitLab

git add .
git commit -m "添加模板文件"
git push origin master

Jenkins创建并执行

创建自由风格构建任务，添加参数化构建、添加选项参数和文本参数

添加Git仓库地址和账号

添加Shell构建，Shell内容为

#!/bin/bash
# 关闭命令行扩展环境，避免环境参数日志干扰
set +x
# 加载虚拟环境
source /home/deploy/.py3-a2.5-env/bin/activate
# 加载ansible到当前用户目录
source /home/deploy/.py3-a2.5-env/ansible/hacking/env-setup -q

cd $WORKSPACE
ls -a
# 查看版本
ansible --version
ansible-playbook --version

# 引入资源清单，测试远程命令
ansible-playbook -i inventory/$deploy_env ./deploy.yml -e project=nginx -e branch=$branch -e env=$deploy_env

保存并构建

结果

构建成功

访问测试

在工作主机上访问目标地址+资源路径即可，访问通过

后续构建

后期只要在工作主机变更代码推送到git仓库，再到Jenkins的web界面点击参数化构建即可让Jenkins自动拉取最新分支并执行Ansible脚本

GitLab汉化

Tue, 12 Jun 2018 22:30:00 +0800

准备文件

目前GitLab汉化由xhang维护，故直接克隆其工程到本地目录即可

git clone https://gitlab.com/xhang/gitlab.git

当前GitLab最新版为10.8.4，其他版本自行看release切换到对应的分支或者下载对应的发布版本即可下载完毕为文件夹

修改系统名称别名

CentOS 7中出于安全考虑，设置了部分别名，防止误操作，但在此会导致复制文件需要反复输入y，故将其关闭通过alias查看当前的别名设置，若有alias cp='cp -i'，则说明每次复制文件都会出现提示信息

使用vi ~/.bashrc修改配置文件

source ~/.bashrc使配置文件生效，但还需要重启会话，否则修改的命令可能依旧无效再次查看，此时别名配置修改成功

配置汉化

此处为初次安装汉化，故与补丁方式汉化不同，直接覆盖文件即可覆盖前备份一下

cp -r /opt/gitlab/embedded/service/gitlab-rails{,.ori}

覆盖文件

cp gitlab/* /opt/gitlab/embedded/service/gitlab-rails/ -rf

重启GitLab

gitlab-ctl restart

成功

英文

中文

GitLab跨版本升级

Tue, 12 Jun 2018 21:50:00 +0800

当前版本

当前GitLab版本由宝塔自动安装的8.8.5,但截止至2018-06-12,CentOS的GitLab官方最新版本为10.8.4,显然版本相差较大,故打算跨版本升级至最新版

步骤

升级指南

GitLab社区版不支持直接跨版本升级，需按照官方指南一步步向上升级，若版本越旧，需要跨的版本就越多

下载相关安装包

安装包国内选择清华镜像站速度较快，教育网还有蜜汁加成

若升级其他版本也可自行Ctrl+F

wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-8.17.7-ce.0.el7.x86_64.rpm
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-9.5.8-ce.0.el7.x86_64.rpm
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-10.1.4-ce.0.el7.x86_64.rpm
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-10.8.4-ce.0.el7.x86_64.rpm

升级

关闭GitLab核心服务

gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq
gitlab-ctl stop nginx

安装8.17.7

rpm -Uvh gitlab-ce-8.17.7-ce.0.el7.x86_64.rpm

重启服务

若之前从宝塔安装后未配置GitLab的Nginx启动命令则需要自行配置后才能正常启动，操作见文章GitLab修改配置后nginx无法启动重启服务

[root@localhost ~]# gitlab-ctl restart
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.3.6/lib/omnibus-ctl.rb:533: warning: Insecure world writable dir /www/server/nvm/versions/node/v8.11.2/bin in PATH, mode 040777
ok: run: gitlab-workhorse: (pid 39809) 1s
ok: run: logrotate: (pid 39821) 0s
ok: run: nginx: (pid 39828) 0s
ok: run: postgresql: (pid 39835) 0s
ok: run: redis: (pid 39844) 0s
ok: run: sidekiq: (pid 39849) 1s
ok: run: unicorn: (pid 39854) 0s

重启后稍等片刻即可访问，查看版本号，已经更新为8.17.7，后续步骤同上

可能的报错

部分CentOS版本较低未自带policycoreutils-python组件可能无法升级至10.x,需要手动

yum -y install policycoreutils-python

完成

更新到10.x后明显启动项都变多了 [root@localhost ~]# gitlab-ctl restart /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.5.0/lib/omnibus-ctl.rb:684: warning: Insecure world writable dir /www/server/nvm/versions/node/v8.11.2/bin in PATH, mode 040777 ok: run: alertmanager: (pid 45330) 1s ok: run: gitaly: (pid 45336) 0s ok: run: gitlab-monitor: (pid 45346) 1s ok: run: gitlab-workhorse: (pid 45362) 0s ok: run: logrotate: (pid 45369) 1s ok: run: nginx: (pid 45378) 0s ok: run: node-exporter: (pid 45383) 0s ok: run: postgres-exporter: (pid 45387) 1s ok: run: postgresql: (pid 45394) 0s ok: run: prometheus: (pid 45476) 1s ok: run: redis: (pid 45484) 0s ok: run: redis-exporter: (pid 45488) 1s ok: run: sidekiq: (pid 45493) 0s ok: run: unicorn: (pid 45503) 1s

查看版本号，成功

GitLab修改配置后nginx无法启动

Mon, 11 Jun 2018 12:30:00 +0800

起因

在服务器上通过宝塔搭建了GitLab之后一直没修改GitLab的clone域名和IP配置，今天通过命令自定义域名

 vim /etc/gitlab/gitlab.rb

重启配置

 sudo gitlab-ctl reconfigure

显示启动成功，但网页访问无显示

重启整个GitLab服务

 sudo gitlab-ctl restart

Nginx报错无法启动

查找原因

既然报错，那就只能爬日志看看问题在哪里

 gitlab-ctl tail

报文件存在

 chpst: fatal: unable to run: /opt/gitlab/embedded/sbin/nginx: file does not exist

网上查了查，似乎是宝塔的问题，原因应该是为了防止nginx冲突, 把gitlab的nginx启动文件名改成了gitlab-web, 但有个配置文件没改造成的

解决

修改nginx启动文件

 vi /opt/gitlab/sv/nginx/run

把

 exec chpst -P /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx

改为

 exec chpst -P /opt/gitlab/embedded/sbin/gitlab-web -p /var/opt/gitlab/nginx

重启nginx服务，启动成功，问题解决

GitLab on F嘉阳

k8s环境下GitLab+Helm+GitLab Runner Java项目CICD落地实践

背景

优势

环境

GitLab-Runner部署

配置介绍

使用DinD方式构建已经不再推荐

Use docker-in-docker workflow with Docker executor

Using docker:dind

使用Kaniko构建Docker镜像

使用Helm部署

GitLab-Runner选择可能导致的坑

安装

工程配置

DinD方式构建所需配置

Kubectl和Kubernetes权限配置

Kubernetes关联GitLab配置

GitLab环境配置

CI脚本配置

执行流水线

runner自动扩缩容

使用DinD构建Dockerfile结果

使用Kaniko构建Dockerfile的结果

部署结果

构建结果

GitLab Api使用

使用场景

Api指南

获取工程列表

分支操作

创建分支

保护分支

取消保护分支

删除分支

接口测试

获取工程列表

脚本编写

批量创建分支

批量保护分支

批量取消保护分支

批量分支封板

批量删除分支

DevOps实践心得一

初遇DevOps

Jenkins

Docker+GitLab

DevOps 带来的好处

当前的问题

未来探索方向

Docker 19.03 版本对原Gitlab-Runner的影响

版本变化始末

新版本带来的坑

寻找解决方法

解决方法

更新docker版本

安装Gitlab Runner

修改构建脚本

总结

DevOps GitLab CICD 实践3——CI文件编写

官方文档

GitLab CI/CD Pipeline Configuration Reference

目标

参考

Configure GitLab CI/CD to deploy your application

配置

全局变量

CI脚本

自动构建

执行结果

总结

DevOps GitLab CICD 实践2——Runner 部署

官方文档

Run GitLab Runner in a container

General GitLab Runner Docker image usage

Docker image installation and configuration

安装步骤

获取Gitlab Runner秘钥

Runner注册

准备Docker环境

Using `docker:dind`

使用`DinD`构建`Dockerfile`结果

使用`Kaniko`构建`Dockerfile`的结果