集群 on F嘉阳

PXC系列学习笔记（二）——PXC集群安装配置

Mon, 30 Dec 2019 08:42:07 +0800

1. 介绍

PXC集群具有强一致性特点，用于需要多节点一致性的业务，例如金融系统、交易、支付系统、库存系统等

本次计划搭建PXC集群使用两个分片，每个分片3个节点组成一个PXC集群

名称	IP	CPU	内存	磁盘
pxc2	192.168.1.32	1	2G	40G
pxc3	192.168.1.33	1	2G	40G
pxc4	192.168.1.35	1	2G	40G
pxc5	192.168.1.36	1	2G	40G
pxc6	192.168.1.37	1	2G	40G
pxc7	192.168.1.38	1	2G	40G

2. 删除MariaDB程序包

$ yum -y remove mari*

3. 开放防火墙端口

$ firewall-cmd --zone=public --add-port=3306/tcp --permanent
$ firewall-cmd --zone=public --add-port=4444/tcp --permanent
$ firewall-cmd --zone=public --add-port=4567/tcp --permanent
$ firewall-cmd --zone=public --add-port=4568/tcp --permanent

4. 关闭SELINUX

vi /etc/selinux/config

把SELINUX属性值设置成disabled

reboot

5. 离线安装PXC（推荐）

进入官网下载5.7版本的Percona，注意，此处下载的是集群版本，下载后分发到所有集群节点执行相同的安装和配置操作

进入RPM文件目录，执行安装命令
```
$ yum localinstall *.rpm
```
参考前一篇内容，修改MySQL配置文件、创建账户等操作

6. 创建PXC集群

停止MySQL服务

修改每个PXC节点的/etc/my.cnf文件（在不同节点上，注意调整文件内容）

server-id=1 #PXC集群中MySQL实例的唯一ID，不能重复，必须是数字
wsrep_provider=/usr/lib64/galera3/libgalera_smm.so
wsrep_cluster_name=pxc-cluster #PXC集群的名称
wsrep_cluster_address=gcomm://pxc2,pxc3,pxc4
wsrep_node_name=pxc2 #当前节点的名称
wsrep_node_address=pxc2 #当前节点的IP
wsrep_sst_method=xtrabackup-v2 #同步方法（mysqldump、rsync、xtrabackup）
wsrep_sst_auth= admin:password #同步使用的帐户
pxc_strict_mode=ENFORCING #同步严厉模式
binlog_format=ROW #基于ROW复制（安全可靠）
default_storage_engine=InnoDB #默认引擎
innodb_autoinc_lock_mode=2 #主键自增长不锁表

主节点的管理命令（第一个启动的PXC节点）

$ systemctl start mysql@bootstrap.service
$ systemctl stop mysql@bootstrap.service
$ systemctl restart mysql@bootstrap.service

非主节点的管理命令（非第一个启动的PXC节点）

$ systemctl start mysql
$ systemctl stop mysql
$ systemctl restart mysql

查看PXC集群状态信息
```
show status like 'wsrep_cluster%' ;
```
按照上述配置方法，创建两组PXC集群

7. PXC节点启动与关闭

如果最后关闭的PXC节点是安全退出的，那么下次启动要最先启动这个节点，而且要以主节点启动
如果最后关闭的PXC节点不是安全退出的，那么要先修改/var/lib/mysql/grastate.dat 文件，把其中的safe_to_bootstrap属性值设置为1，再安装主节点启动

Kubernetes 集群搭建

Sun, 20 Oct 2019 12:31:00 +0800

搭建方法分类

如果是搭建Kubernetes的学习环境，则可以直接使用minikube快速搭建单节点的Kubernetes环境，官方推荐使用kubeadm搭建Kubernetes集群生产环境

但kubeadm需要连接谷歌容器仓库获取镜像，在网络受限的情况下无法搭建成功，故有两种搭建方法：

服务器使用代理，代理所有的http和https连接，可以自由访问任意地址
使用阿里云镜像地址

本次采用第二种方式进行搭建

环境准备

本次宿主机为CentOS 7，结合3台virtual box虚拟机模拟集群环境，虚拟机使用Vagrant进行管理，vagrant安装比较简单，此处不再赘述

宿主机配置：

CPU：4C
RAM：8G

虚拟机配置

由于网络受限，vagrant默认拉取的操作系统文件地址不可访问，需要自行从官方下载镜像或者使用代理下载镜像，此处对此操作也不再赘述

三台虚拟机配置文件

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.require_version ">= 1.6.0"

boxes = [
 {
 :name => "k8s-master",
 :eth1 => "192.168.205.120",
 :mem => "2048",
 :cpu => "2"
 },
 {
 :name => "k8s-node1",
 :eth1 => "192.168.205.121",
 :mem => "2048",
 :cpu => "1"
 },
 {
 :name => "k8s-node2",
 :eth1 => "192.168.205.122",
 :mem => "2048",
 :cpu => "1"
 }

]

Vagrant.configure(2) do |config|

 config.vm.box = "centos/7"
 boxes.each do |opts|
 config.vm.define opts[:name] do |config|
 config.vm.hostname = opts[:name]
 config.vm.provider "vmware_fusion" do |v|
 v.vmx["memsize"] = opts[:mem]
 v.vmx["numvcpus"] = opts[:cpu]
 end
 config.vm.provider "virtualbox" do |v|
 v.customize ["modifyvm", :id, "--memory", opts[:mem]]
 v.customize ["modifyvm", :id, "--cpus", opts[:cpu]]
 end
 config.vm.network :private_network, ip: opts[:eth1]
 end
 end
 config.vm.provision "shell", privileged: true, path: "./setup.sh"
end

所有虚拟机启动后都会执行./setup.sh脚本配置基础环境，脚本内容如下

#/bin/sh

sudo yum install -y vim telnet bind-utils wget yum-utils

sudo yum-config-manager \
 --add-repo \
 https://download.docker.com/linux/centos/docker-ce.repo
sudo yum -y install docker-ce-18.09.8 docker-ce-cli-18.09.8 containerd.io

if [ ! $(getent group docker) ];
then 
 sudo groupadd docker;
else
 echo "docker user group already exists"
fi

sudo gpasswd -a $USER docker
sudo systemctl restart docker

# open password auth for backup if ssh key doesn't work, bydefault, username=vagrant password=vagrant
sudo sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/g' /etc/ssh/sshd_config
sudo systemctl restart sshd

sudomkdir /etc/yum.repos.d/bak && mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/bak
sudowget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.cloud.tencent.com/repo/centos7_base.repo
sudowget -O /etc/yum.repos.d/epel.repo http://mirrors.cloud.tencent.com/repo/epel-7.repo

yum clean all && yum makecache

sudo bash -c 'cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF'

sudo wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
 "registry-mirrors": ["https://clayphwh.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

sudo setenforce 0

sudo yum install -y kubelet kubeadm kubectl

sudo bash -c 'cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
EOF'
sudo sysctl --system

sudo systemctl stop firewalld
sudo systemctl disable firewalld
sudo swapoff -a

sudo systemctl enable docker.service
sudo systemctl enable kubelet.service

脚本内已将K8S镜像均配置为阿里云镜像仓库，并配置了阿里云容器加速服务地址，更便于在国内拉取镜像

启动虚拟机

使用命令启动虚拟机

$ vagrant up

启动需要较长时间，若出现大量超时也请耐心等待

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.163.com
 * updates: mirrors.163.com
http://vault.centos.org/7.1.1503/os/x86_64/repodata/0e6e90965f55146ba5025ea450f822d1bb0267d0299ef64dd4365825e6bad995-c7-x86_64-comps.xml.gz: [Errno 12] Timeout on http://vault.centos.org/7.1.1503/os/x86_64/repodata/0e6e90965f55146ba5025ea450f822d1bb0267d0299ef64dd4365825e6bad995-c7-x86_64-comps.xml.gz: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
http://vault.centos.org/7.1.1503/updates/x86_64/repodata/93b71f445d2ec2138d28152612f4fb29c8e76ee31f2666b964d88249b4e0a955-primary.sqlite.bz2: [Errno 12] Timeout on http://vault.centos.org/7.1.1503/updates/x86_64/repodata/93b71f445d2ec2138d28152612f4fb29c8e76ee31f2666b964d88249b4e0a955-primary.sqlite.bz2: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
http://vault.centos.org/7.2.1511/os/x86_64/repodata/c6411f1cc8a000ed2b651b49134631d279abba1ec1f78e5dcca79a52d8c1eada-primary.sqlite.bz2: [Errno 12] Timeout on http://vault.centos.org/7.2.1511/os/x86_64/repodata/c6411f1cc8a000ed2b651b49134631d279abba1ec1f78e5dcca79a52d8c1eada-primary.sqlite.bz2: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
...

启动完成后通过命令查看虚拟机状态

$ vagrant status
Current machine states:

k8s-master running (virtualbox)
k8s-node1 running (virtualbox)
k8s-node2 running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

K8S master节点初始化

进入主节点，查看环境是否已经配置完成

$ vagrant ssh k8s-master

执行以下三条语句查看是否有预期输出

[vagrant@k8s-master ~]$ sudo which kubeadm
/bin/kubeadm
[vagrant@k8s-master ~]$ sudo which kubelet
/bin/kubelet
[vagrant@k8s-master ~]$ sudo which kubectl
/bin/kubectl
[vagrant@k8s-master ~]$ sudo docker version
Client:
 Version: 18.09.8
 API version: 1.39
 Go version: go1.10.8
 Git commit: 0dd43dd87f
 Built: Wed Jul 17 17:40:31 2019
 OS/Arch: linux/amd64
 Experimental: false

Server: Docker Engine - Community
 Engine:
 Version: 18.09.8
 API version: 1.39 (minimum version 1.12)
 Go version: go1.10.8
 Git commit: 0dd43dd
 Built: Wed Jul 17 17:10:42 2019
 OS/Arch: linux/amd64
 Experimental: false

使用初始化命令，并指定拉取镜像地址为阿里云地址，同时指定容器网络地址和注册中心广播地址

$ sudo kubeadm init --pod-network-cidr 172.100.0.0/16 --image-repository registry.aliyuncs.com/google_containers --apiserver-advertise-address 192.168.205.120

输出如下即初始化完成

$ sudo kubeadm init --pod-network-cidr 172.100.0.0/16 --image-repository registry.aliyuncs.com/google_containers --apiserver-advertise-address 192.168.205.120
W1020 01:31:27.560129 20473 version.go:101] could not fetch a `Kubernetes`version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1020 01:31:27.560713 20473 version.go:102] falling back to the local client version: v1.16.2
[init] Using `Kubernetes`version: v1.16.2
[preflight] Running pre-flight checks
 [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a `Kubernetes`cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master `kubernetes`kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.205.120]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.205.120 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.205.120 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 59.511578 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: rod570.p67pymzbil4m6u8d
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your `Kubernetes`control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
 https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.205.120:6443 --token rod570.p67pymzbil4m6u8d \
 --discovery-token-ca-cert-hash sha256:8f4cb9d555c78d58befeb3cfa3f7537989aa599e53e4f1bae929d8cc7afd1476

kubectl 配置

拷贝kubectl配置

$ rm -rf ~/.kube
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

若不配置直接使用则报无法连接错误

[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces
The connection to the server localhost:8080 was refused - did you specify the right host or port?

此时使用kubectl可查看k8s容器运行情况，输出如下则为初始化成功，启动由于未配置网络插件，故coredns一直处于Pending状态

[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58cc8c89f4-c297b 0/1 Pending 0 81s
kube-system coredns-58cc8c89f4-spzhj 0/1 Pending 0 81s
kube-system etcd-k8s-master 1/1 Running 0 39s
kube-system kube-apiserver-k8s-master 1/1 Running 0 49s
kube-system kube-controller-manager-k8s-master 1/1 Running 0 40s
kube-system kube-proxy-xlb62 1/1 Running 0 81s
kube-system kube-scheduler-k8s-master 1/1 Running 0 51s

安装网络插件

网络插件有多个可选，可通过官网查看，此处选择安装flannel CNI网络插件

For flannel to work correctly, you must pass --pod-network-cidr=10.244.0.0/16 to kubeadm init.

Set /proc/sys/net/bridge/bridge-nf-call-iptables to 1 by running sysctl net.bridge.bridge-nf-call-iptables=1 to pass bridged IPv4 traffic to iptables’ chains. This is a requirement for some CNI plugins to work, for more information please see here.

Make sure that your firewall rules allow UDP ports 8285 and 8472 traffic for all hosts participating in the overlay network. see here .

Note that flannel works on amd64, arm, arm64, ppc64le and s390x under Linux. Windows (amd64) is claimed as supported in v0.11.0 but the usage is undocumented.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml
For more information about flannel, see the CoreOS flannel repository on GitHub .

相关的网络配置已在脚本中写入，此时直接执行yml脚本即可

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

再次查看容器运行情况，所有容器均已运行，则master节点配置完成

[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58cc8c89f4-c297b 1/1 Running 0 5m17s
kube-system coredns-58cc8c89f4-spzhj 1/1 Running 0 5m17s
kube-system etcd-k8s-master 1/1 Running 0 4m35s
kube-system kube-apiserver-k8s-master 1/1 Running 0 4m45s
kube-system kube-controller-manager-k8s-master 1/1 Running 0 4m36s
kube-system kube-flannel-ds-amd64-6rchg 1/1 Running 0 3m50s
kube-system kube-proxy-xlb62 1/1 Running 0 5m17s
kube-system kube-scheduler-k8s-master 1/1 Running 0 4m47s

配置worker节点

以k8s-node1为例，执行master初始化时提供的脚本即可加入k8s集群，输出如下则加入成功

[vagrant@k8s-node1 ~]$ sudo kubeadm join 192.168.205.120:6443 --token 8ry5oo.y48ksgurn103zq4h \
> --discovery-token-ca-cert-hash sha256:ead07352591500c2cfe3321bf87d2e068790e16f2a7e0cc23541c864d24006d4
[preflight] Running pre-flight checks
 [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

稍等片刻进入master节点查看容器运行情况，会发现多了几个网络容器，属于预期情况

[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58cc8c89f4-c297b 1/1 Running 0 6m28s
kube-system coredns-58cc8c89f4-spzhj 1/1 Running 0 6m28s
kube-system etcd-k8s-master 1/1 Running 0 5m46s
kube-system kube-apiserver-k8s-master 1/1 Running 0 5m56s
kube-system kube-controller-manager-k8s-master 1/1 Running 0 5m47s
kube-system kube-flannel-ds-amd64-6fwg5 1/1 Running 0 49s
kube-system kube-flannel-ds-amd64-6rchg 1/1 Running 0 5m1s
kube-system kube-flannel-ds-amd64-z4plh 1/1 Running 0 43s
kube-system kube-proxy-cdcs4 1/1 Running 0 49s
kube-system kube-proxy-nlmbb 1/1 Running 0 43s
kube-system kube-proxy-xlb62 1/1 Running 0 6m28s
kube-system kube-scheduler-k8s-master 1/1 Running 0 5m58s

集群验证

在master节点中启动nginx容器测试集群情况

$ kubectl create deployment nginx --image=nginx

稍等片刻，输出如下则集群正常运行，k8s集群配置完成

[vagrant@k8s-master ~]$ kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-86c57db685-2gkxm 1/1 Running 0 58s
kube-system coredns-58cc8c89f4-c297b 1/1 Running 0 7m46s
kube-system coredns-58cc8c89f4-spzhj 1/1 Running 0 7m46s
kube-system etcd-k8s-master 1/1 Running 0 7m4s
kube-system kube-apiserver-k8s-master 1/1 Running 0 7m14s
kube-system kube-controller-manager-k8s-master 1/1 Running 0 7m5s
kube-system kube-flannel-ds-amd64-6fwg5 1/1 Running 0 2m7s
kube-system kube-flannel-ds-amd64-6rchg 1/1 Running 0 6m19s
kube-system kube-flannel-ds-amd64-z4plh 1/1 Running 0 2m1s
kube-system kube-proxy-cdcs4 1/1 Running 0 2m7s
kube-system kube-proxy-nlmbb 1/1 Running 0 2m1s
kube-system kube-proxy-xlb62 1/1 Running 0 7m46s
kube-system kube-scheduler-k8s-master 1/1 Running 0 7m16s

Mesos集群安装

Wed, 03 Oct 2018 22:10:00 +0800

基础环境

4台服务器，一台外部客户机

安装指南

官方支持C++源码编译安装和Docker安装

C++源码编译安装地址

Docker镜像仓库

停止所有容器命令

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

安装

mesos主节点

node1

docker pull mesosphere/mesos-master:1.7.0

node2(host)

docker run --name mesos-zookeeper -p 2181:2181 --restart always -d zookeeper:3.5

node1执行脚本(需替换zk IP地址)

MESOS_QUORUM=1即1为单机运行，3个节点为2，计算方式为 n÷2+1 \

若不手动指定hostname映射的IP，则所有节点默认通过主机名进行通信

使用host通信脚本

docker run -d --net=host \
 -e MESOS_PORT=5050 \
 -e MESOS_ZK=zk://192.168.0.32:2181/mesos \
 -e MESOS_QUORUM=1 \
 -e MESOS_REGISTRY=in_memory \
 -e MESOS_LOG_DIR=/var/log/mesos \
 -e MESOS_WORK_DIR=/var/tmp/mesos \
 -v "$(pwd)/log/mesos:/var/log/mesos" \
 -v "$(pwd)/work/mesos:/var/tmp/mesos" \
 -p 5050:5050 \
 --name=mesos-master \
 mesosphere/mesos-master:1.7.0 \

使用IP通信脚本

docker run -d --net=host \
 --hostname=192.168.0.32 \
 -e MESOS_PORT=5050 \
 -e MESOS_ZK=zk://192.168.0.32:2181/mesos \
 -e MESOS_QUORUM=1 \
 -e MESOS_REGISTRY=in_memory \
 -e MESOS_LOG_DIR=/var/log/mesos \
 -e MESOS_WORK_DIR=/var/tmp/mesos \
 -v "$(pwd)/log/mesos:/var/log/mesos" \
 -v "$(pwd)/work/mesos:/var/tmp/mesos" \
 -p 5050:5050 \
 --name=mesos-master \
 mesosphere/mesos-master:1.7.0 \
 --no-hostname_lookup \
 --ip=192.168.0.33

查看日志

docker logs mesos-master

成功输出日志

I1003 08:41:29.868827 14 group.cpp:341] Group process (zookeeper-group(2)@192.168.0.33:5050) connected to ZooKeeper
I1003 08:41:29.868902 14 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1003 08:41:29.868921 14 group.cpp:419] Trying to create path '/mesos' in ZooKeeper
I1003 08:41:29.902194 14 contender.cpp:268] New candidate (id='1') has entered the contest for leadership
I1003 08:41:29.904409 8 detector.cpp:152] Detected a new leader: (id='1')
I1003 08:41:29.904513 8 group.cpp:700] Trying to get '/mesos/json.info_0000000001' in ZooKeeper
I1003 08:41:29.912968 8 zookeeper.cpp:262] A new leading master (UPID=master@192.168.0.33:5050) is detected
I1003 08:41:29.913091 8 master.cpp:2113] Elected as the leading master!
I1003 08:41:29.913115 8 master.cpp:1668] Recovering from registrar
I1003 08:41:29.916913 11 registrar.cpp:383] Successfully fetched the registry (0B) in 3.734016ms
I1003 08:41:29.917054 11 registrar.cpp:487] Applied 1 operations in 26us; attempting to update the registry
I1003 08:41:29.918582 11 registrar.cpp:544] Successfully updated the registry in 1.449984ms
I1003 08:41:29.918630 11 registrar.cpp:416] Successfully recovered registrar
I1003 08:41:29.918772 8 master.cpp:1782] Recovered 0 agents from the registry (121B); allowing 10mins for agents to reregister

访问IP:5050，对于PWD，可以访问http://SSHIP-5050.direct.labs.play-with-docker.com/#/，将SSH ip替换成实际提供的IP即可

mesos从节点

node3 (slave-1)、node4(slave-2)

启动脚本

使用host通信脚本

mkdir /cgroup
docker run -d --net=host --privileged \
 -e MESOS_PORT=5051 \
 -e MESOS_MASTER=zk://192.168.0.32:2181/mesos \
 -e MESOS_SWITCH_USER=0 \
 -e MESOS_CONTAINERIZERS=docker,mesos \
 -e MESOS_LOG_DIR=/var/log/mesos \
 -e MESOS_WORK_DIR=/var/tmp/mesos \
 -v "$(pwd)/log/mesos:/var/log/mesos" \
 -v "$(pwd)/tmp/mesos:/var/tmp/mesos" \
 -v /var/run/docker.sock:/var/run/docker.sock \
 -v /cgroup:/cgroup \
 -v /sys:/sys \
 -v "$(which docker):/usr/local/bin/docker" \
 --name=mesos-slave \
 -p 5051:5051 \
 mesosphere/mesos-slave:1.7.0 \
 --no-systemd_enable_support

使用IP通信脚本，--ip需要改成实际IP

mkdir /cgroup
docker run -d --net=host --privileged \
 -e MESOS_PORT=5051 \
 -e MESOS_MASTER=zk://192.168.0.32:2181/mesos \
 -e MESOS_SWITCH_USER=0 \
 -e MESOS_CONTAINERIZERS=docker,mesos \
 -e MESOS_LOG_DIR=/var/log/mesos \
 -e MESOS_WORK_DIR=/var/tmp/mesos \
 -v "$(pwd)/log/mesos:/var/log/mesos" \
 -v "$(pwd)/tmp/mesos:/var/tmp/mesos" \
 -v /var/run/docker.sock:/var/run/docker.sock \
 -v /cgroup:/cgroup \
 -v /sys:/sys \
 -v "$(which docker):/usr/local/bin/docker" \
 --name=mesos-slave \
 -p 5051:5051 \
 mesosphere/mesos-slave:1.7.0 \
 --no-systemd_enable_support \
 --no-hostname_lookup \
 --ip=192.168.0.31

查看日志

docker logs mesos-slave

启动成功日志

I1003 09:05:08.615039 12 group.cpp:341] Group process (zookeeper-group(1)@192.168.0.31:5051) connected to ZooKeeper
I1003 09:05:08.615104 12 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1003 09:05:08.615121 12 group.cpp:419] Trying to create path '/mesos' in ZooKeeper
I1003 09:05:08.618291 13 state.cpp:66] Recovering state from '/var/tmp/mesos/meta'
I1003 09:05:08.618405 15 slave.cpp:6930] Finished recovering checkpointed state from '/var/tmp/mesos/meta', beginning agent recovery
I1003 09:05:08.618475 8 task_status_update_manager.cpp:207] Recovering task status update manager
I1003 09:05:08.618628 8 docker.cpp:908] Recovering Docker containers
I1003 09:05:08.626359 13 detector.cpp:152] Detected a new leader: (id='3')
I1003 09:05:08.626487 15 group.cpp:700] Trying to get '/mesos/json.info_0000000003' in ZooKeeper
I1003 09:05:08.628664 15 zookeeper.cpp:262] A new leading master (UPID=master@192.168.0.33:5050) is detected
I1003 09:05:08.997036 15 docker.cpp:921] Got the list of Docker containers
I1003 09:05:08.997324 15 docker.cpp:1157] Finished processing orphaned Docker containers
I1003 09:05:08.997436 8 slave.cpp:7159] Recovering executors
I1003 09:05:08.997553 8 slave.cpp:7312] Finished recovery
I1003 09:05:08.998044 11 task_status_update_manager.cpp:181] Pausing sending task status updates
I1003 09:05:08.998045 8 slave.cpp:1275] New master detected at master@192.168.0.33:5050
I1003 09:05:08.998159 8 slave.cpp:1329] No credentials provided. Attempting to register without authentication
I1003 09:05:08.998181 8 slave.cpp:1340] Detecting new master
I1003 09:05:09.446756 11 slave.cpp:1500] Registered with master master@192.168.0.33:5050; given agent ID b597695d-6e37-418a-9879-4ffe46117e17-S0
I1003 09:05:09.449091 10 task_status_update_manager.cpp:188] Resuming sending task status updates

访问主节点web界面可看见资源邀约有效

节点情况

marathon安装

node5

启动脚本

docker run -d --net=host \
 -p 8080:8080 \
 mesosphere/marathon:latest \
 --master zk://192.168.0.32:2181/mesos \
 --zk zk://192.168.0.32:2181/marathon

通过IP:8080查看运行结果

marathon-lb安装

node6

启动脚本

docker run -d -e PORTS=9090 --net=host \
 -p 9090:9090 \
 mesosphere/marathon-lb sse \
 --group external \
 --marathon http://192.168.0.29:8080

访问IP:9090/haproxy?stats可查看启动状态