搭建Kubernetes集群

k8s集群安装准备工作

修改服务器hostname

1
2
hostnamectl set-hostname k8s-master-0
# 退出重新登陆即可显示新设置的主机名master01，各服务器修改为对应的主机名

修改hosts

1
2
3
4
5
cat >> /etc/hosts << EOF
10.23.73.44 kz-k8s-master-1
10.23.73.34 kz-k8s-master-2
10.23.73.13 kz-k8s-master-3
EOF

生产环境开启防火墙

1
2
systemctl start firewalld 
systemctl enable firewalld

升级 linux 内核4.14.105

采用源码编译安装

  1. 去官网下载源码包，找那种长期支持的 比如：https://mirrors.edge.kernel.org/pub/linux/kernel/v4.x/linux-4.14.105.tar.gz

  2. 解压源码包

    1
    tar -zxvf linux-4.14.105.tar.gz

  3. 安装工具包

    1
    2
    3
    yum install -y gcc kernel-devel
    yum install -y epel-release
    yum install -y bc wget xz ncurses-devel bison flex elfutils-libelf-devel openssl-devel

  4. 准备编译

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    # 复制一份之前的标准.config 到源码根目录
    cp -v /boot/config-$(uname -r) .config

    # 也可以使用 make menuconfig  进行手动干预
    make menuconfig

    # 开始编译
    make -j $(nproc)  # 可以使用多个线程编译，根据CPU数量来
    make modules_install
    make install

    # 查看启动器是否是新内核的选项
    cat /boot/grub2/grub.cfg

    # 如果没有进行指定
    grub2-mkconfig -o /boot/grub2/grub.cfg
     
    # 设置默认版本
    grubby --set-default /boot/vmlinuz-4.14.105

    # 重启服务器
    reboot

验证mac地址uuid

要保证mac地址uuid每个节点都是唯一的（目前我就一个节点），Kubernetes会使用这些值来唯一的标识集群中的节点。如果这些值不是每个节点唯一的，在安装的过程中可能会失败。

1
2
3
4
5
# 查看uuid
cat /sys/class/dmi/id/product_uuid

# 查看mac地址
ifconfig -a | grep ether | awk '{print $2}'

检查网络

检查网络适配器

  1. As a requirement for your Linux Node’s iptables to correctly see bridged traffic, you should ensure net.bridge.bridge-nf-call-iptables is set to 1

    1
    2
    3
    4
    5
    cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    EOF
    sudo sysctl --system

    执行上述命令，需要检查系统是否加载了 br_netfilter 模块 官方文档

    1
    2
    3
    4
    # 查看是否开启
    lsmod | grep br_netfilter
    # 开启模块
    sudo modprobe br_netfilter

Docker安装

Control plane 和 work节点都要执行

安装依赖

1
yum install -y yum-utils device-mapper-persistent-data lvm2

设置Docker源

1
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

安装Docker

  1. 查看docker版本

    1
    yum list docker-ce --showduplicates | sort -r

  2. 安装

    1
    yum install docker-ce-18.09.9 docker-ce-cli-18.09.9 containerd.io -y

启动Docker

1
2
systemctl start docker
systemctl enable docker

Docker加速器

增加Docker镜像加速器，使用阿里云的 阿里云镜像加速器

1
2
3
4
5
6
7
8
9
vim /etc/docker/daemon.json
# 加入以下内容
{
    "registry-mirrors": ["https://9zce14xb.mirror.aliyuncs.com"]
} 

# 重启Docker Daemon
systemctl daemon-reload
systemctl restart docker

修改Cgroup Driver

  1. 修改daemon.json，修改cgroupdriver是为了消除告警：
    [WARNING IsDockerSystemdCheck]: detected “cgroupfs” as the Docker cgroup driver. The recommended driver is “systemd”. Please follow the guide at https://kubernetes.io/docs/setup/cri/

    1
    2
    3
    4
    5
    6
    vim /etc/docker/daemon.json 
    # 更改为以下内容
    {
        "registry-mirrors": ["https://9zce14xb.mirror.aliyuncs.com"],
        "exec-opts": ["native.cgroupdriver=systemd"]
    }

  2. 重新加载docker

    1
    2
    systemctl daemon-reload
    systemctl restart docker

安装k8s

新增kubernetes的镜像源，更改为国内的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# Set SELinux in permissive mode (effectively disabling it)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

# 更新缓存
yum clean all
yum -y makecache

Note:

  • Setting SELinux in permissive mode by running setenforce 0 and sed ... effectively disables it. This is required to allow containers to access the host filesystem, which is needed by pod networks for example. You have to do this until SELinux support is improved in the kubelet.
  • You can leave SELinux enabled if you know how to configure it but it may require settings that are not supported by kubeadm.

选择版本

目前跟阿里云保持一致，v1.16.6 k8s更新日志 ，可以查看支持的Docker版本（在页面内搜索docker versions）

1
yum list kubelet --showduplicates | sort -r

安装kubelet、kubeadm、kubelctl（选择安装）

1
2
yum install -y kubelet-1.16.6 kubeadm-1.16.6 kubectl-1.16.6
# 不用启动kubectl  后面会自动启动

三个安装包的说明

  • kubelet运行在集群的所有节点上，用于启动Pod和容器等对象的工具
  • kubeadm用于初始化集群，启动集群的命令工具
  • kubectl 用于和集群通信的命令行，通过kubectl可以部署和管理应用，查看各种资源，创建、更新和删除各种组件

开机启动KUBELET

1
systemctl enable kubelet

下载镜像

  1. kubernetes所有的安装组件和Docker镜像都放在啊Google自己的镜像源中，下载很慢。先通过国内的镜像源下载所有的镜像，然后再切换回默认的tag

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    # File images.sh
    #!/bin/bash
    url=registry.aliyuncs.com/google_containers
    version=v1.16.6
    images=(`kubeadm config images list --kubernetes-version=$version|awk -F '/' '{print $2}'`)
    for imagename in ${images[@]} ; do
      docker pull $url/$imagename
      docker tag $url/$imagename k8s.gcr.io/$imagename
      docker rmi -f $url/$imagename
    done

开始搭建高可用集群

官方提供了两种方法：

  1. With stacked control plane nodes. This approach requires less infrastructure. The etcd members and control plane nodes are co-located.

    需要更少的基础架构，etcd与control plane在同一层面

  2. With an external etcd cluster. This approach requires more infrastructure. The control plane nodes and etcd members are separated.

    etcd集群是独立的

这里选择第一种架构，拓扑图如下

拓扑图

Keepalived

  1. yum install -y keepalived

    1
    2
    3

    2. 修改keepalived的配置文件

    
vim /etc/keepalived/keepalived.conf

    1
    2
    3

    配置文件内容如下：

    
! Configuration File for keepalived

global_defs {
    router_id k8s-master-0
}

vrrp_script haproxy-check {
    # 检查进程是否存在，存在返回0
    script 'killall -0 haproxy'
    #每隔一秒执行一次上面的检测
    interval 1
    #权重减5，而主的上面我配置的是权重减10
    weight -5
    rise 1
    fall 2
}

vrrp_instance haproxy-vip {
    state MASTER | BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    
    # 单播的源地址，写本机的IP
    unicast_src_ip 10.23.73.46

    # 指定vip地址
    virtual_ipaddress {
        172.22.1.1
    }

    track_script {
        haproxy-check weight 20
    }
}
virtual_server 10.23.73.1 16443 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    protocol TCP
    real_server 10.23.73.46 6443 {
        weight 10
        TCP_CHECK {
            connect_timeout 10
            retry 3
            delay_before_retry 3
            connect_port 6443
        }
    }
    real_server 10.23.73.34 6443 {
        weight 10
        TCP_CHECK {
            connect_timeout 10
            retry 3
            delay_before_retry 3
            connect_port 6443
        }
    }
    real_server 10.23.73.13 6443 {
        weight 10
        TCP_CHECK {
            connect_timeout 10
            retry 3
            delay_before_retry 3
            connect_port 6443
        }
    }
}

    1
    2
    3
    4
    5

       在每个节点都配置keepalived，注意IP不要写错了

    3. keepalived 使用VRRP协议(ip protocol 112)以及多播地址 `224.0.0.18` ，需要在防火墙上开启:

    
# 防火墙操作
 systemctl status | start | stop  firewalld
 # 添加规则
firewall-cmd --add-rich-rule='rule protocol value="vrrp" accept' --permanent
firewall-cmd --reload
# 查看添加的规则
firewall-cmd --list-rich-rules

    1
    2
    3

    4. 启动keepalived

    
sudo systemctl start keepalived
sudo systemctl enable keepalived

    1
    2
    3

    5. 启动后观察服务器ip，可以看到其中一台服务器的 eth0 上会绑定浮动IP 172.22.1.10

    
ip a

    1
    2
    3
    4
    5
    6
    7
    8
    9

       ![keepalived](https://tva1.sinaimg.cn/large/007S8ZIlly1gez1znn5gqj30nj049q3h.jpg)

    ### HAProxy

    HAProxy将检查后端 **kubemaster-X**服务器上的**kube-apiserver**端口健康状态，并且负载均衡到集群的健康实例上，同时也将对局域网提供Kubernetes web UI(Dashboard)服务，对外提供服务的虚拟VIP即Keepalived的浮动IP地址 `172.22.1.10` 。

    1. 安装

    
yum install -y haproxy

    1
    2
    3

    2. 修改haproxy配置文件

    
vim /etc/haproxy/haproxy.cfg

    1
    2
    3

    内容

    
global

    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main *:5000
    acl url_static       path_beg       -i /static /images /javascript /stylesheets
    acl url_static       path_end       -i .jpg .gif .png .css .js

    use_backend static          if url_static
    default_backend             app

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend static
    balance     roundrobin
    server      static 127.0.0.1:4331 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend app
    balance     roundrobin
    server  app1 127.0.0.1:5001 check
    server  app2 127.0.0.1:5002 check
    server  app3 127.0.0.1:5003 check
    server  app4 127.0.0.1:5004 check

frontend k8s-api
    bind *:16443
    mode tcp
    option tcplog
    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 }
    default_backend k8s-api
backend k8s-api
    mode tcp
    option tcplog
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server k8s-api-0 10.23.73.46:6443 check
    server k8s-api-1 10.23.73.34:6443 check
    server k8s-api-2 10.23.73.13:6443 check

    1
    2
    3

    3. 防火墙

    
sudo firewall-cmd --zone=public --add-port=6443/tcp --permanent
sudo firewall-cmd --reload

    1
    2
    3

    4. 启动HAProxy

    
systemctl start haproxy
systemctl enable haproxy

    1
    2
    3
    4
    5

    ### 初始化Master

    1. 创建配置文件

    
# 文件 kubeadm.conf
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.6
apiServer:
  certSANs:    #填写所有kube-apiserver节点的hostname、IP、VIP
  - k8s-master-0
  - k8s-master-1
  - k8s-master-2
  - 10.23.73.46
  - 10.23.73.34
  - 10.23.73.13
controlPlaneEndpoint: "10.23.73.1:16443"
networking:
  podSubnet: "10.23.0.0/16"
  
  
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.23.73.44
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master-0
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: '172.22.1.1:16443'
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.16.6
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: '172.22.0.0/16'
scheduler: {}

    1

    
kubeadm init --config kubeadm.yaml

    1
    2
    3

    2. 初始化master节点

    
kubeadm init --config=kubeadm.yaml

    1
    2
    3
    4
    5
    6
    7
    8
    9

    ![initsuccess](https://tva1.sinaimg.cn/large/007S8ZIlgy1gez08h5t13j30ox0bsdhm.jpg)

    记录kubeadm join的输出，之后需要将worker节点或者其他的control plane节点加入集群

    **初始化失败**

    如果初始化失败，需要重新初始化，利用kubeadm reset

    
kubeadm reset
rm -rf $HOME/.kube/config

    1
    2
    3
    4
    5

    #### 安装FLANNEL网络

    在master0上创建flannel网络

    
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

    1
    2
    3
    4
    5
    6
    7

    ## 加入其它master节点

    ### 证书分发

    1. 创建秘钥

    
ssh-keygen -t rsa

    1
    2
    3

    1. master0分发证书给其它节点

    
USER=root # customizable
CONTROL_PLANE_IPS="10.23.73.13 10.23.73.34"
for host in ${CONTROL_PLANE_IPS}; do
    scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/ca.key "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/sa.key "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host:/etc/kubernetes/pki/
    scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:/etc/kubernetes/pki/etcd/ca.crt
    # Quote this line if you are using external etcd
    scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:/etc/kubernetes/pki/etcd/ca.key
done

    1
    2
    3

    2. Master1 加入集群

    
kubeadm join 172.22.1.10:16443 --token qvvte9.uvatz35uby9jwwcn \
  --discovery-token-ca-cert-hash sha256:fc193b660cd104472633d56c3399bc61385776eb5239860c790507b4a630477d \
  --control-plane