kubernetes init流程概览

olei 2,297 views 2

kubernetes init流程

  • 引导前检查
  • 生成私钥以及数字证书
  • 生成控制平面的kubeconfig文件
  • 生成控制平面组件的manifest文件
  • 下载镜像，等待控制平面启动
  • 保存MasterConfiguration
  • 设定Master标志
  • 进行基于TLS的安全引导相关的配置
  • 安装DNS和kube-porxy插件

引导前检查

kubeadm init pre-flight check:

  • kubeadm版本要与安装的kubernetes版本的比对检查
  • kubernetes安装的系统需求检查
  • 其他检查：用户<要求是root>，主机，端口，swap，工具等

生成私钥以及数字证书

kubeadm init生成私钥与证书：

目录在/etc/kubernetes/pki下

-rw-r--r-- 1 root root 1224 Oct 12 11:18 apiserver.crt
-rw-r--r-- 1 root root 1094 Oct 12 11:18 apiserver-etcd-client.crt
-rw------- 1 root root 1679 Oct 12 11:18 apiserver-etcd-client.key
-rw------- 1 root root 1675 Oct 12 11:18 apiserver.key
-rw-r--r-- 1 root root 1099 Oct 12 11:18 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 Oct 12 11:18 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1025 Oct 12 11:18 ca.crt
-rw------- 1 root root 1679 Oct 12 11:18 ca.key
drwxr-xr-x 2 root root 4096 Oct 12 11:18 etcd
-rw-r--r-- 1 root root 1025 Oct 12 11:18 front-proxy-ca.crt
-rw------- 1 root root 1679 Oct 12 11:18 front-proxy-ca.key
-rw-r--r-- 1 root root 1050 Oct 12 11:18 front-proxy-client.crt
-rw------- 1 root root 1679 Oct 12 11:18 front-proxy-client.key
-rw------- 1 root root 1679 Oct 12 11:18 sa.key
-rw------- 1 root root  451 Oct 12 11:18 sa.pub
  • 自建CA，生成ca.key与ca.crt

查看公钥证书

$ openssl x509 -in ca.crt -noout -text
--------------------------------------------
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Oct 12 03:18:52 2018 GMT
            Not After : Oct  9 03:18:52 2028 GMT
        Subject: CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c9:b3:b3:4e:db:bb:a7:be:d1:7e:97:8c:f8:06:
                    a5:df:04:93:72:32:96:b5:f9:f1:16:20:87:61:10:
                    44:b3:33:86:16:24:97:bf:4d:ac:04:5f:f9:c9:ec:
                    61:9d:6c:ce:ad:97:2d:a2:6b:03:e7:b8:89:04:72:
                    9a:91:2a:da:31:52:8e:f9:86:f3:e4:96:27:56:dd:
                    e3:8a:84:4e:11:9e:de:0b:c2:2c:73:cd:fd:1f:66:
                    bf:89:12:38:d5:22:b2:f2:0d:4e:97:e1:59:8c:3d:
                    8c:df:53:74:d4:2c:84:9f:11:55:84:b7:16:6f:44:
                    b9:f1:fd:82:fa:67:1c:08:d2:3c:da:01:0f:d7:a4:
                    4b:85:01:3e:d6:79:dc:96:21:e9:67:b7:0f:f4:bc:
                    ad:1a:84:71:20:e9:e6:81:f6:a1:8b:26:6b:63:85:
                    8a:23:f3:f9:6e:bd:ca:28:9f:5e:fe:dd:01:78:53:
                    0b:fd:01:e9:3a:13:54:5a:32:50:c4:6c:1e:09:4a:
                    96:33:20:a7:71:03:7a:e9:6b:d8:06:a7:16:86:d8:
                    cb:15:85:0d:d4:3b:c4:27:69:b0:d8:59:25:b3:b0:
                    df:60:d5:91:ed:b3:53:77:7a:7b:51:2a:f5:54:56:
                    db:75:e3:2c:c4:9f:a0:b4:99:b3:da:55:d1:f7:1c:
                    d8:11
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         02:fb:42:37:10:9d:fb:db:e0:bc:9e:d9:5f:fe:b9:c1:a8:e2:
         00:06:85:df:41:25:9b:4f:63:ac:ae:61:9a:5e:0c:62:d6:b8:
         24:94:66:ca:c8:bc:33:3e:fe:a8:7b:50:4f:92:86:32:dc:f1:
         55:23:97:ed:6e:d9:44:ee:f7:50:01:99:0d:ae:be:c6:9e:16:
         76:7c:1d:f5:a0:18:3b:7b:fa:7c:af:90:1b:a3:6c:2a:2b:2e:
         aa:c1:ad:fe:82:08:cd:35:47:91:9d:ff:4a:7b:d1:c3:2c:7e:
         59:1a:25:d9:77:43:f3:a3:0f:88:43:94:9a:d7:21:4b:01:70:
         37:b3:11:9d:6b:58:98:0d:41:f8:d9:64:39:99:bb:45:b7:5d:
         f9:46:cf:b3:38:cd:ca:f0:28:0c:68:2d:95:61:97:1f:af:b9:
         e7:b1:62:a8:e2:13:83:b0:de:08:7b:6f:7a:5a:ef:1f:9f:bb:
         12:ed:e4:5b:8d:49:8e:44:3a:d0:af:36:ef:00:25:1c:bd:47:
         12:98:e1:4c:27:eb:ac:48:90:1c:2b:1f:f1:8c:c7:1a:28:b6:
         b7:e0:a7:9e:66:19:45:b9:e2:89:9c:14:b4:ac:6d:ba:a8:e8:
         f0:b1:c9:52:c2:50:6c:b7:cd:a7:3c:dd:df:d5:0b:16:e2:59:
         68:f0:14:c8
  • Serial Number: 0 (0x0)表示ca证书是第一个证书，所以序列号是0
  • Signature Algorithm表示使用的是sha256的非对称加密算法
  • Subject表示comment name
  • X509v3 Key Usage表示证书的用途，该证书的用途是数字签名、秘钥加密、证书的签发
  • CA:TRUE，表明了这个是CA的公钥证书
  • apiserver的私钥与公钥证书

查看公钥证书

$ openssl x509 -in apiserver.crt -noout -text
--------------------------------------------------
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 3463380858865092747 (0x3010668a968a888b)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Oct 12 03:18:52 2018 GMT
            Not After : Oct 12 03:18:52 2019 GMT
        Subject: CN=kube-apiserver
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c2:fb:f7:42:32:e4:74:fb:cc:5a:b8:5d:6e:6f:
                    74:00:c1:3e:21:e4:4c:76:0e:d2:2d:ea:9f:90:9a:
                    10:6a:57:a6:d1:d6:24:1c:90:6e:22:65:8c:67:cd:
                    6a:67:35:0b:4e:50:17:3e:51:05:61:ad:a5:f0:2c:
                    1a:6a:bd:ac:79:a4:57:da:f6:a4:35:51:8b:5e:04:
                    1c:03:98:d0:b5:88:87:91:54:cd:15:d6:5a:e4:7f:
                    38:fa:ee:01:9d:8f:2a:f0:ac:88:5a:a5:8b:6e:ad:
                    74:d5:43:81:2d:44:01:0b:5a:14:01:03:9e:99:d6:
                    82:d5:55:7f:40:80:16:e3:33:6c:d8:a7:8e:2d:e9:
                    7a:ee:66:d0:3d:52:cb:66:ff:f4:a7:a3:a0:5a:db:
                    b7:38:e3:1b:b3:8e:99:31:d0:bb:7e:92:8d:9d:b2:
                    df:5e:62:3e:eb:b9:16:3b:14:dc:db:d5:cb:41:05:
                    e8:c9:cf:1b:75:ba:ba:5f:99:a7:13:90:36:0b:ac:
                    f1:1c:99:82:c5:4c:b1:3f:ff:04:be:dc:ee:19:c6:
                    db:4e:16:3e:68:b7:44:78:c2:4c:76:f8:8b:58:8b:
                    b2:8f:c4:24:6e:d6:64:d1:2a:84:5a:a7:06:6b:95:
                    e1:94:dd:6f:0c:83:48:32:1c:17:51:50:52:a3:b8:
                    f0:01
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Alternative Name:
                DNS:k8s1, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:xxxxx, IP Address:xxxxx
    Signature Algorithm: sha256WithRSAEncryption
         17:8c:7e:17:6c:ac:6d:d4:48:f6:8d:81:6b:67:fa:a1:a3:8d:
         9f:2c:7b:ed:d6:4c:07:15:96:e5:37:3f:dd:88:30:6a:1d:3a:
         52:40:2d:c3:fd:ae:5e:23:d1:d9:8b:f6:56:21:07:e4:98:a6:
         04:53:37:93:fb:21:75:3f:fa:3e:23:bb:29:81:9b:ef:7b:ea:
         9f:df:5a:5a:22:da:b4:31:4b:b1:f0:bc:ee:63:69:2e:93:f5:
         a2:3a:4e:4f:bb:17:5f:f1:87:08:84:ee:59:84:36:ea:6a:b8:
         d7:29:db:1d:45:3b:c8:34:f2:29:4d:36:d2:6e:a3:43:63:13:
         88:c5:e2:27:78:ed:91:b9:67:81:5d:a6:93:c9:99:25:a5:33:
         b7:c5:5f:a8:03:ce:b0:29:55:4f:de:97:3b:75:31:30:9d:58:
         75:a1:00:02:5b:c7:41:f9:ac:81:7f:4c:e3:a3:5c:22:7b:7c:
         41:25:92:2a:c9:71:c2:90:18:65:48:10:81:8e:c9:34:69:60:
         61:a1:4e:4b:cc:6d:36:af:05:01:96:e2:d2:a8:20:60:22:60:
         bb:56:bc:e0:11:d9:5b:c5:ec:bd:58:9a:34:1f:95:99:61:c1:
         fa:1b:4b:47:1d:68:97:dd:23:3e:42:5c:98:b6:21:8f:96:5d:
         52:8c:d9:84
  • X509v3 Subject Alternative Name中的xxx表示的是我公网IP，这个证书的用于多种域名以及IP地址
  • X509v3 Key Usage证书用途只有，证书签名，秘钥加密，没有签署证书的用途
  • apiserver访问kubelet使用的客户端私钥与证书

该证书是apiserver-kubelet-client.crt,没有什么特殊的地方，就不展示了

  • services account需要的sa.key与sa.pub
  • sa.key用于对account的token进行数字签名
  • sa.pub是key对应的公钥文件
  • Etcd相关的私钥与数字证书
  • 整个集群的控制中心，集群中唯一可以访问它的只有apiserver，其他都是通过apiserver的api来获取的
  • 为建立apiserver与etcd之间的安全通道，会生成apiserver访问etcd公钥与私钥证书：apiserver-etcd-client.crt与apiserver-etcd-client.key
  • 在etcd目录下，还有一堆证书
-rw-r--r-- 1 root root 1025 Oct 12 11:18 ca.crt
-rw------- 1 root root 1679 Oct 12 11:18 ca.key
-rw-r--r-- 1 root root 1099 Oct 12 11:18 healthcheck-client.crt
-rw------- 1 root root 1675 Oct 12 11:18 healthcheck-client.key
-rw-r--r-- 1 root root 1090 Oct 12 11:18 peer.crt
-rw------- 1 root root 1675 Oct 12 11:18 peer.key
-rw-r--r-- 1 root root 1078 Oct 12 11:18 server.crt
-rw------- 1 root root 1679 Oct 12 11:18 server.key

可以看到也有CA证书，那么外面的apiserver-etcd-client.crt是由那个CA证书签发的呢？ 首先跟外面的对比一下

$ openssl verify -CAfile ca.crt ./apiserver-etcd-client.crt
-------------------------------------------------------------
./apiserver-etcd-client.crt: O = system:masters, CN = kube-apiserver-etcd-client
error 7 at 0 depth lookup:certificate signature failure
140364182943384:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:103:
140364182943384:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:705:
140364182943384:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:218:

可以看到是错误的信息，说明不是由外部的CA签发的

  • 验证是不是etcd里面的签发的
$ openssl verify -CAfile etcd/ca.crt ./apiserver-etcd-client.crt
-----------------------------------------------------------------
./apiserver-etcd-client.crt: OK

说明是由etcd目录下的CA签发的

生成控制平面的kubeconfig文件

组件kubeconfig文件：

  • .kube/config、/etc/kubernetes/*.conf与KUBECONFIG环境变量
  • 位于/etc/kubernetes目录下的admin.conf、kubelet.conf、scheduler.conf、controller-manager.conf
  • admin.conf包含整个集群的最高权限配置，常用来使用KUBECONFIG环境变量来设置kubectl的kubeconfig信息
  • kubelet.conf被kubelet所使用，用于访问apiserver
  • scheduler.conf用于master上的kube-scheduler组件所使用,用于访问apiserver
  • controller-manager.conf用于master上的kube-controller-manager组件所使用，用于访问apiserver
  • kubeconfig配置：包含cluster、user、context信息

使用kubectl来查看admin.conf中的内容

$ kubectl config view
-----------------------------------
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: https://xxxxxx:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED
  • context是用来绑定cluster与users的
  • 通过current-context是用来确定当前使用的context
  • kubernetes-admin@kubernetes定义了，使用kubernetes-admin这个用户来访问kubernetes这个集群
  • 允许kubectl快速切换context,管理多集群

如果kubeconfig定义了多个context，那么可以通过kubectl set context命令来设定current-context

生成控制平面组件的manifest文件

组件manifest文件

  • 生成各个控制平面的组件的文件：/etc/kubernetes/manifests
-rw------- 1 root root 1640 Oct 12 11:19 etcd.yaml
-rw------- 1 root root 2599 Oct 12 11:19 kube-apiserver.yaml
-rw------- 1 root root 2053 Oct 12 11:19 kube-controller-manager.yaml
-rw------- 1 root root  978 Oct 12 11:19 kube-scheduler.yaml

都是标准的pod文件，每一个对应一个在master上的控制组件

  • 跟普通组件不同的是，控制平面组件是以Static Pod形式运行的
  • 静态节点Static Pod是以节点上的kubelet来管理的，不通过master节点的apiserver来管理，也不管理任何控制器
  • 由kubelet自己来监控，当Static Pod崩溃的时候，kubelet会自动重启这些Pod
  • 始终绑定在某个kubelet上，并且始终运行在同一节点上
  • 并且kubelet会自动为每一个Static Pod在kubernetse的apiserver上创建一个镜像的pod
  • 所以可以通过apiserver查询到该server，并不能管理控制它
  • kubelet读取manifests目录并管理各控制平台组件的pod启动停止

下载镜像，等待控制平面启动

kubeadm依赖kubelet下载镜像并启动static pod

  • 从k8s.gcr.io上面下载组件镜像

由于国内网络原因，不科学上网，是从那上面下载不下来的，解决方法是下载国内的一些镜像，然后将名称更改为所要用的名称

  • kubeadm会一直探测并等待localhost:6443/healthz服务返回成功

配置文件存放在manifests文件夹下的kube-apiserver.yaml中

$ cat manifests/kube-apiserver.yaml
-------------------------------------------
... ...
livenessProbe:
      failureThreshold: 8
      httpGet:
        host: xxxxxx
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
... ...
  • 这一过程之后，控制平面组件都已经启动ok了，可以通过如下命令来查看
$ kubectl get pods -n kube-system -o wide

安装DNS与kube-proxy插件

安装Addons

  • 是以DaemonSet方式部署kube-proxy

查看以DaemonSet方式安装的kube-proxy

$ kubectl get daemonset -n kube-system
----------------------------------------------
NAME         DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-proxy   3         3         3         3            3           <none>          4d
weave-net    3         3         3         3            3           <none>          4d
  • 部署kube-dns（也可以使用CoreDNS代替）
  • 安装好的DNS插件，会显示pending状态，直到cluster网络就绪

可以通过安装weave-net来安装网络插件，使得DNS处于running状态

$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

若是网络IP冲突，可以将该文件下载下来，查找到weave的容器（搜索：containers），写入

- name: IPALLOC_RANG
  value: xxxxxx/xxxxxxx
## xxx表示自己在init的时候，给定的IP地址
  • 验证
$ kubectl get pods -n kube-system -o wide

原dns那个也running了

本文作者为olei，转载请注明。

发表评论 取消回复
表情 图片 链接 代码

  1. 祈祷
    祈祷 Lv 1

    请问/etc/kubernetes/pki下生成的apiserver-kubelet-client.crt作用是什么？

    2018-11-12 10:33 回复
    • olei
      olei 站长

      @祈祷这个是apiserver访问kubelet使用的客户端的公钥证书

      2018-11-12 10:38 回复
分享