生成CA证书（K8S)

Edited on In

前言

了解证书

k8s 的证书都在 /etc/kubernetes/pki 下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
├── apiserver.crt  apiserver 证书
├── apiserver.key  apiserver 证书
├── apiserver-kubelet-client.crt
├── apiserver-kubelet-client.key
├── apiserver-etcd-client.crt apiserver访问etcd的证书
├── apiserver-etcd-client.key apiserver访问etcd的证书
├── ca.crt 根证书
├── ca.key 根证书
├── etcd
│   ├── etcd-ca.crt
│   ├── healthcheck-client.crt pod中Liveness探针客户端证书
│   ├── healthcheck-client.key pod中Liveness探针客户端证书
│   ├── etcd.crt  节点通信证书
│   └── etcd.key
├── front-proxy-ca.crt 代理根证书
├── front-proxy-ca.key
├── front-proxy-client.crt 代理根证书签发的客户端证书
├── front-proxy-client.key
├── sa.key
└── sa.pub

非对称加密会生成一个密钥对，如上面的sa.key sa.pub就是密钥对，公钥用于加密私钥用于解密。

kubelet 与 apiserver 会互相访问，所有他们都有证书。

kubelet变化频繁，一般只需要指定 ca 根证书。

apiserver变化不频繁，所以在创建集群时，分配好用作 kube-apiserver 的IP或主机名/域名

但是由于部署在node节点上的kubelet会因为集群规模的变化而频繁变化, 而无法预知node的所有IP信息, 所以kubelet上一般不会明确指定服务端证书,
而是只指定ca根证书, 让kubelet根据本地主机信息自动生成服务端证书并保存到配置的cert-dir文件夹中

代理根证书：

1
2
front-proxy-ca.crt
front-proxy-ca.key

由代理根证书签发的客户端证书：

1
2
front-proxy-client.crt
front-proxy-client.key

比如使用kubectl proxy代理访问时，kube-apiserver使用这个证书来验证客户端证书是否是自己签发的证书。

我这里有

  • 192.168.2.158 master-158
  • 192.168.2.159 master-159
  • 192.168.2.160 master-160
  • 192.168.2.161 node-161

CA

配置时间

1
2
yum install  ntpdate -y 
ntpdate time1.aliyun.com

cfssl 生成自签名 TLS 证书的方法

在 host158上执行

生成自签名 root CA 证书

1
2
3
4
5
6
7
8
9
10
11
12
13
#rm -f /opt/cfssl* 
rm -rf /opt/certs 
mkdir -p /opt/certs 
cd  /opt/certs 
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64  -o /usr/local/bin/cfssl
chmod +x /usr/local/bin/cfssl
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64 -o /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssljson
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl-certinfo_1.5.0_linux_amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl-certinfo
# 查看版本
/usr/local/bin/cfssl version
/usr/local/bin/cfssljson -h

生成

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
# 创建根证书签名请求文件
cat > /opt/certs/ca-csr.json <<EOF
{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "maxzhao-ca",
      "OU": "etcd Security",
      "L": "NanJing",
      "ST": "Jiang Su",
      "C": "CN"
    }
  ],
  "CN": "maxzhao"
}
EOF
# CN：Common Name：kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)，
# O：Organization：kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；
# kube-apiserver 将提取的 User、Group 作为 RBAC 授权的用户标识； 

# 证书配置文件
cat > /opt/certs/ca-config.json <<EOF
{
  "signing": {
    "default": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "175200h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "175200h"
      },
      "etcd": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "175200h"
      }
    }
  }
}
EOF
# signing：表示该证书可用于签名其它证书（生成的 ca.pem 证书中 CA=TRUE）；
# server auth：表示 client 可以用该该证书对 server 提供的证书进行验证；
# client auth：表示 server 可以用该该证书对 client 提供的证书进行验证；
# "expiry": "175200h" 有效期20年

生成ca 证书和私钥

1
2
3
4
5
# 生成
cfssl gencert --initca /opt/certs/ca-csr.json | cfssljson --bare /opt/certs/ca

# verify
openssl x509 -in /opt/certs/ca.pem -text -noout

结果

1
2
3
4
5
6
7
8
9
10
# CSR configuration
/opt/certs/ca-csr.json
# CSR 双向认证
/opt/certs/ca.csr
# self-signed root CA public key  其它文档里会叫 ca.crt
/opt/certs/ca.pem
# self-signed root CA private key
/opt/certs/ca-key.pem
#  证书配置文件 for other TLS assets
/opt/certs/ca-config.json

使用私钥生成本地颁发的证书（etcd）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# peer 
cat > /opt/certs/etcd-158-ca-csr.json <<EOF
{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "maxzhao-ca",
      "OU": "etcd Security",
      "L": "NanJing",
      "ST": "Jiang Su",
      "C": "CN"
    }
  ],
  "CN": "etcd-158",
  "hosts": [
    "127.0.0.1",
    "192.168.2.158",
    "192.168.2.159",
    "192.168.2.160",
    "192.168.2.161",
    "10.96.0.1",
    "10.244.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local",
    "kubernetes.default.svc.cluster.158"
  ]
}
EOF
# 生成etcd用的证书文件 peer
cfssl gencert \
  --ca /opt/certs/ca.pem \
  --ca-key /opt/certs/ca-key.pem \
  --config /opt/certs/ca-config.json \
  -profile=etcd \
  /opt/certs/etcd-158-ca-csr.json | cfssljson --bare /opt/certs/etcd-158
#   --profile=k8s-server-client  表示客户端与服务端要双向通讯
# verify
openssl x509 -in /opt/certs/etcd-158.pem -text -noout

信任自签名的 CA 证书

1
2
3
yum install -y ca-certificates
cp /opt/certs/etcd/ca.pem /etc/pki/ca-trust/source/anchors/maxzhao-ca.crt
update-ca-trust

生成之后

1
2
3
4
5
6
7
8
9
10
11
12
13
# 传输到每一个节点
rm -rf /etc/certs/etcd
mkdir -p /etc/certs/etcd
\cp /opt/certs/ca.pem /etc/certs/etcd/ca.pem
\cp /opt/certs/etcd-158-key.pem /etc/certs/etcd/etcd-158-key.pem
\cp /opt/certs/etcd-158.pem /etc/certs/etcd/etcd-158.pem
# 拷贝 ca.pem, etcd-158.pem, etcd-158-key.pem
ssh root@192.168.2.159 "mkdir -p /etc/certs/etcd"
ssh root@192.168.2.160 "mkdir -p /etc/certs/etcd"
ssh root@192.168.2.161 "mkdir -p /etc/certs/etcd"
scp -r /etc/certs/etcd/* root@192.168.2.159:/etc/certs/etcd/
scp -r /etc/certs/etcd/* root@192.168.2.160:/etc/certs/etcd/
scp -r /etc/certs/etcd/* root@192.168.2.161:/etc/certs/etcd/

ca-cert-hash

1
2
#默认证书 /etc/kubernetes/pki/ca.crt
openssl x509 -pubkey -in /etc/certs/etcd/ca.pem | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

base64

1
cat /etc/certs/etcd/ca.pem | base64 -w 0

K8S 中

当前 ca base64 值

1
cat /etc/kubernetes/ca.pem | base64 -w 0

会在各个配置中，
比如：

  • .kube/config
  • /etc/kubernetes/kubelet.conf
  • /var/lib/kubelet/config.yaml

在 K8S 中使用自定义CA

kubernetes verion=1.24.1

为用户帐户配置证书

你必须手动配置以下管理员帐户和服务帐户

文件名 凭据名称 默认 CN O (位于 Subject 中)
admin.conf default-admin kubernetes-admin system:masters
kubelet.conf default-auth system:node:<nodeName> （参阅注释） system:nodes
controller-manager.conf default-controller-manager system:kube-controller-manager
scheduler.conf default-scheduler system:kube-scheduler 
1
2
3
4
5
6
7
8
9
10
11
KUBECONFIG=/etc/kubernetes/admin.conf 
#配置集群的管理员
kubectl config set-cluster default-cluster --server=https://192.168.2.159:6443 --certificate-authority /etc/certs/etcd/ca.pem --embed-certs
kubectl config set-cluster default-cluster --server=https://192.168.2.160:6443 --certificate-authority /etc/certs/etcd/ca.pem --embed-certs
kubectl config set-cluster default-cluster --server=https://192.168.2.161:6443 --certificate-authority /etc/certs/etcd/ca.pem --embed-certs
#	集群中的每个节点都需要一份
kubectl config set-credentials <credential-name> --client-key <path-to-key>.pem --client-certificate <path-to-cert>.pem --embed-certs
#必需添加到 manifests/kube-controller-manager.yaml 清单中
kubectl config set-context default-system --cluster default-cluster --user <credential-name>
# 必需添加到 manifests/kube-scheduler.yaml 清单中
kubectl config use-context default-system

PKI 证书和要求

1
2
3
4
/etc/kubernetes/admin.conf
/etc/kubernetes/kubelet.conf
/etc/kubernetes/controller-manager.conf
/etc/kubernetes/scheduler.conf

修复证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# root 用户执行
rm -f /etc/kubernetes/pki/apiserver.crt
rm -f /etc/kubernetes/pki/apiserver.key
rm -f /etc/kubernetes/pki/apiserver-kubelet-client.crt
rm -f /etc/kubernetes/pki/apiserver-kubelet-client.key
rm -f /etc/kubernetes/pki/apiserver.*
# 如果两个文件都已存在，则 kubeadm 将跳过生成步骤，使用现有文件。
kubeadm init phase certs apiserver  --config=kubeadm-config-init.yaml 
kubeadm init phase certs apiserver-kubelet-client  --config=kubeadm-config-init.yaml
kubectl get pods -A -o wide
kubectl delete pod -n kube-system kube-apiserver-master-158
kubectl delete pod -n kube-system kube-apiserver-master-159
kubectl delete pod -n kube-system kube-apiserver-master-160
systemctl restart kubelet

参考：

kubeadm init phase

[[Invalid x509 certificate for kubernetes master](https://stackoverflow.com/questions/46360361/invalid-x509-certificate-for-kubernetes-master)](

本文地址： https://github.com/maxzhao-it/blog/post/ebede6dg/