kubernetes - 身份验证握手失败 : x509: certificate signed by unknown authority

Question

我正在使用此命令启动 kubernetes api 服务器 (v1.15.3):

systemctl start kube-apiserver.service

这是日志输出:

● kube-apiserver.service - Kubernetes API Service
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
   Active: activating (start) since 六 2019-08-24 20:12:18 CST; 4s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 9563 (kube-apiserver)
    Tasks: 13
   Memory: 11.0M
   CGroup: /system.slice/kube-apiserver.service
           └─9563 /usr/local/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=https://172.19.104.231:2379,https://172.19.104.230:2379,https://172.19.150.82:2379 --advertise-address=172.19.104.231 --bind-address=172.19.104.231 --insecure-bind-address=172.19.104.231 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota --authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h

8月 24 20:12:19 iZuf63refzweg1d9dh94t8Z kube-apiserver[9563]: W0824 20:12:19.994504    9563 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {172.19.150.82:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes\")". Reconnecting...
8月 24 20:12:20 iZuf63refzweg1d9dh94t8Z kube-apiserver[9563]: W0824 20:12:20.985988    9563 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {172.19.104.231:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes\")". Reconnecting...
8月 24 20:12:20 iZuf63refzweg1d9dh94t8Z kube-apiserver[9563]: W0824 20:12:20.986331    9563 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {172.19.104.230:2379 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kubernetes\")". Reconnecting...

kubernetes config(kubernetes-csr.json) 的 CA 证书:

{
    "CN": "kubernetes",
    "hosts": [
      "127.0.0.1",
      "172.19.104.230",
      "172.19.150.82",
      "172.19.104.231"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

我应该怎么做才能解决这个问题？我已经在 CentOS 7 中尝试过自签名证书:

openssl x509 -outform der -in kubernetes.pem -out kubernetes.crt
cp /data/k8s/ssl/kubernetes.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust

我的 etcd 集群使用相同的证书文件。这是生成证书的命令:

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

这是 etcd 列表:

[root@iZuf63refzweg1d9dh94t8Z ssl]# etcdctl member list
55a782166ce91d01, started, infra3, https://172.19.150.82:2380, https://172.19.150.82:2379
67bca27e43a8258a, started, infra2, https://172.19.104.230:2380,
696a771758a889c4, started, infra1, https://172.19.104.231:2380, https://172.19.104.231:2379

Answer 1

这可能是你的证书文件产生了encount警告，你应该使用新版本的cfssl(v1.2以上)，并确保没有警告。这是由于使用cfssl(v1.3)时的这个提示引起的生成证书:

This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements")

尝试将 cfssl 升级到 v1.3.4 并重新生成证书。

/usr/local/go/bin/go get -u github.com/cloudflare/cfssl/cmd/cfssl

验证版本。

[root@iZuf63refzweg1d9dh94t8Z ssl]# /root/go/bin/cfssl version
Version: 1.3.4
Revision: dev
Runtime: go1.12.9