docker - 无法在 etcd3 中找到 Kubernetes apiserver 的数据

Question

我正在使用来自 kargo 的 ansible 脚本来构建我的集群。尽管查看了来自 apiserver 的详细日志，但我无法找到数据存储在 etcd3 中的位置。

这是我看到的 hyperkube apiserver 日志:

$ docker logs k8s_kube-apiserver.fd19548d_kube-apiserver-kube-master-01_kube-system_2f6ad6b0bf81ca6a0e2b4d499a25fc89_aa25196e
[[ SNIP ]]
I0127 23:31:55.871267       1 storage_factory.go:242] storing { podtemplates} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>}
I0127 23:31:55.875975       1 storage_factory.go:242] storing { events} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>}
I0127 23:31:55.876169       1 reflector.go:234] Listing and watching *api.PodTemplate from k8s.io/kubernetes/pkg/storage/cacher.go:215
I0127 23:31:55.877950       1 compact.go:55] compactor already exists for endpoints [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379]
I0127 23:31:55.878148       1 storage_factory.go:242] storing { limitranges} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>}
I0127 23:31:55.879372       1 compact.go:55] compactor already exists for endpoints [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379]

hyperkube apiserver 使用这些参数启动:

$ docker inspect k8s_kube-apiserver.b6395694_kube-apiserver-kube-master-01_kube-system_2f6ad6b0bf81ca6a0e2b4d499a25fc89_4338b366
[
    {
        "Id": "33c76fa64bbd5d5a656e329cf87ed3707077659c69dc281127751f594460242b",
        "Created": "2017-01-27T23:35:10.691147667Z",
        "Path": "/hyperkube",
        "Args": [
            "apiserver",
            "--advertise-address=10.60.68.23",
            "--etcd-servers=https://10.60.68.11:2379,https://10.60.68.39:2379,https://10.60.68.35:2379",
            "--etcd-quorum-read=true",
            "--etcd-cafile=/etc/ssl/etcd/ssl/ca.pem",
            "--etcd-certfile=/etc/ssl/etcd/ssl/node-kube-master-01.pem",
            "--etcd-keyfile=/etc/ssl/etcd/ssl/node-kube-master-01-key.pem",
            "--insecure-bind-address=127.0.0.1",
            "--apiserver-count=3",
            "--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota",
            "--service-cluster-ip-range=10.233.0.0/18",
            "--service-node-port-range=30000-32767",
            "--client-ca-file=/etc/kubernetes/ssl/ca.pem",
            "--basic-auth-file=/etc/kubernetes/users/known_users.csv",
            "--tls-cert-file=/etc/kubernetes/ssl/apiserver.pem",
            "--tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem",
            "--token-auth-file=/etc/kubernetes/tokens/known_tokens.csv",
            "--service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem",
            "--secure-port=443",
            "--insecure-port=8080",
            "--v=4",
            "--allow-privileged=true",
            "--cloud-provider=openstack",
            "--cloud-config=/etc/kubernetes/cloud_config",
            "--anonymous-auth=False"
        ],

它没有在哪里覆盖 /registry 的默认 etcd 前缀。我不知道 apiserver 在哪里存储数据。

$ docker exec -it etcd3 etcdctl --peers https://10.60.68.11:2379 ls /registry
Error:  100: Key not found (/registry) [163]

Answer 1

通过 kubernetes 获取存储在 etcd v3 中的键和值:

ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get --prefix /registry

从 etcd v3 获取 1 个指定的键和值，例如:

ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get /registry/services/specs/default/kubernetes

基于:https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/interacting_v3.md