kubernetes - 通过 Docker 在本地运行 Kubernetes 时主节点未启动

Question

我正在关注 Running Kubernetes locally via Docker引导，无法让master正常启动。

第一步:运行etcd

docker run --net=host -d gcr.io/google_containers/etcd:2.0.9/usr/local/bin/etcd --addr=127.0.0.1:4001 --bind-addr=0.0 .0.0:4001 --data-dir=/var/etcd/data

etcd 容器似乎正常启动。在 docker logs 中没有看到任何错误，我最终得到了一个监听 4001 的 etcd 进程。

第二步:运行master

docker run --net=host -d -v/var/run/docker.sock:/var/run/docker.sock gcr.io/google_containers/hyperkube:v0.21.2/hyperkube kubelet -- api_servers=http://localhost:8080 --v=2 --address=0.0.0.0 --enable_server --hostname_override=127.0.0.1 --config=/etc/kubernetes/manifests

我相信这就是我的问题开始的地方。以下是 docker logs 的输出:

W1021 13:23:04.093281       1 server.go:259] failed to set oom_score_adj to -900: write /proc/self/oom_score_adj: permission deniedW1021 13:23:04.093426       1 server.go:462] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead.W1021 13:23:04.093445       1 server.go:424] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults.I1021 13:23:04.093503       1 server.go:271] Using root directory: /var/lib/kubeletI1021 13:23:04.093519       1 plugins.go:69] No cloud provider specified.I1021 13:23:04.093526       1 server.go:290] Successfully initialized cloud provider: "" from the config file: ""I1021 13:23:05.126191       1 docker.go:289] Connecting to docker on unix:///var/run/docker.sockI1021 13:23:05.126396       1 server.go:651] Adding manifest file: /etc/kubernetes/manifestsI1021 13:23:05.126409       1 file.go:47] Watching path "/etc/kubernetes/manifests"I1021 13:23:05.126416       1 server.go:661] Watching apiserverE1021 13:23:05.127148       1 reflector.go:136] Failed to list *api.Pod: Get http://localhost:8080/api/v1/pods?fieldSelector=spec.nodeName%3D127.0.0.1: dial tcp 127.0.0.1:8080: connection refusedE1021 13:23:05.127295       1 reflector.go:136] Failed to list *api.Service: Get http://localhost:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refusedE1021 13:23:05.127336       1 reflector.go:136] Failed to list *api.Node: Get http://localhost:8080/api/v1/nodes?fieldSelector=metadata.name%3D127.0.0.1: dial tcp 127.0.0.1:8080: connection refusedI1021 13:23:05.343848       1 plugins.go:56] Registering credential provider: .dockercfgW1021 13:23:05.394268       1 container_manager_linux.go:96] Memory limit 0 for container /docker-daemon is too small, reset it to 157286400I1021 13:23:05.394284       1 container_manager_linux.go:100] Configure resource-only container /docker-daemon with memory limit: 157286400I1021 13:23:05.395019       1 plugins.go:180] Loaded volume plugin "kubernetes.io/aws-ebs"I1021 13:23:05.395040       1 plugins.go:180] Loaded volume plugin "kubernetes.io/empty-dir"I1021 13:23:05.395052       1 plugins.go:180] Loaded volume plugin "empty"I1021 13:23:05.395068       1 plugins.go:180] Loaded volume plugin "kubernetes.io/gce-pd"I1021 13:23:05.395080       1 plugins.go:180] Loaded volume plugin "gce-pd"I1021 13:23:05.395098       1 plugins.go:180] Loaded volume plugin "kubernetes.io/git-repo"I1021 13:23:05.395112       1 plugins.go:180] Loaded volume plugin "git"I1021 13:23:05.395124       1 plugins.go:180] Loaded volume plugin "kubernetes.io/host-path"I1021 13:23:05.395136       1 plugins.go:180] Loaded volume plugin "kubernetes.io/nfs"I1021 13:23:05.395147       1 plugins.go:180] Loaded volume plugin "kubernetes.io/secret"I1021 13:23:05.395156       1 plugins.go:180] Loaded volume plugin "kubernetes.io/iscsi"I1021 13:23:05.395166       1 plugins.go:180] Loaded volume plugin "kubernetes.io/glusterfs"I1021 13:23:05.395178       1 plugins.go:180] Loaded volume plugin "kubernetes.io/persistent-claim"I1021 13:23:05.395194       1 plugins.go:180] Loaded volume plugin "kubernetes.io/rbd"I1021 13:23:05.395274       1 server.go:623] Started kubeletI1021 13:23:05.395296       1 server.go:63] Starting to listen on 0.0.0.0:10250I1021 13:23:05.395507       1 server.go:82] Starting to listen read-only on 0.0.0.0:10255

第三步:运行服务代理

docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v0.21.2/hyperkube proxy --master=http://127.0.0.1:8080 --v=2

此步骤中的 docker 日志包含与我在第二步中看到的类似的错误。

I1021 13:32:03.177004       1 server.go:88] Running in resource-only container "/kube-proxy"I1021 13:32:03.177432       1 proxier.go:121] Setting proxy IP to 192.168.19.200 and initializing iptablesE1021 13:32:03.195731       1 api.go:108] Unable to load services: Get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refusedE1021 13:32:03.195924       1 api.go:180] Unable to load endpoints: Get http://127.0.0.1:8080/api/v1/endpoints: dial tcp 127.0.0.1:8080: connection refused

docker ps 输出:

CONTAINER ID        IMAGE                                        COMMAND                  CREATED             STATUS              PORTS               NAMES576d15c22537        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube proxy --m"   About an hour ago   Up About an hour                        high_pasteura98637c9d523        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube kubelet -"   About an hour ago   Up 34 minutes                           drunk_jones618afb1de613        gcr.io/google_containers/etcd:2.0.9          "/usr/local/bin/etcd "   2 hours ago         Up 2 hours                              high_yonath

Step Two 日志中的第一个错误让我相信该错误可能与 iptables 有关。

iptables -L 输出:

Chain INPUT (policy ACCEPT)target     prot opt source               destination         Chain FORWARD (policy ACCEPT)target     prot opt source               destination         DOCKER     all  --  anywhere             anywhere            ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHEDACCEPT     all  --  anywhere             anywhere            ACCEPT     all  --  anywhere             anywhere            Chain OUTPUT (policy ACCEPT)target     prot opt source               destination         Chain DOCKER (1 references)target     prot opt source               destination 

docker exec -ti a98637c9d523 cat/etc/kubernetes/manifests/master.json 输出:

{    "apiVersion": "v1",    "kind": "Pod",    "metadata": {"name":"k8s-master"},    "spec":{        "hostNetwork": true,        "containers":[            {                "name": "controller-manager",                "image": "gcr.io/google_containers/hyperkube:v1.0.6",                "command": [                    "/hyperkube",                    "controller-manager",                    "--master=127.0.0.1:8080",                    "--v=2"                ]            },            {                "name": "apiserver",                "image": "gcr.io/google_containers/hyperkube:v1.0.6",                "command": [                    "/hyperkube",                    "apiserver",                    "--portal-net=10.0.0.1/24",                    "--address=127.0.0.1",                    "--etcd-servers=http://127.0.0.1:4001",                    "--cluster-name=kubernetes",                    "--v=2"                ]            },            {                "name": "scheduler",                "image": "gcr.io/google_containers/hyperkube:v1.0.6",                "command": [                    "/hyperkube",                    "scheduler",                    "--master=127.0.0.1:8080",                    "--v=2"                 ]            }        ]    }}

Docker 版本 1.8.3

内核版本 4.2.3

如有任何见解，我们将不胜感激。

Answer 1

可以先把docker版本降到1.7.2吗？我用 docker 1.7.2 做了你上面所做的，一切正常。

$ curl 127.0.0.1:8080/

{
  "paths": [
    "/api",
    "/api/v1",
    "/api/v1beta3",
    "/healthz",
    "/healthz/ping",
    "/logs/",
    "/metrics",
    "/resetMetrics",
    "/swagger-ui/",
    "/swaggerapi/",
    "/ui/",
    "/version"
  ]
}

$ docker ps

CONTAINER ID        IMAGE                                        COMMAND                CREATED             STATUS              PORTS               NAMES
0141e596414c        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube proxy --   15 minutes ago      Up 15 minutes                           nostalgic_nobel                                                                                          
10634ce798e9        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube schedule   16 minutes ago      Up 16 minutes                           k8s_scheduler.b725e775_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_43562383            
5618a39eb11d        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube apiserve   16 minutes ago      Up 16 minutes                           k8s_apiserver.70750283_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_e5d145be            
25f336102b26        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube controll   16 minutes ago      Up 16 minutes                           k8s_controller-manager.aad1ee8f_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_fe538b9b   
7f1391840920        gcr.io/google_containers/pause:0.8.0         "/pause"               17 minutes ago      Up 17 minutes                           k8s_POD.e4cc795_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_26fd84fd                   
a11715435f45        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube kubelet    17 minutes ago      Up 17 minutes                           jovial_hodgkin                                                                                           
a882a1a4b917        gcr.io/google_containers/etcd:2.0.9          "/usr/local/bin/etcd   18 minutes ago      Up 18 minutes                           adoring_hodgkin                                                                                          

docker 1.8.3 有几个已知问题，尤其是 docker#17190 .我们必须通过 kubernetes#16052 解决此类问题.但是这些变化并不是 Kubernetes 1.0 版本中特意挑选出来的。从您上面发布的输出中，我注意到没有暂停容器。您也可以运行 docker ps -a检查某些容器是否已死，并复制并粘贴 docker logs <dead-container> 的输出在这里？

我将提交一个问题以确保 Kubernetes 1.1 版本与 docker 1.8.3 一起正常工作。谢谢!