kubernetes - Kubernetes集群VirtualBox与网络有关的问题(NAT和仅主机适配器)

Question

我正在尝试在VirtualBox上设置一个kubernetes集群(两个节点，1个主节点，1个工作线程)。我的主机运行Windows 10，并在VirtualBox上安装了Codename cosmic Ubuntu 18.10。

我在每个VirtualBox上配置了两个适配器，一个NAT和一个仅主机适配器。我这样做是因为我需要使用主机IP(NAT)访问一些内部资源，并且还需要在主机和虚拟机之间建立一个稳定的网络(仅主机网络)。

我已经安装了Kubernetes v1.12.4，并成功地将worker加入了主节点。

NAME                STATUS   ROLES    AGE   VERSION
kubernetes-master   Ready    master   36m   v1.12.4
kubernetes-slave    Ready    <none>   25m   v1.12.4

我正在使用Flannel进行网络连接。

所有 pod 似乎都还可以。

 NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
default       nginx-server-7bb6997d9c-kdcld               1/1     Running   0          27m
kube-system   coredns-576cbf47c7-btrvb                    1/1     Running   1          38m
kube-system   coredns-576cbf47c7-zfscv                    1/1     Running   1          38m
kube-system   etcd-kubernetes-master                      1/1     Running   1          38m
kube-system   kube-apiserver-kubernetes-master            1/1     Running   1          38m
kube-system   kube-controller-manager-kubernetes-master   1/1     Running   1          38m
kube-system   kube-flannel-ds-amd64-29p96                 1/1     Running   1          28m
kube-system   kube-flannel-ds-amd64-sb2fq                 1/1     Running   1          37m
kube-system   kube-proxy-59v6b                            1/1     Running   1          38m
kube-system   kube-proxy-bfd78                            1/1     Running   0          28m
kube-system   kube-scheduler-kubernetes-master            1/1     Running   1          38m

我已经部署了Nginx来验证一切正常

NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   41m
nginx-http   ClusterIP   10.111.151.28   <none>        80/TCP    29m

但是，当我尝试到达nginx时，我超时了。 describe pod给了我以下事件。

Events:
  Type     Reason                  Age                 From                       Message
  ----     ------                  ----                ----                       -------
  Normal   Scheduled               32m                 default-scheduler          Successfully assigned default/nginx-server-7bb6997d9c-kdcld to kubernetes-slave
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dbb2595628fc2579c29779e31e27e27eaeff2dbcf2bdb68467c47f22a3590bd0" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "801e0f3f8ca4a9b7cc21d87d41141485e1b1da357f2d89e1644acf0ecf634016" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "77214c757449097bfbe05b24ebb5fd3c7f1d96f7e3e9a3cd48f3b37f30224feb" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ebffdd723083d916c0910489e12368dc4069dd99c24a3a4ab1b1d4ab823866ff" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d87b93815380246a05470e597a88d50eb31c132a50e30000ab41a456d1e65107" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "3ef233ef0a6c447134c7b027747a701d6576a80e76c9cc8ffd8287e8ee5f02a4" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6b621aab3c57154941b37360240228fe939b528855a5fe8cd9536df63d41ed93" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fa992bde90e0a1839180666bedaf74965fb26f3dccb33a66092836a25882ab44" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m                 kubelet, kubernetes-slave  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "81f74f687e17d67bd2853849f84ece33a118744278d78ac7af3bdeadff8aa9c7" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  32m (x2 over 32m)   kubelet, kubernetes-slave  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "29188c3e73d08e81b08b2258254dc2691fcaa514ecc96e9df86f2e61ba455b76" network for pod "nginx-server-7bb6997d9c-kdcld": NetworkPlugin cni failed to set up pod "nginx-server-7bb6997d9c-kdcld_default" network: open /run/flannel/subnet.env: no such file or directory
  Normal   SandboxChanged          32m (x11 over 32m)  kubelet, kubernetes-slave  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulling                 32m                 kubelet, kubernetes-slave  pulling image "nginx"
  Normal   Pulled                  32m                 kubelet, kubernetes-slave  Successfully pulled image "nginx"
  Normal   Created                 32m                 kubelet, kubernetes-slave  Created container

我尝试使用仅配置给虚拟机的网桥适配器进行完全相同的安装，然后一切正常。

我认为这是一个配置问题，但是我无法解决。有人可以告诉我。

Answer 1

正如我在已删除评论中提到的那样，我在Ubuntu 18.04主机上重新创建了它。创建了两个Ubuntu 18.10 VM，带有两个适配器(NAT和一个仅主机适配器)。我的配置与您在此处指定的配置相同。一切正常。

我要做的是手动添加第二个适配器，在节点上运行netplan和kubeadm init之前，我使用kubeadm join做到了。

以防万一您没有这样做-将仅主机适配器网络添加到/etc/netplan/50-cloud-init.yaml中的yaml文件中，然后运行sudo netplan generate和sudo netplan apply。对于nginx，我使用了Kubernetes官方文档中的deployment。然后，我公开了该服务:
kubectl create service nodeport nginx --tcp=80:80从主机 curl 我在NodePort上的节点IP地址可以正常工作。

这只是为了演示我的操作，使其在我的环境中起作用。从描述的pod错误来看，Flannel本身似乎有问题:
/run/flannel/subnet.env: no such file or directory
我在master上检查了这个目录，它看起来像这样:

/run/flannel/subnet.env

FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

请检查文件是否存在，如果这样做没有帮助，如果您提供更多信息，我们可以尝试进一步解决。但是，未知数太多，因此我不得不在某些地方进行猜测，我的建议是将其全部销毁，然后使用提供的信息重试，然后使用NodePort而不是ClusterIP类型运行nginx。只能从群集内部访问ClusterIP-例如Node。