kubernetes - GKE 1.10 kubernetes 集群上的网络连接/DNS 问题

Question

我在 GKE 1.10 kubernetes 集群上遇到了 DNS 问题。有时 pod 在没有任何网络连接的情况下启动。重新启动 pod 往往会解决问题。

这是在没有网络和有网络的容器中执行相同命令的结果。

损坏:

kc exec -it -n  iotest app1-b67598997-p9lqk  -c userapp sh

/app $ nslookup www.google.com
nslookup: can't resolve '(null)': Name does not resolve

/app $ cat /etc/resolv.conf
nameserver 10.63.240.10
search iotest.svc.cluster.local svc.cluster.local cluster.local c.myproj.internal google.internal
options ndots:5

/app $ curl -I 10.63.240.10
curl: (7) Failed to connect to 10.63.240.10 port 80: Connection refused

/app $ netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8001          0.0.0.0:*               LISTEN      1/python
tcp        0      0 ::1:50051               :::*                    LISTEN      1/python
tcp        0      0 ::ffff:127.0.0.1:50051  :::*                    LISTEN      1/python

工作:

kc exec -it -n  iotest app1-7d985bfd7b-h5dbr -c userapp sh

/app $ nslookup www.google.com
nslookup: can't resolve '(null)': Name does not resolve

Name:      www.google.com
Address 1: 74.125.206.147 wk-in-f147.1e100.net
Address 2: 74.125.206.105 wk-in-f105.1e100.net
Address 3: 74.125.206.99 wk-in-f99.1e100.net
Address 4: 74.125.206.104 wk-in-f104.1e100.net
Address 5: 74.125.206.106 wk-in-f106.1e100.net
Address 6: 74.125.206.103 wk-in-f103.1e100.net
Address 7: 2a00:1450:400c:c04::68 wk-in-x68.1e100.net

/app $ cat /etc/resolv.conf
nameserver 10.63.240.10
search iotest.svc.cluster.local svc.cluster.local cluster.local c.myproj.internal google.internal
options ndots:5

/app $ curl -I 10.63.240.10
HTTP/1.1 404 Not Found
date: Sun, 29 Jul 2018 15:13:47 GMT
server: envoy
content-length: 0

/app $ netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:15000         0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:15001           0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:8001          0.0.0.0:*               LISTEN      1/python
tcp        0      0 10.60.2.6:56508         10.60.48.22:9091        ESTABLISHED -
tcp        0      0 127.0.0.1:57768         127.0.0.1:50051         ESTABLISHED -
tcp        0      0 10.60.2.6:43334         10.63.255.44:15011      ESTABLISHED -
tcp        0      0 10.60.2.6:15001         10.60.45.26:57160       ESTABLISHED -
tcp        0      0 10.60.2.6:48946         10.60.45.28:9091        ESTABLISHED -
tcp        0      0 127.0.0.1:49804         127.0.0.1:50051         ESTABLISHED -
tcp        0      0 ::1:50051               :::*                    LISTEN      1/python
tcp        0      0 ::ffff:127.0.0.1:50051  :::*                    LISTEN      1/python
tcp        0      0 ::ffff:127.0.0.1:50051  ::ffff:127.0.0.1:49804  ESTABLISHED 1/python
tcp        0      0 ::ffff:127.0.0.1:50051  ::ffff:127.0.0.1:57768  ESTABLISHED 1/python

这些 pod 是相同的，只有一个被重启了。

有没有人对如何分析和解决这个问题有建议？

Answer 1

尝试的一些步骤:

1) ifconfig eth0 或任何主接口(interface)。接口(interface)起来了吗？ tx 和 rx 数据包计数是否在增加？

2) 如果接口(interface)正常，您可以在运行您发布的 nslookup 命令时尝试 tcpdump。查看 dns 请求数据包是否发送出去。

3) 当网络连接中断时，查看 pod 被安排在哪个节点上。也许它每次都在同一个节点上？如果是，该节点上的其他 pod 是否遇到类似问题？