kubernetes - 几小时的正常运行时间后，Kubernetes开始出现错误

Question

我在this guide之后在OpenStack上安装了K8S。

安装进行得很好，我可以运行pod了，但是一段时间后我的应用程序停止运行。我仍然可以创建Pod，但是请求不会从集群外部以及Pod内部到达服务。基本上，网络中的某些东西会变得混乱。 iptables -L -vnt nat 仍显示正确的配置，但无法正常工作。

为了使其正常运行，我必须重建集群，删除所有服务，并且复制 Controller 不起作用。

我试图查看日志。以下是kube-proxy的日记:

Dec 20 02:12:18 minion01.novalocal systemd[1]: Started Kubernetes Proxy.
    Dec 20 02:15:52 minion01.novalocal kube-proxy[1030]: I1220 02:15:52.269784    1030 proxier.go:487] Opened iptables from-containers public port for service "default/opensips:sipt" on TCP port 5060
    Dec 20 02:15:52 minion01.novalocal kube-proxy[1030]: I1220 02:15:52.278952    1030 proxier.go:498] Opened iptables from-host public port for service "default/opensips:sipt" on TCP port 5060
    Dec 20 03:05:11 minion01.novalocal kube-proxy[1030]: W1220 03:05:11.806927    1030 api.go:224] Got error status on WatchEndpoints channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested index is outdated and cleared (the requested history has been cleared [1433/544]) [2432] Reason: Details:<nil> Code:0}
    Dec 20 03:06:08 minion01.novalocal kube-proxy[1030]: W1220 03:06:08.177225    1030 api.go:153] Got error status on WatchServices channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink: ResourceVersion:} Status:Failure Message:401: The event in requested index is outdated and cleared (the requested history has been cleared [1476/207]) [2475] Reason: Details:<nil> Code:0}
..
..
..
Dec 20 16:01:23 minion01.novalocal kube-proxy[1030]: E1220 16:01:23.448570    1030 proxier.go:161] Failed to ensure iptables: error creating chain "KUBE-PORTALS-CONTAINER": fork/exec /usr/sbin/iptables: too many open files:
Dec 20 16:01:23 minion01.novalocal kube-proxy[1030]: W1220 16:01:23.448749    1030 iptables.go:203] Error checking iptables version, assuming version at least 1.4.11: %vfork/exec /usr/sbin/iptables: too many open files
Dec 20 16:01:23 minion01.novalocal kube-proxy[1030]: E1220 16:01:23.448868    1030 proxier.go:409] Failed to install iptables KUBE-PORTALS-CONTAINER rule for service "default/kubernetes:"
Dec 20 16:01:23 minion01.novalocal kube-proxy[1030]: E1220 16:01:23.448906    1030 proxier.go:176] Failed to ensure portal for "default/kubernetes:": error checking rule: fork/exec /usr/sbin/iptables: too many open files:
Dec 20 16:01:23 minion01.novalocal kube-proxy[1030]: W1220 16:01:23.449006    1030 iptables.go:203] Error checking iptables version, assuming version at least 1.4.11: %vfork/exec /usr/sbin/iptables: too many open files
Dec 20 16:01:23 minion01.novalocal kube-proxy[1030]: E1220 16:01:23.449133    1030 proxier.go:409] Failed to install iptables KUBE-PORTALS-CONTAINER rule for service "default/repo-client:"

我发现很少有与“无法安装iptables”有关的帖子，但它们似乎并不相关，因为最初一切正常，但几个小时后便陷入混乱。

Answer 1

这是什么版本的Kubernetes？很久以前(〜1.0.4)，我们在kube-proxy中有一个错误，它泄漏了套接字/文件描述符。

如果您没有运行1.1.3二进制文件，请考虑升级。

另外，您应该能够使用lsof找出谁打开了所有文件。