kubernetes - 为什么 kubelet 启动容器比 docker cli 慢？

Question

我试图理解为什么我的容器中的一个容器在由 kubelet 启动时比直接在 GKE 节点本身上通过 docker cli 启动时更慢。

这是 kubelet 日志。容器已启动，但处于未就绪状态 23 秒:

18:49:55.000 Container image "eu.gcr.io/proj/ns/myimage@sha256:fff668" already present on machine
18:49:55.000 Created container
18:49:56.000 Started container
18:49:56.000 Readiness probe failed: cat: /tmp/healthy: No such file or directory
18:49:58.000 Readiness probe failed: cat: /tmp/healthy: No such file or directory
18:50:00.000 Readiness probe failed: cat: /tmp/healthy: No such file or directory
18:50:02.000 Readiness probe failed: cat: /tmp/healthy: No such file or directory
18:50:04.000 Readiness probe failed: cat: /tmp/healthy: No such file or directory
18:50:06.000 Readiness probe failed: cat: /tmp/healthy: No such file or directory
18:50:08.000 Readiness probe failed: cat: /tmp/healthy: No such file or directory
18:50:10.000 Readiness probe failed: cat: /tmp/healthy: No such file or directory
18:50:12.000 Readiness probe failed: cat: /tmp/healthy: No such file or directory
18:50:14.000 Readiness probe failed: cat: /tmp/healthy: No such file or directory
18:50:16.000 Readiness probe failed: cat: /tmp/healthy: No such file or directory
18:50:18.000 Readiness probe failed: cat: /tmp/healthy: No such file or directory

最后，容器实际上在 23 秒后才真正启动。我知道这一点，因为它所做的第一件事就是打印以下日志行，然后为 readinessProbe 编写/tmp/healthy 文件。

18:50:18.000 17:50:18,572|MainThread|INFO|cli|Starting application 

但是，正如以下命令通过打印当前日期显示的那样，然后使用 docker cli 启动容器(在与上面运行 kubelet 相同的节点上)启动容器应该只需要大约 1 秒。

mark@gke-cluster-3 ~ $ date ++%Y-%m-%d %H:%M:%S.%N; docker run -it eu.gcr.io/proj/ns/myimage@sha256:fff668
2017-11-25 16:37:01.188799045
2017-11-25 16:37:02,246|MainThread|INFO|cli|Starting application

这让我有点发疯!关于可能导致这种情况的任何想法都欢迎:)

Answer 1

事实证明，这些容器启动缓慢的问题是 Python 解释器在启动期间的 CPU 受限。我添加了一个 bash 脚本，它会在启动 Python 进程之前打印日期时间，当改变容器可用的 CPU 资源时，问题变得非常清楚。

cpu: 10m

2017-12-18 08:05:46,1513584346 starting script
2017-12-18 08:06:22,318|MainThread|INFO|cli|Application startup

cpu: 50m

2017-12-18 08:15:11,1513584911 starting script
2017-12-18 08:15:27,317|MainThread|INFO|cli|Application startup

cpu: 100m

2017-12-18 08:07:46,1513584466 starting script
2017-12-18 08:07:53,218|MainThread|INFO|cli|Application startup

cpu: 150m

2017-12-18 08:18:16,1513585096 starting script
2017-12-18 08:18:20,730|MainThread|INFO|cli|Application startup

cpu: 200m

2017-12-18 08:09:14,1513584554 starting script
2017-12-18 08:09:17,922|MainThread|INFO|cli|Application startup

这有点令人沮丧，因为应用程序在运行时消耗大约 10m CPU。我将从这里调查模块导入和其他建议: https://lwn.net/Articles/730915/