docker - pod 死了。容器在 K8s 1.13 中启动

Question

有没有人遇到过 Pod 从一个节点(即节点 A)被逐出然后立即安排在另一个节点(即节点 B)上的情况。
但是 Node A 的 kubelet 一直提示无法删除 pod 的容器。容器已启动并且没有被杀死。

Aug 30 20:29:36 staging-node-4 kubelet[2173]: I0830 20:29:36.358238    2173 kubelet_pods.go:1073] Killing unwanted pod "thanos-compactor-0"
Aug 30 20:29:36 staging-node-4 kubelet[2173]: I0830 20:29:36.362581    2173 kuberuntime_container.go:559] Killing container "docker://b22287cd406c3fe9eff4ff2df1792c6f84b5b92d001359f05ea73f8788715609" with 30 second grace period
Aug 30 20:29:36 staging-node-4 kubelet[2173]: E0830 20:29:36.363416    2173 kuberuntime_container.go:71] Can't make a ref to pod "thanos-compactor-0_thanos(23096418-c7b2-11e9-9be7-005056bc2883)", container thanos: selfLink was empty, can't make reference
Aug 30 20:29:38 staging-node-4 kubelet[2173]: I0830 20:29:38.362191    2173 kubelet_pods.go:1073] Killing unwanted pod "thanos-compactor-0"
Aug 30 20:29:38 staging-node-4 kubelet[2173]: I0830 20:29:38.366884    2173 kuberuntime_container.go:559] Killing container "docker://b22287cd406c3fe9eff4ff2df1792c6f84b5b92d001359f05ea73f8788715609" with 30 second grace period
Aug 30 20:29:38 staging-node-4 kubelet[2173]: E0830 20:29:38.367901    2173 kuberuntime_container.go:71] Can't make a ref to pod "thanos-compactor-0_thanos(23096418-c7b2-11e9-9be7-005056bc2883)", container thanos: selfLink was empty, can't make reference
Aug 30 20:29:38 staging-node-4 kubelet[2173]: I0830 20:29:38.997510    2173 setters.go:72] Using node IP: "172.17.21.146"
Aug 30 20:29:40 staging-node-4 kubelet[2173]: I0830 20:29:40.364388    2173 kubelet_pods.go:1073] Killing unwanted pod "thanos-compactor-0"
Aug 30 20:29:40 staging-node-4 kubelet[2173]: I0830 20:29:40.368226    2173 kuberuntime_container.go:559] Killing container "docker://b22287cd406c3fe9eff4ff2df1792c6f84b5b92d001359f05ea73f8788715609" with 30 second grace period
Aug 30 20:29:40 staging-node-4 kubelet[2173]: E0830 20:29:40.369235    2173 kuberuntime_container.go:71] Can't make a ref to pod "thanos-compactor-0_thanos(23096418-c7b2-11e9-9be7-005056bc2883)", container thanos: selfLink was empty, can't make reference

b22287cd 是没有被删除的 docker containerid。

Answer 1

几周前我遇到了这个问题。
Pod 被逐出节点，有时会“卡”在终止状态。

通过运行 kubectl -n <NAMESPACE> get pod 检查您的 pod 状态

使用 kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>解决了这个问题。