kubernetes - Kubernetes的GC不会删除退出的Docker容器

Question

我有一个运行Kubernetes 1.6.1的3个节点的集群，每个节点都有2个CPU和4G RAM。

我通过替换传递给容器的环境变量值来更新Pod模板哈希值，从而不断使用相同的Docker标签重新部署我的应用程序。
sed "s/THIS_STRING_IS_REPLACED_DURING_BUILD/$(date)/g" nginx-deployment.yml | kubectl replace -f -

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        env:
        - name: FOR_GODS_SAKE_PLEASE_REDEPLOY
          value: 'THIS_STRING_IS_REPLACED_DURING_BUILD'

如果执行此操作几百次，则无法再重新部署-新的Pod处于“待处理”状态。 kubectl get events产生以下内容:

Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  1h        50s     379 default-scheduler           Warning     
FailedScheduling    No nodes are available that match all of the following predicates:: Insufficient pods (3). 

同时，我可以在每个Kube节点上看到大约200个退出的nginx容器。

查看kube-controller-manager日志，我可以看到PodGC试图删除一些Pod，但是找不到它们。

I0516 12:53:41.137311       1 gc_controller.go:175] Found unscheduled terminating Pod nginx-deployment-2927112463-qczvv not assigned to any Node. Deleting.
I0516 12:53:41.137320       1 gc_controller.go:62] PodGC is force deleting Pod: default:nginx-deployment-2927112463-qczvv
E0516 12:53:41.190592       1 gc_controller.go:177] pods "nginx-deployment-2927112463-qczvv" not found
I0516 12:53:41.195020       1 gc_controller.go:175] Found unscheduled terminating Pod nginx-deployment-3265736979-jrpzb not assigned to any Node. Deleting.
I0516 12:53:41.195048       1 gc_controller.go:62] PodGC is force deleting Pod: default:nginx-deployment-3265736979-jrpzb
E0516 12:53:41.238307       1 gc_controller.go:177] pods "nginx-deployment-3265736979-jrpzb" not found

我有什么办法可以防止这种情况的发生？

Answer 1

Kubernetes允许您调整kubelet的垃圾收集标志。这可以通过更改标志--maximum-dead-containers或--maximum-dead-containers-per-container来完成。在此处阅读更多有关文档的信息:

Configuring kubelet Garbage Collection

Configuring Out Of Resource Handling

Garbage Collection