重启后 Docker 服务无法启动 - 无法启动 containerd

Question

我们的一个虚拟机已重启，现在 docker 无法启动。当我尝试通过 systemctl start docker-standalone 启动服务时，我得到:

Job for docker-standalone.service failed because the control processexited with error code. See "systemctl statusdocker-standalone.service" and "journalctl -xe" for details.

当我检查 journalctl -xe 时，我看到了这个:

Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.689962814-06:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit sta Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.689984624-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690001784-06:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured" Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690010545-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690029639-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690106664-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690248278-06:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker-standalone/containerd/daemon/io.c Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690261292-06:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690274990-06:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured" Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690282381-06:00" level=info msg="metadata content store policy set" policy=shared Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690356752-06:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690370466-06:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690399678-06:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690427410-06:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690438870-06:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690449099-06:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690462529-06:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690476538-06:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690489694-06:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690511122-06:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690524355-06:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1 Oct 29 13:56:16 server35.mydomain.com dockerd[4382]: Failed to start containerd: timeout waiting for containerd to start Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service: main process exited, code=exited, status=1/FAILURE Oct 29 13:56:16 server35.mydomain.com systemd[1]: Failed to start Docker Application Container Engine Standalone Version.
-- Subject: Unit docker-standalone.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-standalone.service has failed.
--
-- The result is failed. Oct 29 13:56:16 server35.mydomain.com systemd[1]: Unit docker-standalone.service entered failed state. Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service failed. Oct 29 13:56:16 server35.mydomain.com polkitd[800]: Unregistered Authentication Agent for unix-process:4406:22575 (system bus name :1.24, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected f Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service holdoff time over, scheduling restart. Oct 29 13:56:16 server35.mydomain.com systemd[1]: Stopped Docker Application Container Engine Standalone Version.
-- Subject: Unit docker-standalone.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-standalone.service has finished shutting down. Oct 29 13:56:16 server35.mydomain.com systemd[1]: start request repeated too quickly for docker-standalone.service Oct 29 13:56:16 server35.mydomain.com systemd[1]: Failed to start Docker Application Container Engine Standalone Version.
-- Subject: Unit docker-standalone.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-standalone.service has failed.
--
-- The result is failed. Oct 29 13:56:16 server35.mydomain.com systemd[1]: Unit docker-standalone.service entered failed state. Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service failed.Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.689962814-06:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit sta Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.689984624-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690001784-06:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured" Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690010545-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690029639-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690106664-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690248278-06:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker-standalone/containerd/daemon/io.c Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690261292-06:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690274990-06:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured" Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690282381-06:00" level=info msg="metadata content store policy set" policy=shared Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690356752-06:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690370466-06:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690399678-06:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690427410-06:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690438870-06:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690449099-06:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690462529-06:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690476538-06:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690489694-06:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690511122-06:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690524355-06:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1 Oct 29 13:56:16 server35.mydomain.com dockerd[4382]: Failed to start containerd: timeout waiting for containerd to start Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service: main process exited, code=exited, status=1/FAILURE Oct 29 13:56:16 server35.mydomain.com systemd[1]: Failed to start Docker Application Container Engine Standalone Version.
-- Subject: Unit docker-standalone.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-standalone.service has failed.
--
-- The result is failed. Oct 29 13:56:16 server35.mydomain.com systemd[1]: Unit docker-standalone.service entered failed state. Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service failed. Oct 29 13:56:16 server35.mydomain.com polkitd[800]: Unregistered Authentication Agent for unix-process:4406:22575 (system bus name :1.24, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected f Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service holdoff time over, scheduling restart. Oct 29 13:56:16 server35.mydomain.com systemd[1]: Stopped Docker Application Container Engine Standalone Version.
-- Subject: Unit docker-standalone.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-standalone.service has finished shutting down. Oct 29 13:56:16 server35.mydomain.com systemd[1]: start request repeated too quickly for docker-standalone.service Oct 29 13:56:16 server35.mydomain.com systemd[1]: Failed to start Docker Application Container Engine Standalone Version.
-- Subject: Unit docker-standalone.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-standalone.service has failed.
--
-- The result is failed. Oct 29 13:56:16 server35.mydomain.com systemd[1]: Unit docker-standalone.service entered failed state. Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service failed.

我注意到这条关键线:

containerd 启动失败:WAITING containerd 启动超时

我们尝试重新安装 docker，但没有帮助。

我们的版本是:

$ ds version
Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.40
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:55:49 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:02:21 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

我们正在运行带有 Linux 内核 3.10.0-514.el7.x86_64 的 Centos 7

我们还尝试根据建议删除 /var/run 中的 docker 文件 in this SO post ，但那是 Ubuntu 并且涉及 snap，清除 /var/run 中的 docker 文件没有帮助。

为了消除对服务名称的任何混淆，ds 只是一个帮助脚本，定义如下:

#!/bin/bash

docker -H unix:///var/run/docker-standalone.sock "$@"

我们如何让 docker 重新启动？

Answer 1

此处记录了解决方案:https://github.com/docker/for-linux/issues/475

我们需要做的就是运行:

systemctl daemon-reload
systemctl restart containerd.service
systemctl start docker