linux - 尽管 df 显示很多，但 aufs 上的 Docker 磁盘空间不足

Question

症状:无法启动任何容器

$ docker run continuumio/miniconda3
docker: Error response from daemon: mkdir /var/run/docker/libcontainerd/a40677f8a1bbaf43d0bd3c1935c4ee69330ac1b0c6bb577ebf66f78edaae2d48: no space left on device.
$ sudo df -h /var/run/docker/libcontainerd
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           201M  748K  200M   1% /run

我已经删除了退出的容器和悬空或旧图像。一些信息(我在 2 GB 内存的 AWS 上运行 Ubuntu):

$ docker info
Containers: 9
 Running: 7
 Paused: 0
 Stopped: 2
Images: 278
Server Version: 1.12.3
Storage Driver: aufs
 Root Dir: /home/ubuntu/docker/volumes/docker_images/var/aufs
 Backing Filesystem: extfs
 Dirs: 336
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge overlay null host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-48-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.954 GiB
Name: ip-192-168-1-205
ID: 2CFI:6RYE:PRVQ:ZOSP:4DNU:O727:IQ57:BL4H:TGMD:L3KY:HD7G:VAUJ
Docker Root Dir: /home/ubuntu/docker/volumes/docker_images/var/aufs
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
$ uname -a
Linux ip-192-168-1-205 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

我的 /var/lib/docker 是一个符号链接(symbolic link)到 /home/ubuntu/docker/volumes/docker_images/var/

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            996M   12K  996M   1% /dev
tmpfs           201M  748K  200M   1% /run
/dev/xvda1      7.8G  2.2G  5.2G  30% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none           1001M  592K 1000M   1% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvdf        16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images
/dev/xvdg        99G  3.3G   91G   4% /home/ubuntu/docker/volumes/hub
none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/e693c6c76faf6bcc61b262cda110985f491f3bf8dcb3557d0bce857027759fc8
none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/9915d2a8761471958ba4268a15cca6149ddedfdfa9007cfc718536cd4189e9ae
none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/1ff065d23fd79042d62697c678f980eb5e7bb08493422e905835b064dad96e66
none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/e94aed0d3162e222de6c96a4e897d70e82f1fd24dbc8a57aa4a5c4ddc15fc827
none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/9aa83823063aab902fa82515d7cb5c6ee3d3e64a789be23367d70d445a319fb6
none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/a800385412c324efd6bdd7df48aee0c5db7f69a8d333b60b9ae96839b05339d2
none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/e109c3c4aa7acefcc69f2436275a9d61bdf8b1b968e150d9063bbbd276e5cf3c

有什么解决办法吗？我的构建系统停滞了:/我没有尝试重新启动 docker 守护进程，因为它需要一些手动操作来重新启动每个容器。

更新:

$ df -i
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
udev            254821    398  254423    1% /dev
tmpfs           256060 256060       0  100% /run
/dev/xvda1      524288  86504  437784   17% /
none            256060     11  256049    1% /sys/fs/cgroup
none            256060      1  256059    1% /run/lock
none            256060     25  256035    1% /run/shm
none            256060      2  256058    1% /run/user
/dev/xvdf      1048576 150320  898256   15% /home/ubuntu/docker/volumes/docker_images
/dev/xvdg      6553600   3207 6550393    1% /home/ubuntu/docker/volumes/hub

tmpfs 100% 使用!接下来我需要弄清楚我可以删除什么。

更新 2:我被 https://github.com/docker/docker/issues/22513 击中了

我的解决方案(以 root 身份运行):

# ifconfig | grep veth | cut -d' ' -f1 | xargs echo
veth087c8e6 veth1fb6674 veth5b8c14c veth801416c veth8a51820

# find /run/network -name 'ifstate.*' | egrep -w -v -e `ifconfig | grep veth | cut -d' ' -f1 | xargs echo | tr ' ' '|'` | xargs -n1 rm

Answer 1

正如我在问题中所描述的(我是 OP)，问题是由于 https://github.com/docker/docker/issues/22513 中描述的内核错误导致 inode 耗尽

我的解决方案是删除那些与 ifconfig 列出的任何接口(interface)都不对应的 ifstate 文件:

# ifconfig | grep veth | cut -d' ' -f1 | xargs echo
veth087c8e6 veth1fb6674 veth5b8c14c veth801416c veth8a51820

# find /run/network -name 'ifstate.*' | egrep -w -v -e `ifconfig | grep veth | cut -d' ' -f1 | xargs echo | tr ' ' '|'` | xargs -n1 rm