python - 使用 PyTorch 分布式 NCCL 连接失败

Question

我正在尝试使用 torch.distributed 将 PyTorch 张量从一台机器发送到另一台机器。 dist.init_process_group 函数正常工作。但是，dist.broadcast 函数中存在连接失败。这是我在节点 0 上的代码:

import torch
from torch import distributed as dist
import numpy as np
import os

master_addr = '47.xxx.xxx.xx'
master_port = 10000
world_size = 2
rank = 0
backend = 'nccl'

os.environ['MASTER_ADDR'] = master_addr
os.environ['MASTER_PORT'] = str(master_port)
os.environ['WORLD_SIZE'] = str(world_size)
os.environ['RANK'] = str(rank)
dist.init_process_group(backend, init_method='tcp://47.xxx.xxx.xx:10000', timeout=datetime.timedelta(0, 10), rank=rank, world_size=world_size)
print("Finished initializing process group; backend: %s, rank: %d, "
"world_size: %d" % (backend, rank, world_size))

a = torch.from_numpy(np.random.rand(3, 3)).cuda()
dist.broadcast(tensor=a, src=0)

这是我在节点 1 上的代码:

import torch
from torch import distributed as dist
import numpy as np
import os

master_addr = '47.xxx.xxx.xx'
master_port = 10000
world_size = 2
rank = 1
backend = 'nccl'

os.environ['MASTER_ADDR'] = master_addr
os.environ['MASTER_PORT'] = str(master_port)
os.environ['WORLD_SIZE'] = str(world_size)
os.environ['RANK'] = str(rank)
dist.init_process_group(backend, init_method='tcp://47.xxx.xxx.xx:10000', timeout=datetime.timedelta(0, 10), rank=rank, world_size=world_size)
print("Finished initializing process group; backend: %s, rank: %d, "
"world_size: %d" % (backend, rank, world_size))

a = torch.zeros((3,3)).cuda()
dist.broadcast(tensor=a, src=0)

我在运行代码之前设置了 NCCL_DEBUG=INFO。这是我在节点 1 上获得的信息:

iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.0.181<0>
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/IB : No device found.
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.0.181<0>
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Setting affinity for GPU 0 to ffff
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Call to connect returned Connection timed out, retrying
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Call to connect returned Connection timed out, retrying

iZbp11ufz31riqnssil53cZ:13530:13553 [0] include/socket.h:395 NCCL WARN Connect to 192.168.0.143<59811> failed : Connection timed out
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO bootstrap.cc:100 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO bootstrap.cc:326 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO init.cc:695 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO init.cc:951 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO misc/group.cc:69 -> 2 [Async thread]
Traceback (most recent call last):
  File "test_dist_1.py", line 25, in <module>
    dist.broadcast(tensor=a, src=0)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 806, in broadcast
    work = _default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: /tmp/pip-req-build-58y_cjjl/torch/lib/c10d/ProcessGroupNCCL.cpp:290, unhandled system error

而且节点 0 似乎卡在函数 dist.broadcast 中:

iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.0.143<0>
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/IB : No device found.
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.0.143<0>
iZuf6cu11ru7evq9ybagdjZ:13530:13553 [0] NCCL INFO Setting affinity for GPU 0 to ffff

谁能帮我解决这个问题？如何将张量从节点 0 发送到节点 1？如果有任何帮助，我将不胜感激!

Answer 1

unhandled system error意味着 NCCL 方面存在一些潜在错误。您应该首先使用 NCCL_DEBUG=INFO 重新运行您的代码(正如OP所做的那样)。然后从调试日志中找出错误是什么(尤其是日志中的警告)。

在 OP 的日志中，我认为行 iZbp11ufz31riqnssil53cZ:13530:13553 [0] include/socket.h:395 NCCL WARN Connect to 192.168.0.143<59811> failed : Connection timed out是unhandled system error的原因