gpt4 book ai didi

python - Pytorch "NCCL error": unhandled system error, NCCL 版本 2.4.8"

转载 作者:行者123 更新时间:2023-12-04 11:11:49 25 4
gpt4 key购买 nike

我使用pytorch分布式训练我的模型。我有两个节点和每个节点两个gpu,我为一个节点运行代码:

python train_net.py  --config-file configs/InstanceSegmentation/pointrend_rcnn_R_50_FPN_1x_coco.yaml  --num-gpu 2  --num-machines 2 --machine-rank 0 --dist-url tcp://192.168.**.***:8000
和另一个:
python train_net.py  --config-file configs/InstanceSegmentation/pointrend_rcnn_R_50_FPN_1x_coco.yaml  --num-gpu 2  --num-machines 2 --machine-rank 1 --dist-url tcp://192.168.**.***:8000
但是另一个有 RuntimeError 问题
global_rank 3 machine_rank 1 num_gpus_per_machine 2 local_rank 1
global_rank 2 machine_rank 1 num_gpus_per_machine 2 local_rank 0
Traceback (most recent call last):
File "train_net.py", line 109, in <module>
args=(args,),
File "/root/detectron2_repo/detectron2/engine/launch.py", line 49, in launch
daemon=False,
File "/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
while not spawn_context.join():
File "/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/root/detectron2_repo/detectron2/engine/launch.py", line 72, in _distributed_worker
comm.synchronize()
File "/root/detectron2_repo/detectron2/utils/comm.py", line 79, in synchronize
dist.barrier()
File "/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 1489, in barrier
work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:410, unhandled system error, NCCL version 2.4.8
如果我把mask-rank=1改成mask-rank=0,那么就不会报错,但是不能分布式训练,有谁知道为什么会出现这个错误?

最佳答案

很多原因都可能导致此问题,例如参见 1 , 2 .添加行

import os
os.environ["NCCL_DEBUG"] = "INFO"
到您的脚本将记录导致错误的更具体的调试信息,为您提供更有用的错误消息给谷歌。

关于python - Pytorch "NCCL error": unhandled system error, NCCL 版本 2.4.8",我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/61075390/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com