mpi - Slurm 的 srun 比 mpirun 慢

Question

如果在下面将“mpirun”替换为“srun”，我的 mpi+openmp 作业运行速度会慢 3 倍!

#!bin/bash
#
#SBATCH --job-name=locstack4d_mpi
#SBATCH --output=./res_locstack4d_mpi.txt
#
#SBATCH --ntasks=3
#SBATCH --cpus-per-task=16
#SBATCH --ntasks-per-node=1
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=100
#SBATCH --verbose
export OMP_NUM_THREADS=28

mpirun  ./locstack4d  ./locstack4d_input.dat

mpi 实现细节:

Version:                                 3.1
Release Date:                            Thu Feb 20 11:41:13 CST 2014
CC:                              gcc  -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC -Wl,-z,noexecstack 
CXX:                             g++  -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC -Wl,-z,noexecstack 
F77:                             gfortran -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC -Wl,-z,noexecstack 
F90:                             gfortran -m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC -Wl,-z,noexecstack 
Configure options:                       '--disable-option-checking' '--prefix=/usr' '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-sharedlibs=gcc' '--enable-shared' '--enable-lib-depend' '--disable-rpath' '--enable-fc' '--with-device=ch3:nemesis' '--with-pm=hydra:gforker' '--includedir=/usr/include/mpich-x86_64' '--bindir=/usr/lib64/mpich/bin' '--libdir=/usr/lib64/mpich/lib' '--datadir=/usr/share/mpich' '--mandir=/usr/share/man/mpich' '--docdir=/usr/share/mpich/doc' '--htmldir=/usr/share/mpich/doc' '--with-hwloc-prefix=system' 'FC=gfortran' 'F77=gfortran' 'CFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC -O2' 'CXXFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC ' 'FCFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC ' 'FFLAGS=-m64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC -O2' 'LDFLAGS=-Wl,-z,noexecstack ' 'MPICH2LIB_CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'MPICH2LIB_CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'MPICH2LIB_FCFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'MPICH2LIB_FFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'LIBS=-lrt -lpthread ' 'CPPFLAGS= -I/builddir/build/BUILD/mpich-3.1-rh/src/mpl/include -I/builddir/build/BUILD/mpich-3.1-rh/src/mpl/include -I/builddir/build/BUILD/mpich-3.1-rh/src/openpa/src -I/builddir/build/BUILD/mpich-3.1-rh/src/openpa/src -I/builddir/build/BUILD/mpich-3.1-rh/src/mpi/romio/include'
Process Manager:                         pmi
Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
Topology libraries available:            hwloc
Resource management kernels available:   user slurm ll lsf sge pbs cobalt
Checkpointing libraries available:       
Demux engines available:                 poll select

Answer 1

您的 MPI 实现可能没有正确使用 Slurm 的 PMI，导致生成三个独立的 1-cpu 进程，而不是预期的 3-cpu 作业。您需要确保 MPI 版本和 Slurm 版本兼容(例如 OpenMPI >=1.5 和 Slurm >=2.6)并且 OpenMPI 是在支持 Slurm 的情况下编译的(参见 http://slurm.schedmd.com/mpi_guide.html)