c - If MPI Isend/Irecv 和 MPI wait 之间的语句阻止程序进行。是什么原因造成的？

Question

我正在创建一个程序来使用 MPI 计算两个导体之间的电势。我正在使用非阻塞发送和接收，因此可以在处理器之间发送信息时进行计算。

但是，isend 和 irecv 之间的 if 语句以及包含计算的 waits 命令未被输入。删除 if 语句和计算后，程序将继续执行 wait 语句。

我已检查计算是否正确并且没有引起问题。我已检查 if 语句的条件是否正确。

下面是一段测试代码:

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <time.h>
#include <mpi.h>

int main(int argc, char *argv[])
{
  /*MPI Specific Variables*/
  int my_size, my_rank, up, down;
  MPI_Request reqU, reqD, sreqU, sreqD;
  MPI_Status rUstatus, rDstatus, sUstatus, sDstatus;  

  /*Physical Dimensions*/
  double phi_0 = 1000.0;/*V*/

  /*Other Variables*/
  int grid_size = 100;
  int slice = 50;
  int x,y;
  double grid_res_y = 0.2;
  double grid_res_x = 0.1;
  int xboundary = 10;
  int yboundary = 25;
  int boundary_proc = 2;

  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
  MPI_Comm_size(MPI_COMM_WORLD, &my_size);

  /*Determining neighbours*/
  if (my_rank != 0) /*if statemets used to stop highest and lowest rank neighbours arent outside 0 - my_size-1 range of ranks*/
    {
      up = my_rank-1;
    }
  else
    {
      up = MPI_PROC_NULL;
    }

  if(my_rank != my_size-1)
    {
      down = my_rank+1;
    }
  else
   {
      down = MPI_PROC_NULL;
    }

  /*cross-check: presumed my_size is a factor of gridsize else there are odd sized slices and this is not coded for*/
  if (grid_size%my_size != 0)
    {
      printf("ERROR - number of procs =  %d, this is not a factor of grid_size %d\n", my_size, grid_size);
      exit(0);
    }

  /*Set Up Distributed Data Approach*/
  double phi[slice+2][grid_size]; /*extra 2 rows to allow for halo data*/

  for (y=0; y < slice+2; y++)
    {
      for (x=0; x < grid_size; x++)
        { 
          phi[y][x] = 0.0;
        }
    }

  if(my_rank == 0) /*Boundary Containing rank does 2 loops. One over part with inner conductor and one over part without inner conductor*/
    {
      for(y=0; y < slice+1; y++)
        {
          for(x=xboundary; x < grid_size; x++)
            {
              phi[y][x] = phi_0;
            }
        }   
    }


  if (my_rank < my_size-1)
    {
      /*send top most strip up one node to be recieved as bottom halo*/
      MPI_Isend(&phi[1][0], grid_size  , MPI_DOUBLE, down, 1, MPI_COMM_WORLD, &sreqU);  
      /*recv top halo from up one node*/
      MPI_Irecv(&phi[slice+1][0], grid_size, MPI_DOUBLE, down, 2, MPI_COMM_WORLD, &reqU);
    }

  if (my_rank > 0)
    {
      /*recv top halo from down one node*/
      MPI_Irecv(&phi[0][0], grid_size , MPI_DOUBLE, up, 2, MPI_COMM_WORLD, &reqD);
      /*send bottom most strip down one node to be recieved as top halo*/
      MPI_Isend(&phi[slice][0], grid_size , MPI_DOUBLE, up, 1, MPI_COMM_WORLD, &sreqD);
    }

  printf("send/recv complete");

  if (my_rank < boundary_proc)
     {
        printf("rank %d Entered if", my_rank);
        /*Calculations*/
     }

  else if(my_rank > boundary_proc)
    {
        printf("rank %d Entered else if", my_rank);
        /*calculations*/
    }

  else
     {
        printf("rank %d Entered else", my_rank);
        /*calculations*/
     }

  if (my_rank<my_size-1)
   {
     /*Wait for send to down one rank to complete*/
     MPI_Wait(&sreqD, &sDstatus);
     /*Wait for recieve from up one rank to complete*/
     MPI_Wait(&reqD, &rDstatus);
   }

  if (my_rank>0)
   {
     /*Wait for send to up down one rank to complete*/
     MPI_Wait(&sreqU, &sUstatus);
     /*Wait for recieve from down one rank to complete*/
     MPI_Wait(&reqU, &rUstatus);
   }

  printf("Wait complete");
  MPI_Finalize();
  return 0;
}

所有的打印语句都应该打印出各自的等级。目前它只能达到“发送/接收完成”我只在 2 个处理器 atm 上测试。

Answer 1

标签不匹配

标签必须匹配每对通信操作，即发送和接收必须具有相同的标签。在您的情况下，两个发送端有自己的标签，而接收端有不同的标签。更改它，使 down 发送和 up 接收具有相同的标签，反之亦然，例如

if (my_rank < my_size-1) {
    /*send top most strip up one node to be recieved as bottom halo*/
    MPI_Isend(&phi[1][0], grid_size  , MPI_DOUBLE, down, 1, MPI_COMM_WORLD, &sreqU);  
    /*recv top halo from up one node*/
    MPI_Irecv(&phi[slice+1][0], grid_size, MPI_DOUBLE, down, 2, MPI_COMM_WORLD, &reqU);
}

if (my_rank > 0) {
    /*recv top halo from down one node*/
    MPI_Irecv(&phi[0][0], grid_size , MPI_DOUBLE, up, 1, MPI_COMM_WORLD, &reqD);
    /*send bottom most strip down one node to be recieved as top halo*/
    MPI_Isend(&phi[slice][0], grid_size , MPI_DOUBLE, up, 2, MPI_COMM_WORLD, &sreqD);
}

不匹配的请求对象

在边界行列中，您正在等待错误的请求，这只是通过交换 MPI_Wait if 主体来解决。

等待多个非阻塞操作

与现在已删除的答案中的一些讨论相反，正确等待多个正在进行的非阻塞通信与多个等待¹。

不过，使用请求数组和 MPI_Waitall 绝对更好。它导致更清晰的代码，从一开始就可以防止混合请求的错误。它还为 MPI 实现提供了更大的优化自由度。这可能类似于以下内容:

MPI_Request requests[MAX_REQUESTS];
int num_requests = 0;

// ...

MPI_Isend(..., requests[num_requests++]);

// ...

MPI_Waitall(num_requests, requests, statuses);

或者，您可以利用 MPI_Waitall 允许请求数组的元素为 MPI_REQUEST_NULL 这一事实。这使您可以关联特定的请求，并且最终成为风格问题。

typedef enum {
    RECV_UP, RECV_DOWN, SEND_UP, SEND_DOWN, MAX_REQUESTS
} MyRequests;

MPI_Request requests[MAX_REQUESTS];
MPI_Status statuses[MAX_REQUESTS];

if (my_rank < my_size-1) {
    /*send top most strip up one node to be recieved as bottom halo*/
    MPI_Isend(&phi[1][0], grid_size  , MPI_DOUBLE, down, 1, MPI_COMM_WORLD, &requests[SEND_DOWN]);  
    /*recv top halo from up one node*/
    MPI_Irecv(&phi[slice+1][0], grid_size, MPI_DOUBLE, down, 2, MPI_COMM_WORLD, &requests[RECV_DOWN]);
} else {
    requests[RECV_DOWN] = requests[SEND_DOWN] = MPI_REQUEST_NULL;
}

if (my_rank > 0) {
    /*recv top halo from down one node*/
    MPI_Irecv(&phi[0][0], grid_size , MPI_DOUBLE, up, 1, MPI_COMM_WORLD, &requests[RECV_UP]);
    /*send bottom most strip down one node to be recieved as top halo*/
    MPI_Isend(&phi[slice][0], grid_size , MPI_DOUBLE, up, 2, MPI_COMM_WORLD, &requests[SEND_UP]);
} else {
    requests[RECV_UP] = requests[SEND_UP] = MPI_REQUEST_NULL;
}

// ...

MPI_Waitall(MAX_REQUESTS, requests, statuses);

---

^{1:这是 MPI 标准 (3.7.4) 中非阻塞进度保证的强制要求}

Progress A call to MPI_WAIT that completes a receive will eventually terminate and return if a matching send has been started, unless the send is satisfied by another receive. In particular, if the matching send is nonblocking, then the receive should complete even if no call is executed by the sender to complete the send. Similarly, a call to MPI_WAIT that completes a send will eventually return if a matching receive has been started, unless the receive is satisfied by another send, and even if no call is executed to complete the receive.