multithreading - ForkManager在 Debug模式下更快完成

Question

我试图弄清楚如何为我正在研究的项目的一部分正确实现ForkManager，但是遇到了FM似乎正在产生过程和在做事情但要花很长时间的情况。

但是，当我在调试代码中尝试FM(通过将最大进程数设置为0)时，代码会在合理的预期时间内完成。

这是我遇到麻烦的代码...

use strict;
use warnings;
use Parallel::ForkManager;

sub read_table {
  # takes a filename and reads in a CSV file.
  # works fine and thus is omitted here
}
sub foo {
  # originally an Inline::C subroutine
  # for purpose of debugging replaced with randgen
  return rand;
}

my $cpu_count = 0; my $epsilon = 1e-16;
my @tt = read_table('tt.csv');
my @tc = read_table('tc.csv');
my @nm = ($epsilon) x scalar @tc;
my @results;
my $pm = new Parallel::ForkManager($cpu_count);
$pm->run_on_finish(sub{
  my $i = $_[1]; my $m = $_[5]; 
  $results[$i] = $m;
});
foreach my $i (0..$#tt) {
  $pm->start and next;
  my @r;
  if (scalar @{$tt[$i]} > 1) {
    foreach my $j (0..$#tc) {
        if (scalar @{$tc[$j]} > 1) {
            push @r, foo(scalar @{$tt[$i]}, scalar @{$tc[$j]}, \@{$tt[$i]}, \@{$tc[$j]});
          } else {
              push @r, $epsilon; 
          }
      }
  } else {
      @r = @nm;
  }
  $pm->finish($i, [@r]);
  undef @r;
}
$pm->wait_all_children;

因此，如果我将 $cpu_count设置为0，则该过程可以顺利完成，没有问题，几分钟之内就完成了原始C代码(仅 sub foo {return rand;}大约2秒钟)，但是当FM开启时，它似乎会继续运行许久。但是当我输入诸如 print "at rows $i and $j"之类的打印语句来诊断问题时，它似乎似乎正在运行。

如果我取出所有与FM相关的代码，而只是尝试使用常规的double foreach循环，则运行时间是相同的。

提前致谢!

Answer 1

这是因为从子级发送到父级的数据被写入磁盘(请参阅Parallel::ForkManager中的RETROEVING DATASTRUCTURES):

The data structure referenced in a given child process is serialized and written out to a file by Storable. The file is subsequently read back into memory and a new data structure belonging to the parent process is created. Please consider the performance penalty it can imply, so try to keep the returned structure small.

在 Debug模式下，不会发生派生，因此可以直接传递结构而无需保存和加载。

Thread::Queue可能会产生更好的结果。

#!/usr/bin/perl
use strict;
use warnings;

use threads;
use Thread::Queue;

sub read_table {
    map [ map rand, 1 .. 100 ], 1 .. 100;
}
sub foo {
    [ @_ ]
}

my $cpu_count = 20; my $epsilon = 1e-16;
my @tt = read_table('tt.csv');
my @tc = read_table('tc.csv');
my @nm = ($epsilon) x scalar @tc;
my @results;

my ($q_in, $q_out) = map 'Thread::Queue'->new, 1, 2;
my @workers = map threads->create(sub{
    while(defined(my $i = $q_in->dequeue)) {
        warn $i;
        my @r;
        if (scalar @{$tt[$i]} > 1) {
            for my $j (0 .. $#tc) {
                if (scalar @{$tc[$j]} > 1) {
                    push @r, foo(scalar @{$tt[$i]}, scalar @{$tc[$j]}, \@{$tt[$i]}, \@{$tc[$j]});
                } else {
                    push @r, $epsilon; 
                }
            }
        } else {
            @r = @nm;
        }
        $q_out->enqueue([$i, @r]);
    }
}), 1 .. $cpu_count;

$q_in->enqueue(0 .. $#tt);
$q_in->end;


for (0 .. $#tt) {
    my $r = $q_out->dequeue;
    my $i = shift @$r;
    warn "$i: $r->[2][2][1]";
}
$_->join for @workers;