distributed - bigint 数组上的多语言环境计算

Question

我正在使用 Chapel，我正在尝试对 bigint 执行计算多语言环境设置中的数组。读取每行包含一个整数的文件。每行转换为 bigint记录然后插入到单个数组中。我有 4 个语言环境，因此我要求每个语言环境仅读取输入文件的 1/4 并仅处理该部分。

我已将问题减少到以下最小示例，该示例也受到影响:

module Hello {

use BigInteger;
use Math;
use Time;

config const inputPath = "/path/to/file";
config const inputSize = 10000000;
config const power = 2000;

proc dwriteln(args ...?n) {
        var curr = getCurrentTime(unit=TimeUnits.seconds);
        writeln("[ ", here.id, ": ", here.name, " ] [ ", curr, " ] ", (...args));
}

proc main() throws {
    writeln("Input path: ", inputPath);
    writeln("numLocales: ", numLocales);

    var elementsPerLocale = divceil(inputSize, numLocales);
    writeln("elementsPerLocale: ", elementsPerLocale);

    coforall loc in Locales {
        on loc {
            dwriteln("hello");
            var inputFile = open(inputPath, iomode.r, hints=IOHINT_CACHED);
            var reader = inputFile.reader();
            var startI = here.id * elementsPerLocale;
            var endI = startI+elementsPerLocale;
            dwriteln("startI = ", startI, " endI= ", endI);

            var a: [1..0] bigint;
            var i = 0;
            for line in reader.lines() {
                    // i in [startI;endI[
                    if i >= startI && i < endI {
                        a.push_back(new bigint(line, 16));
                    }
                    i +=1;
            }

            reader.close();
            inputFile.close();

            dwriteln("created array of size: ", a.size);

            forall elem in a {
                // perform some computation
                elem = elem ** power;
            }
            dwriteln("Computed.");
        }
    }


}


}

我希望语言环境可以并行执行操作，但事实并非如此。

但是，在运行代码时，似乎每个语言环境都按顺序进行处理。换句话说，语言环境 0 读取文件并进行处理，然后语言环境 1 读取文件并进行处理，依此类推。这可能是什么原因？

Answer 1

Chapel 授予并发和并行 I/O 操作:

I/O Overview

A file in Chapel identifies a file in the underlying operating system. Reads and writes to a file are done via one or more channels associated with the file. Each channel uses a buffer to provide sequential read or write access to its file, optionally starting at an offset.

Design Rationale

Since channels operate independently, concurrent I/O to the same open file is possible without contending for locks. Furthermore, since the channel (and not the file) stores the current file offset, it is straightforward to create programs that access the same open file in parallel. Note that such parallel access is not possible in C when multiple threads are using the same FILE* to write to different regions of a file because of the race condition between fseek and fwrite. Because of these issues, Chapel programmers wishing to perform I/O will need to know how to open files as well as create channels.

然而， IO模块详细信息可能有助于更好地安排资源:

据记载，

The default value of the iohints type is undefined.
...
proc open( out
error: syserr,
path: string = "",
mode: iomode,
hints: iohints = IOHINT_NONE,
style: iostyle = defaultIOStyle(),
url: string = ""
): file

使用非默认的、相当明确的设置重新运行实验，其中 I/O 提示可以更好地反射(reflect)预期的多语言环境处理 ~ IOHINT_PARALLEL .

由于 for line in reader.lines() {...} 的性质ad-hoc 迭代器，也可能有兴趣通过调整 IO 查看实验的吞吐量与 的意图IOHINT_CACHED 和 IOHINT_SEQUENTIAL 设置或 openreader() channel 具体仪器。

A value of the iohints type defines a set of hints about the I/O that the file or channel will perform. These hints may be used by the implementation to select optimized versions of the I/O operations.

The iohints type is implementation-defined. The following iohints constants are provided:

  *  IOHINT_NONE       defines an empty set, which provides no hints.
  *  IOHINT_RANDOM     suggests to expect random access.
  *  IOHINT_SEQUENTIAL suggests to expect sequential access.
  *  IOHINT_CACHED     suggests that the file data is or should be cached
                       in memory, possibly all at once.
  *  IOHINT_PARALLEL   suggests to expect many channels working with this file
                       in parallel.