python - 产生多个进程来写入不同的文件 Python

Question

想法是使用N 进程编写N 文件。

要写入的文件的数据来自多个文件，这些文件存储在以列表作为值的字典中，如下所示:

dic = {'file1':['data11.txt', 'data12.txt', ..., 'data1M.txt'],
       'file2':['data21.txt', 'data22.txt', ..., 'data2M.txt'], 
        ...
       'fileN':['dataN1.txt', 'dataN2.txt', ..., 'dataNM.txt']}

所以 file1 是 data11 + data12 + ... + data1M 等...

所以我的代码是这样的:

jobs = []
for d in dic:
    outfile = str(d)+"_merged.txt"
    with open(outfile, 'w') as out:
        p = multiprocessing.Process(target = merger.merger, args=(dic[d], name, out))
        jobs.append(p)
        p.start()
        out.close()

merger.py 看起来像这样:

def merger(files, name, outfile):
    time.sleep(2)
    sys.stdout.write("Merging %n...\n" % name)

    # the reason for this step is that all the different files have a header
    # but I only need the header from the first file.
    with open(files[0], 'r') as infile:
        for line in infile:
            print "writing to outfile: ", name, line
            outfile.write(line) 
    for f in files[1:]:
        with open(f, 'r') as infile:
            next(infile) # skip first line
            for line in infile:
                outfile.write(line)
    sys.stdout.write("Done with: %s\n" % name)

我确实看到文件写在它应该去的文件夹上，但它是空的。没有标题，什么都没有。我已经把照片放在那里看看是否一切正确但没有任何效果。

帮助!

Answer 1

由于工作进程与创建它们的主进程并行运行，名为 out 的文件在工作人员可以写入它们之前被关闭。即使由于 with 语句而删除 out.close() 也会发生这种情况。而是将文件名传递给每个进程，让进程打开和关闭文件。