python - Numpy savez 异常

Question

我最近发现了这样的说法:

np.savez is not multi-process safe, because it always uses gettempdir() + key + ".npy". So, if you're running the same script over different data sets, you will roach your own data when one process removes the /tmp/arr_0.npy of another process.

As luck would have it, I'm in that exact situation, so I have an example error:

Traceback (most recent call last):    File
"/home/pedro/prospectus/prelim/bbvs/bin/skldthresh", line 99, in <module> main()    
File "/home/pedro/prospectus/prelim/bbvs/bin/skldthresh", line 90, in  main
 np.savez("%smask-t%0.2f" % (outfile,threshold), result)    
File "/usr/lib/python2.5/site-packages/numpy/lib/io.py", line 230, in savez
 zip.write(filename, arcname=fname)    
 File "/usr/lib/python2.5/zipfile.py", line 541, in write
  st = os.stat(filename)  OSError: [Errno 2] No such file or directory: '/tmp/arr_0.npy'

Fortunately, replacing tempfile.gettempdir() with tempfile.mkdtemp() AND remembering to clean up at the end with os.rmdir(direc) is all it takes to fix it as far as I can tell.

这到底意味着什么？我个人该如何避免这种情况？ numpy 是否将写入 .npz 文件的数据存储在临时文件中，这些临时文件的名称可能与其他脚本中的其他临时 .npz 文件类似？!

我有一些科学实验，这些实验运行了几个小时，并通过 np.savez 保存结果。 savez 保存结果的目标路径不同，因此每个实验都有自己的结果路径。但是，它们的脚本本身位于同一目录中。

有趣的是，脚本有时会在发生此错误之前运行几个小时:

> Traceback (most recent call last):   File
> "/work/var/slurmd/state.node348.d/job20832/slurm_script", line 53, in
> <module>
>     E = Exp6_06()   File
> "/work/experiments/s06/cs06_ex06.py", line
> 150, in __init__
>     self.__start()   File "/work/experiments/s06/cs06_ex06.py", line
> 374, in __start
>     File "/home/fx092/.local/lib/python2.7/site-packages/numpy/lib/npyio.py",
> line 600, in savez_compressed
>     _savez(file, args, kwds, True)   File "/home/fx092/.local/lib/python2.7/site-packages/numpy/lib/npyio.py",
> line 630, in _savez
>     fd, tmpfile = tempfile.mkstemp(suffix='-numpy.npy')   File "/sw/env/openmpi/2Q4/lib/python2.7/tempfile.py",
> line 314, in mkstemp
>     return _mkstemp_inner(dir, prefix, suffix, flags)   File "/sw/env/openmpi/2Q4/lib/python2.7/tempfile.py",
> line 244, in _mkstemp_inner
>     fd = _os.open(file, flags, 0600) OSError: [Errno 2] No such file or directory:
> '/work/tmp/node001.admin.2016-03-06-152506.fx092.27432/tmp5UulEz-numpy.npy'

使用的numpy版本:1.10.4

Answer 1

您的错误发生在mkstemp中。引用建议使用 mkdtemp 代替:

https://docs.python.org/2/library/tempfile.html

tempfile.mkstemp([suffix=''[, prefix='tmp'[, dir=None[, text=False]]]])

Creates a temporary file in the most secure manner possible. There are no race conditions in the file’s creation, assuming that the platform properly implements the os.O_EXCL flag for os.open(). The file is readable and writable only by the creating user ID. If the platform uses permission bits to indicate whether a file is executable, the file is executable by no one. The file descriptor is not inherited by child processes.

和

tempfile.mkdtemp([suffix=''[, prefix='tmp'[, dir=None]]])

Creates a temporary directory in the most secure manner possible. There are no race conditions in the directory’s creation. The directory is readable, writable, and searchable only by the creating user ID.

The user of mkdtemp() is responsible for deleting the temporary directory and its contents when done with it.

引用的错误似乎是在创建临时文件并将它们收集到 zip 后发生的。您的错误似乎是在创建临时文件之一时发生的。

我对这些函数的使用还不够多，无法理解其中的区别。 我们可能需要研究lib/npyio.py中的_savez。

您使用什么操作系统？ Linux、Mac、Windows？

<小时/>

这是 _savez 的核心，现在位于 lib/npyio.py 文件中:

def _savez(file, args, kwds, compress):
    ....
    zip = zipfile_factory(file, mode="w", compression=compression)

    # Stage arrays in a temporary file on disk, before writing to zip.
=>  fd, tmpfile = tempfile.mkstemp(suffix='-numpy.npy')
    os.close(fd)
    try:
        for key, val in namedict.items():
            fname = key + '.npy'
            fid = open(tmpfile, 'wb')
            try:
                format.write_array(fid, np.asanyarray(val))
                fid.close()
                fid = None
=>              zip.write(tmpfile, arcname=fname)
            finally:
                if fid:
                    fid.close()
    finally:
        os.remove(tmpfile)
    zip.close()

当临时“保存”文件添加到存档中时，引用的错误发生在循环结束时。当它获取临时目录和临时文件(fd)时，您的错误很早就发生了。请注意，丢弃打开的文件句柄，仅使用名称(对每个数组重复)。

令人惊讶的是，临时目录在 mkstemp 函数调用期间消失了。它几乎感觉像是一个 openmpi 问题，而不是一个 savez 问题。