python - "filedescriptor out of range in select()"将 python 的子进程与 rsync 一起使用时

Question

下面的代码用于将上传的图片同步到另一个地方。可以用，但是过了一段时间(大约10天)，服务不可用，报错:'filedescriptor out of range in select()'，重启服务解决问题。

# sync.py

def sync_file(source_pic, hashval, retry_num=3):

    pic_path = os.path.join(gen_dir(hashval), os.path.split(source_pic)[1])
    filename = tempfile.mkstemp()[1]
    with open(filename, 'w') as f:
        f.write(pic_path)

    for sync_path in options.sync_paths:
        try_num = 0
        rsync_cmd = ['rsync','-au', '--files-from='+filename, options.pic_path, sync_path]

        while try_num < retry_num:
            proc = subprocess.Popen(rsync_cmd,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            stdout_value, stderr_value = proc.communicate()

            if len(stderr_value) != 0:
                logger.error('sync failed: %s' % stderr_value)
                try_num = try_num + 1
                #raise UploadException('sync failed')
            else:
                break

    os.remove(filename)

日志信息:

File "/path/to/sync.py", line 25, in sync_file
    stdout_value, stderr_value = proc.communicate()
File "/usr/lib/python2.6/subprocess.py", line 691, in communicate
    return self._communicate(input)
File "/usr/lib/python2.6/subprocess.py", line 1211, in _communicate
    rlist, wlist, xlist = select.select(read_set, write_set, [])
    ValueError: filedescriptor out of range in select()

是否有未关闭的文件描述符导致错误？似乎子进程没有关闭文件描述符，所以当它运行 1024 次时，文件描述符超出范围。 (我们使用的是 python 2.6，子进程被迫使用 select.select()，它有 1024 个文件描述符的限制，即使 epoll 可用)

Answer 1

https://bugzilla.redhat.com/show_bug.cgi?id=609020

Prior to Python 2.7, programs that used ulimit -n to enable communication with large numbers of subprocesses could still monitor only 1024 file descriptors at a time, which caused an exception:

ValueError: filedescriptor out of range in select()

This was due to the subprocess module using the select system call. The module now uses the poll system call, removing this limitation.

可能的修复:使用 python 2.7+，或使用 poll 向后移植代码。