python - 异步 : How to handle multiple open files OS error

Question

我正在尝试运行约 500 个异步子进程。我在下面的主函数中将文件作为列表 p_coros 传递。

async def run_check(shell_command):
    p = await asyncio.create_subprocess_shell(shell_command,
                    stdin=PIPE, stdout=PIPE, stderr=STDOUT)
    fut = p.communicate()
    try:
        pcap_run = await asyncio.wait_for(fut, timeout=5)
    except asyncio.TimeoutError:
        p.kill()
        await p.communicate()

def get_coros():
    for pcap_loc in print_dir_cointent():
        for pcap_check in get_pcap_executables():
            tmp_coro = (run_check('{args}'
            .format(e=sys.executable, args=args)))
            if tmp_coro != False:
                coros.append(tmp_coro)
     return coros

async def main(self):
    ## Here p_coros has over 500 files
    p_coros = get_coros()
    for f in asyncio.as_completed(p_coros):
        res = await f




loop = asyncio.get_event_loop()
loop.run_until_complete(get_coros())
loop.close()

我认为这里的问题是 asyncio.as_completed，因为它试图并行打开所有文件，因为如果我删除 asyncio.as_completed 它可以正常工作但需要很多时间的时间。我想在不浪费太多时间的情况下处理打开文件问题 OSError(24, 'Too many open files')。

日志:

Exception ignored when trying to write to the signal wakeup fd:
BlockingIOError: [Errno 11] Resource temporarily unavailable

ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<ClassificationCheck.run_check() running at ./regression.py:74> wait_for=<Future finished exception=RuntimeError('Event loop is closed',)> cb=[as_completed.<locals>._on_completion() at /usr/lib/python3.5/asyncio/tasks.py:478]>

回溯:

Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "./regression.py", line 74, in run_check
    stdin=PIPE, stdout=PIPE, stderr=STDOUT)
  File "/usr/lib/python3.5/asyncio/subprocess.py", line 197, in create_subprocess_shell
    stderr=stderr, **kwds)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 1049, in subprocess_shell
    protocol, cmd, True, stdin, stdout, stderr, bufsize, **kwargs)
  File "/usr/lib/python3.5/asyncio/unix_events.py", line 184, in _make_subprocess_transport
    **kwargs)
  File "/usr/lib/python3.5/asyncio/base_subprocess.py", line 40, in __init__
    stderr=stderr, bufsize=bufsize, **kwargs)
  File "/usr/lib/python3.5/asyncio/unix_events.py", line 640, in _start
    stdin, stdin_w = self._loop._socketpair()
  File "/usr/lib/python3.5/asyncio/unix_events.py", line 53, in _socketpair
    return socket.socketpair()
  File "/usr/lib/python3.5/socket.py", line 478, in socketpair
    a, b = _socket.socketpair(family, type, proto)
OSError: [Errno 24] Too many open files
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<ClassificationCheck.run_check() done, defined at ./regression.py:72> exception=OSError(24, 'Too many open files')>

Answer 1

当我将大量文件传递给异步工作时，它抛出了操作系统错误。我处理它的方式是创建一个列表列表，每个子列表包含固定数量的 PCAP，不会导致 OS Error，然后一次传递一个列表。

因此我了解到在继续处理更多文件之前关闭已打开的文件很重要。

def get_coros(pcap_list):
    for pcap_loc in pcap_list:
        for pcap_check in get_pcap_executables():
            tmp_coro = (run_check('{args}'
            .format(e=sys.executable, args=args)))
            if tmp_coro != False:
                coros.append(tmp_coro)
     return coros

async def main():
    pcap_list_gen = print_dir_cointent() # Passing a list of lists
    for pcap_list in pcap_list_gen:
        p_coros = get_coros(pcap_list)
        for f in asyncio.as_completed(p_coros):
            res = await f