python - 如何在 ProcessPoolExecutor 中使用 asyncio

Question

我正在网上搜索大量地址，我想在我的任务中同时使用 asyncio 和 ProcessPoolExecutor 来快速搜索地址。

    async def main():
        n_jobs = 3
        addresses = [list of addresses]
        _addresses = list_splitter(data=addresses, n=n_jobs)
        with ProcessPoolExecutor(max_workers=n_jobs) as executor:
             futures_list = []
             for _address in _addresses:
                futures_list +=[asyncio.get_event_loop().run_in_executor(executor, execute_parallel, _address)]

                for f in tqdm(as_completed(futures_list, loop=asyncio.get_event_loop()), total=len(_addresses)):
                results = await f

asyncio.get_event_loop().run_until_complete(main())

预期的:
我要 execute_parallel函数应该并行运行。

错误:

    Traceback (most recent call last):
  File "/home/awaish/danamica/scraping/skraafoto/aerial_photos_scraper.py", line 228, in <module>
    asyncio.run(main())
  File "/usr/local/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/home/awaish/danamica/scraping/skraafoto/aerial_photos_scraper.py", line 224, in main
    results = await f
  File "/usr/local/lib/python3.7/asyncio/tasks.py", line 533, in _wait_for_one
    return f.result()  # May raise f.exception().
TypeError: can't pickle coroutine objects

Answer 1

我不确定我是否在回答正确的问题，但您的代码的意图似乎是使用 Asyncio 在多个进程中运行您的 execute_parallel 函数。与使用 ProcessPoolExecutor 相反，为什么不尝试使用普通的多处理池并设置单独的 Asyncio 循环以在每个循环中运行。您可以为每个内核设置一个进程，并让 Asyncio 在每个进程中发挥其魔力。

async def run_loop(addresses):
    loop = asyncio.get_event_loop()
    loops = [loop.create_task(execute_parallel, address) for address in addresses]
    loop.run_until_complete(asyncio.wait(loops))

def main():
    n_jobs = 3
    addresses = [list of addresses]
    _addresses = list_splitter(data=addresses, n=n_jobs)
    with multiprocessing.Pool(processes=n_jobs) as pool:
        pool.imap_unordered(run_loop, _addresses)

我已经使用 Pool.imap_unordered 取得了巨大的成功，但根据您的需要，您可能更喜欢 Pool.map 或其他一些功能。您可以使用 chunksize 或每个列表中的地址数量来获得最佳结果(即，如果您遇到很多超时，您可能希望减少同时处理的地址数量)