parallel-processing - pymongo - 执行并行查询

Question

这是我想并行化但不知道从哪里开始的伪代码

from pymongo import MongoClient


client = MongoClient('localhost', 27017)
db = client['myDB']
collection = db.myCollection

test_list = ['foo', 'bar']
result_list = list()

for el in test_list:
     result_list.append(collection.distinct('attrib',{'version': el}))

我知道如何使用 joblib 创建并行循环，但我不确定如何并行查询 MongoDB，我应该创建多个客户端还是集合？如果我在不关心 MongoDB 的情况下简单地用 joblib 重写上面的代码，它会工作吗？

Answer 1

您可以在单独的线程中运行请求:

from multiprocessing.dummy import Pool as ThreadPool 

from pymongo import MongoClient


client = MongoClient('localhost', 27017)
db = client['myDB']
collection = db.myCollection

thread_pool_size = 4
pool = ThreadPool(thread_pool_size) 

def my_function(el):
    return collection.distinct('attrib', {'version': el}))

test_list = ['foo', 'bar']
result_list = pool.map(my_function, test_list)