python - 如何在 multiprocessing.Process 中传递未 pickle 的对象作为参数？

Question

我正在解封一个对象并作为 process 的参数之一传递。没有输出。

我想知道这种取消pickle并将对象作为参数传递的方法是否会导致多处理错误。有什么办法可以解决这个问题？

from multiprocessing import Process, Queue
def func(arg1, arg2,q):
    df_temp = arg1[arg1['col'].isin(arg2)]
    q.put(df_temp)

if __name__ == '__main__':
    import pickle
    import pandas as pd
    arg1= pickle.load(open('paths.p','rb'))
    arg2= pd.Series(pd.date_range(end = 'some_Date', periods=12,freq = 'MS')).dt.to_pydatetime()
    arg2=[i.date() for i in arg2]
    q = Queue()
    p =Process(target=func, args=(arg1,arg2,q))
    p.start()
    p.join()    
    while not q.empty():
        w=q.get() 

Answer 1

您由于其他原因陷入僵局。

By default if a process is not the creator of the queue then on exit it will attempt to join the queue’s background thread. The process can call cancel_join_thread() to make join_thread() do nothing. docs

您的Process不会退出，因为它正在加入来自 multiprocessing.Queue 的供给线程，一旦您 queue.put() 就会开始第一次。您需要queue.get()在您加入该流程之前在您的 parent 中。

Warning: As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe. This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children. docs

也不要使用while not q.empty() ，这是一种反模式，一旦有多个消费者就会导致死锁。使用哨兵值来通知消费者没有更多商品即将到来。有关更多信息，here .