gpt4 book ai didi

python - 多处理过程完成后如何退出python脚本?

转载 作者:行者123 更新时间:2023-11-28 18:38:56 25 4
gpt4 key购买 nike

更新:在dano的帮助下,我解决了这个问题。

我没有使用 join() 调用生产者,这让我的脚本挂起。只需要像 dano 说的那样添加一行:

...
producer = multiprocessing.Process(target=produce,args=(file_queue,row_queue))
producer.daemon = True
producer.start()
...

旧脚本:

import multiprocessing
import Queue

QUEUE_SIZE = 2000


def produce(file_queue, row_queue,):

while not file_queue.empty():
src_file = file_queue.get()
zip_reader = gzip.open(src_file, 'rb')

try:
csv_reader = csv.reader(zip_reader, delimiter=SDP_DELIMITER)

for row in csv_reader:
new_row = process_sdp_row(row)
if new_row:
row_queue.put(new_row)
finally:
zip_reader.close()


def consume(row_queue):
'''processes all rows, once queue is empty, break the infinit loop'''
while True:
try:
# takes a row from queue and process it
pass
except multiprocessing.TimeoutError as toe:
print "timeout, all rows have been processed, quit."
break
except Queue.Empty:
print "all rows have been processed, quit."
break
except Exception as e:
print "critical error"
print e
break


def main(args):

file_queue = multiprocessing.Queue()
row_queue = multiprocessing.Queue(QUEUE_SIZE)

file_queue.put(file1)
file_queue.put(file2)
file_queue.put(file3)

# starts 3 producers
for i in xrange(4):
producer = multiprocessing.Process(target=produce,args=(file_queue,row_queue))
producer.start()

# starts 1 consumer
consumer = multiprocessing.Process(target=consume,args=(row_queue,))
consumer.start()

# blocks main thread until consumer process finished
consumer.join()

# prints statistics results after consumer is done

sys.exit(0)


if __name__ == "__main__":
main(sys.argv[1:])

目的:

我正在使用 python 2.7 multiprocessing 生成 3 个生产者同时读取 3 个文件,然后将文件行放入 row_queue 并生成 1 个消费者对所有行进行更多处理。消费者完成后在主线程打印统计结果,所以我使用join()方法。最后调用 sys.exit(0) 退出脚本。

问题:无法退出脚本。

我尝试将 sys.exit(0) 替换为 print "the end","the end"显示在控制台上。难道我做错了什么?为什么脚本不退出,如何让它退出?谢谢

最佳答案

您的生产者没有设置multiprocessing.Process.daemon属性:

daemon

The process’s daemon flag, a Boolean value. This must be set before start() is called.

The initial value is inherited from the creating process.

When a process exits, it attempts to terminate all of its daemonic child processes.

Note that a daemonic process is not allowed to create child processes. Otherwise a daemonic process would leave its children orphaned if it gets terminated when its parent process exits. Additionally, these are not Unix daemons or services, they are normal processes that will be terminated (and not joined) if non-daemonic processes have exited.

https://docs.python.org/2/library/multiprocessing.html#multiprocessing.Process.daemon

只需添加producer.daemon = True:

...
producer = multiprocessing.Process(target=produce,args=(file_queue,row_queue))
producer.daemon = True
producer.start()
...

consumer 加入时,整个程序应该可以结束。

顺便说一句,您或许也应该加入制作人。

关于python - 多处理过程完成后如何退出python脚本?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/29417877/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com