作者热门文章
- mongodb - 在 MongoDB mapreduce 中,如何展平值对象?
- javascript - 对象传播与 Object.assign
- html - 输入类型 ="submit"Vs 按钮标签它们可以互换吗?
- sql - 使用 MongoDB 而不是 MS SQL Server 的优缺点
我希望将 multiprocessing.Queue
转储到列表中。对于该任务,我编写了以下函数:
import Queue
def dump_queue(queue):
"""
Empties all pending items in a queue and returns them in a list.
"""
result = []
# START DEBUG CODE
initial_size = queue.qsize()
print("Queue has %s items initially." % initial_size)
# END DEBUG CODE
while True:
try:
thing = queue.get(block=False)
result.append(thing)
except Queue.Empty:
# START DEBUG CODE
current_size = queue.qsize()
total_size = current_size + len(result)
print("Dumping complete:")
if current_size == initial_size:
print("No items were added to the queue.")
else:
print("%s items were added to the queue." % \
(total_size - initial_size))
print("Extracted %s items from the queue, queue has %s items \
left" % (len(result), current_size))
# END DEBUG CODE
return result
但由于某种原因它不起作用。
观察以下 shell session :
>>> import multiprocessing
>>> q = multiprocessing.Queue()
>>> for i in range(100):
... q.put([range(200) for j in range(100)])
...
>>> q.qsize()
100
>>> l=dump_queue(q)
Queue has 100 items initially.
Dumping complete:
0 items were added to the queue.
Extracted 1 items from the queue, queue has 99 items left
>>> l=dump_queue(q)
Queue has 99 items initially.
Dumping complete:
0 items were added to the queue.
Extracted 3 items from the queue, queue has 96 items left
>>> l=dump_queue(q)
Queue has 96 items initially.
Dumping complete:
0 items were added to the queue.
Extracted 1 items from the queue, queue has 95 items left
>>>
这里发生了什么?为什么不是所有的元素都被倾倒了?
最佳答案
试试这个:
import Queue
import time
def dump_queue(queue):
"""
Empties all pending items in a queue and returns them in a list.
"""
result = []
for i in iter(queue.get, 'STOP'):
result.append(i)
time.sleep(.1)
return result
import multiprocessing
q = multiprocessing.Queue()
for i in range(100):
q.put([range(200) for j in range(100)])
q.put('STOP')
l=dump_queue(q)
print len(l)
多处理队列有一个内部缓冲区,该缓冲区有一个馈线线程,该线程从缓冲区中提取工作并将其刷新到管道中。如果不是所有的对象都被刷新,我可以看到 Empty 过早引发的情况。使用哨兵来指示队列的结束是安全的(可靠的)。此外,使用 iter(get, sentinel) 习惯用法比依赖 Empty 更好。
我不喜欢它可能由于刷新时间而升空(我添加了 time.sleep(.1) 以允许上下文切换到馈线线程,您可能不需要它,没有它它也可以工作 - 它是释放 GIL 的习惯)。
关于python - 将 multiprocessing.Queue 转储到列表中,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/1540822/
我正在为我的应用程序使用 Tank-Auth。我唯一的问题是激活和重置帐户密码。 用于登录、注册、注销;我对这些代码没有问题; $route['login'] = "/auth/login"; $ro
我是一名优秀的程序员,十分优秀!