python - Python中的多线程缩略图生成

Question

我想递归图像目录并为每个图像生成缩略图。我的机器上有 12 个可用内核。使用它们的好方法是什么？我没有太多编写多线程应用程序的经验，因此不胜感激任何简单的示例代码。提前致谢。

Answer 1

摘要

使用进程，而不是线程，因为 GIL 导致 Python 在处理 CPU 密集型线程时效率低下.多处理的两种可能的解决方案是:

multiprocessing模块

如果您使用内部缩略图制作器(例如 PIL)，这是首选。简单写个缩略图制作函数，并行启动12个。当其中一个进程完成时，在其槽中运行另一个进程。

改编自 Python 文档，这是一个应使用 12 个内核的脚本:

from multiprocessing import Process
import os

def info(title):  # For learning purpose, remove when you got the PID\PPID idea
    print title
    print 'module:', __name__
    print 'parent process:', os.getppid(), 
    print 'process id:', os.getpid()
 
def f(name):      # Working function
    info('function f')
    print 'hello', name

if __name__ == '__main__':
    info('main line')
    processes=[Process(target=f, args=('bob-%d' % i,)) for i  in range(12)]
    [p.start() for p in processes]
    [p.join()  for p in processes]

附录:使用multiprocess.pool()

根据soulman的评论，您可以使用提供的流程拉取。

我改编了 multiprocessing manual 中的一些代码.请注意，您可能应该使用 multiprocessing.cpu_count()而不是 4 来自动确定 CPU 的数量。

from multiprocessing import Pool
import datetime

def f(x):  # You thumbnail maker function, probably using some module like PIL
    print '%-4d: Started at %s' % (x, datetime.datetime.now())
    return x*x

if __name__ == '__main__':
    pool = Pool(processes=4)              # start 4 worker processes
    print pool.map(f, range(25))          # prints "[0, 1, 4,..., 81]"

给出(注意打印输出没有严格排序!):

0   : Started at 2011-04-28 17:25:58.992560
1   : Started at 2011-04-28 17:25:58.992749
4   : Started at 2011-04-28 17:25:58.992829
5   : Started at 2011-04-28 17:25:58.992848
2   : Started at 2011-04-28 17:25:58.992741
3   : Started at 2011-04-28 17:25:58.992877
6   : Started at 2011-04-28 17:25:58.992884
7   : Started at 2011-04-28 17:25:58.992902
10  : Started at 2011-04-28 17:25:58.992998
11  : Started at 2011-04-28 17:25:58.993019
12  : Started at 2011-04-28 17:25:58.993056
13  : Started at 2011-04-28 17:25:58.993074
14  : Started at 2011-04-28 17:25:58.993109
15  : Started at 2011-04-28 17:25:58.993127
8   : Started at 2011-04-28 17:25:58.993025
9   : Started at 2011-04-28 17:25:58.993158
16  : Started at 2011-04-28 17:25:58.993161
17  : Started at 2011-04-28 17:25:58.993179
18  : Started at 2011-04-28 17:25:58.993230
20  : Started at 2011-04-28 17:25:58.993233
19  : Started at 2011-04-28 17:25:58.993249
21  : Started at 2011-04-28 17:25:58.993252
22  : Started at 2011-04-28 17:25:58.993288
24  : Started at 2011-04-28 17:25:58.993297
23  : Started at 2011-04-28 17:25:58.993307
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 
 289, 324, 361, 400, 441, 484, 529, 576]

subprocess模块

subprocess模块对于运行外部进程很有用，因此如果您计划使用像 imagemagick 的 convert 这样的外部缩略图生成器，那么它是首选。代码示例:

import subprocess as sp

processes=[sp.Popen('your-command-here', shell=True, 
                    stdout=sp.PIPE, stderr=sp.PIPE) for i in range(12)]

现在，迭代流程。如果有任何进程已完成(使用 subprocess.poll() )，请将其删除并将新进程添加到您的列表中。