python - 如何在第一个完成使用 gevent 后产生不同的功能？

Question

基本思路如下:

请求到达 views1，它首先返回用户名。在 views1 完成后，do_something_else 会单独完成一些繁重的工作。您可以将此视为创建新用户，但必须对后台进行一些繁重的检查。

def views1(..):
   username = get_uername(...)
   return username

from lib import do_something_else
def do_something_else(...):
   // do heavy stuff here

gevent.joinall([
   gevent.spawn(views1, parmeter1, parmeter2, ...),
   gevent.spawn(do_something_else, parmeter1, parmeter2, ...)
])

问题是我不认为 do_something_else 根据我的日志记录被调用过。我阅读了教程，但不知道将 gevent.sleep(0) 放在哪里。我不想阻止。我希望用户立即看到用户名，并让 do_something_else 在后台运行。

有什么想法吗？

Answer 1

重要的是要了解您需要将“重负载”处理分离到线程池 [1] 中。

在 gevent 线程中发生的每个处理(每个 native 线程可以有一个 gevent HUB)必须只专注于处理网络请求和发送响应。

from gevent import spawn, run
from gevent.threadpool import ThreadPool
from time import sleep as heavy_load, time as now

class Globals:
    jobs = 4
    index = 0
    greenlets = []
    pool = ThreadPool(3) # change size of the pool appropriately

start = now()

def get_uername():
    heavy_load(0.1)
    Globals.index += 1
    return "Alex {0}".format(Globals.index)

def do_something_else(username):
    heavy_load(2.0)
    print "Heavy job done for", username, now() - start

def views1():
    "a request comes to views1 and it first returns the username"
    username = get_uername()
    ## There is some heavy job separate done by do_something_else right after views1 is done
    Globals.greenlets.append( 
        Globals.pool.spawn(do_something_else, username) 
        )
    # return username
    print "Returned requested username", username, now() - start


if __name__ == '__main__':
    ## simulate clients 
    for job_index in xrange(Globals.jobs):
        Globals.greenlets.append( spawn(views1) )

    ## wait for all tasks to complete
    # for greenlet in Globals.greenlets:
        # try:
            # greenlet.join()
        # except AttributeError, e:
            # greenlet.get()
    run()
    print "Test done", now() - start

这是测试的输出:

python threadpool_test.py
Returned requested username Alex 1 0.101000070572
Returned requested username Alex 2 0.201999902725
Returned requested username Alex 3 0.302999973297
Returned requested username Alex 4 0.40299987793
Heavy job done for Alex 1 2.10100007057
Heavy job done for Alex 2 2.2009999752
Heavy job done for Alex 3 2.3029999733
Heavy job done for Alex 4 4.10299992561
Test done 4.10500001907

请注意所有请求是如何首先完成的，并且 do_something_else 任务是如何以大小为 3 的批处理完成的。

当不使用 ThreadPool 时，每个请求都将花费 do_something_else 引入的额外时间，这不是 gevent 必须提供的异步编程。在这种情况下，输出将如下所示:

Heavy job done for Alex 1 2.10100007057
Returned requested username Alex 1 2.10100007057
Heavy job done for Alex 2 4.2009999752
Returned requested username Alex 2 4.20199990273
Heavy job done for Alex 3 6.30200004578
Returned requested username Alex 3 6.3029999733
Heavy job done for Alex 4 8.40299987793
Returned requested username Alex 4 8.40400004387
Test done 8.40400004387

注意第 4 个请求是如何在 8.4 秒内完成的，而不是异步处理时的 0.4 秒。

[1] http://code.google.com/p/gevent/source/browse/examples/threadpool.py