python - python 每秒发送超过 1000 个请求

Question

我试图同时向服务器发送请求，然后使用以下代码记录平均延迟:

import Queue
import time
import threading
import urllib2

data = "{"image_1":"abc/xyz.jpg"}"
headers = {.....}
def get_url(q, url):
    num = 1
    sum = 0
    while num <= 200:
        start = time.time()
        req = urllib2.Request(url, data, headers)
        response = urllib2.urlopen(req)
        end = time.time()
        print end - start
        num = num + 1
        q.put(response.read())
        sum = sum + (end - start)
    print sum


theurls = ["http://example.com/example"]
q = Queue.Queue()

for u in theurls:
    t = threading.Thread(target = get_url, args = (q, u))
    t.daemon = True
    t.start()

while True:
    s = q.get()
    print s

这段代码工作得很好，但现在我打算每秒发送超过 1000 个请求。我遇到了this answer但我不确定如何在我的情况下使用 grequests 。一些见解将会非常有帮助。

谢谢

Answer 1

文档不是很好，但来源很好。阅读源码! Check out the first few lines of grequests.py on github :

"""
grequests
~~~~~~~~~
This module contains an asynchronous replica of ``requests.api``, powered
by gevent. All API methods return a ``Request`` instance (as opposed to
``Response``). A list of requests can be sent with ``map()``.
"""

The package exports the following :

__all__ = (
    'map', 'imap',
    'get', 'options', 'head', 'post', 'put', 'patch', 'delete', 'request'
)

Those symbols are defined further down the file :

# Shortcuts for creating AsyncRequest with appropriate HTTP method
get = partial(AsyncRequest, 'GET')
options = partial(AsyncRequest, 'OPTIONS')
head = partial(AsyncRequest, 'HEAD')
post = partial(AsyncRequest, 'POST')
put = partial(AsyncRequest, 'PUT')
patch = partial(AsyncRequest, 'PATCH')
delete = partial(AsyncRequest, 'DELETE')

partial was imported from functools at the top of the file.

from functools import partial

The documentation for functool.partial says:

Return a new partial object which when called will behave like func called with the positional arguments args and keyword arguments keywords. If more arguments are supplied to the call, they are appended to args. If additional keyword arguments are supplied, they extend and override keywords.

基本上，调用 grequests.get 会调用 AsyncRequest("GET", **其他参数在此处**)。 AsyncRequest is a function which creates a new AsyncRequest.它的文档说:

""" Asynchronous request.
Accept same parameters as ``Session.request`` and some additional:
:param session: Session which will do request
:param callback: Callback called on response.
                 Same as passing ``hooks={'response': callback}``
"""

session 是之前定义的:

from requests import Session

Here's a guide on using requests sessions.