python requests-futures 慢 - 线程不正确？

Question

您好，我已经使用 requests-futures 库编写了一个多线程请求和响应处理程序。

但是，它似乎很慢，并且不像我想象的那样是异步的。输出缓慢且有序，没有像我预期的那样交错，如果它正确线程化。

我的问题是为什么我的代码很慢，我可以做些什么来加快它的速度？一个例子会很棒。

代码如下:

#!/usr/bin/python
import requests
import time
from concurrent.futures import ThreadPoolExecutor
from requests_futures.sessions import FuturesSession

session = FuturesSession(executor=ThreadPoolExecutor(max_workers=12))

def responseCallback( sess, resp ):
    response = resp.text
    if not "things are invalid" in response in response:
        resp.data = "SUCCESS %s" % resp.headers['content-length']
    else:
        resp.data = "FAIL %s" % resp.headers['content-length']

proxies = {
"http":"http://localhost:8080",
"https":"https://localhost:8080"
}

url = 'https://www.examplehere.com/blah/etc/'
headers= {
'Host':'www.examplehere.com',
'Connection':'close',
'Cache-Control':'max-age=0',
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Origin':'https://www.examplehere.com',
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/533.32 (KHTML, like Gecko) Ubuntu Chromium/34.0.1847.123 Chrome/34.0.1847.123 Safari/337.12',
'Content-Type':'application/x-www-form-urlencoded',
'Referer':'https://www.exampleblah.etc/',
'Accept-Encoding':'gzip,deflate,sdch',
'Accept-Language':'en-US,en;q=0.8,de;q=0.6',
'Cookie':'blah=123; etc=456;',
}

for n in range( 0, 9999 ):
    #wibble = n.zfill( 4 )
    wibble = "%04d" % n
    payload = { 
    'name':'test',
    'genNum':wibble,
    'Button1':'Push+Now'
    }
    #print payload
    #r = requests.post( url, data=payload, headers=headers, proxies=proxies, verify=False )
    future = session.post( url, data=payload, headers=headers, verify=False, background_callback=responseCallback )
    response = future.result()
    print( "%s : %s" % ( wibble, response.data ) )

理想情况下，我想修复我的实际代码，仍然使用我已经使用过的库，但如果由于某种原因它不好，我愿意接受建议...

编辑:我目前正在使用 python2 和 concurrent.futures backport。

编辑:慢 - 大约每秒一个请求，而不是并发，而是一个接一个，所以请求 1、响应 1、请求 2、响应 2 - 我希望它们在请求发出和进入多个线程时交错？

Answer 1

下面的代码是提交多个请求的另一种方式，一次处理其中的几个请求，然后打印出结果。结果在准备就绪时打印，不一定按照提交时的顺序打印。

它还使用大量日志记录来帮助调试问题。它捕获日志记录的有效负载。多线程代码很难，日志越多越好!

来源

import logging, sys

import concurrent.futures as cf
from requests_futures.sessions import FuturesSession

URL = 'http://localhost'
NUM = 3

logging.basicConfig(
    stream=sys.stderr, level=logging.INFO,
    format='%(relativeCreated)s %(message)s',
    )

session = FuturesSession()
futures = {}

logging.info('start')
for n in range(NUM):
    wibble = "%04d" % n
    payload = { 
        'name':'test',
        'genNum':wibble,
        'Button1':'Push+Now'
    }
    future = session.get( URL, data=payload )
    futures[future] = payload

logging.info('requests done, waiting for responses')

for future in cf.as_completed(futures, timeout=5):
    res = future.result()
    logging.info(
        "wibble=%s, %s, %s bytes",
        futures[future]['genNum'],
        res,
        len(res.text),
    )

logging.info('done!')

输出

69.3101882935 start
77.9430866241 Starting new HTTP connection (1): localhost
78.3731937408 requests done, waiting for responses
79.4050693512 Starting new HTTP connection (2): localhost
84.498167038 wibble=0000, <Response [200]>, 612 bytes
85.0481987 wibble=0001, <Response [200]>, 612 bytes
85.1981639862 wibble=0002, <Response [200]>, 612 bytes
85.2642059326 done!