python - 如何使用 python 请求执行限时响应下载？

Question

当使用 python 下载大文件时，我不仅要为连接过程设置时间限制，还要为下载设置时间限制。

我正在尝试使用以下 python 代码:

import requests

r = requests.get('http://ipv4.download.thinkbroadband.com/1GB.zip', timeout = 0.5, prefetch = False)

print r.headers['content-length']

print len(r.raw.read())

这不起作用(下载没有时间限制)，正如文档中正确指出的那样:https://requests.readthedocs.org/en/latest/user/quickstart/#timeouts

如果可以，那就太好了:

r.raw.read(timeout = 10)

问题是，如何限制下载时间？

Answer 1

答案是:不要使用请求，因为它是阻塞的。使用非阻塞网络 I/O，例如 eventlet:

import eventlet
from eventlet.green import urllib2
from eventlet.timeout import Timeout

url5 = 'http://ipv4.download.thinkbroadband.com/5MB.zip'
url10 = 'http://ipv4.download.thinkbroadband.com/10MB.zip'

urls = [url5, url5, url10, url10, url10, url5, url5]

def fetch(url):
    response = bytearray()
    with Timeout(60, False):
        response = urllib2.urlopen(url).read()
    return url, len(response)

pool = eventlet.GreenPool()
for url, length in pool.imap(fetch, urls):
    if (not length):
        print "%s: timeout!" % (url)
    else:
        print "%s: %s" % (url, length)

产生预期结果:

http://ipv4.download.thinkbroadband.com/5MB.zip: 5242880
http://ipv4.download.thinkbroadband.com/5MB.zip: 5242880
http://ipv4.download.thinkbroadband.com/10MB.zip: timeout!
http://ipv4.download.thinkbroadband.com/10MB.zip: timeout!
http://ipv4.download.thinkbroadband.com/10MB.zip: timeout!
http://ipv4.download.thinkbroadband.com/5MB.zip: 5242880
http://ipv4.download.thinkbroadband.com/5MB.zip: 5242880