python - grequests - 打开文件太多错误

Question

我正在尝试执行大约 100k GET 请求并解析每个请求的响应正文。我认为 grequests 将是一个好方法，但我收到与“打开文件太多”相关的错误。代码如下:

import grequests

with open("./100k-sites.csv", "r") as f:
    urls = ["http://" + line.rstrip() for line in f]

rs = (grequests.get(u, timeout=1) for u in urls)
responses = grequests.map(rs)

for r in responses:
    try:
        # do something with the response body
    except:
        pass

有人有这方面的经验吗？我收到的错误是:

requests.packages.urllib3.connection.HTTPConnection object at 0x7f817ab36898>: Failed to establish a new connection [Errno 24] Too many open files

Answer 1

也许这只是一种解决方法(正如上面提到的 discussion 中的某人所说)，但恕我直言，值得在这里写一下，可以通过两行来修复它:

import resource
resource.setrlimit(resource.RLIMIT_NOFILE, (110000, 110000))