python - 在请求库中，如何避免 "HttpConnectionPool is full, discarding connection"警告？

Question

我在 session 中使用 python 请求库:

def _get_session(self):
    if not self.session:
        self.session = requests.Session()
    return self.session

有时我会在我的日志中收到此警告:

[2014/May/12 14:40:04 WARNING ] HttpConnectionPool is full, discarding connection: www.ebi.ac.uk

我的问题是:为什么这是警告而不是异常？

这是对此负责的代码(来自 http://pydoc.net/Python/requests/0.8.5/requests.packages.urllib3.connectionpool/ ):

def _put_conn(self, conn):
    try:
        self.pool.put(conn, block=False)
    except Full:
        # This should never happen if self.block == True
        log.warning("HttpConnectionPool is full, discarding connection: %s"
                    % self.host)

为什么会在这里捕获这个异常？如果它被重新引发，我可以通过创建新 session 并删除旧 session 来在我的代码中处理这个异常。

如果它只是一个警告，是否意味着它不会以任何方式影响我的结果？我可以忽略它吗？如果没有，我该如何处理这种情况？

Answer 1

来自 http://docs.python-requests.org/en/latest/api/ 中的请求文档

 class requests.adapters.HTTPAdapter(pool_connections=10, pool_maxsize=10, max_retries=0, pool_block=False)

The built-in HTTP Adapter for urllib3.

Provides a general-case interface for Requests sessions to contact HTTP and HTTPS urls by implementing the Transport Adapter interface. This class will usually be created by the Session class under the covers.

Parameters:

• pool_connections – The number of urllib3 connection pools to cache.
• pool_maxsize – The maximum number of connections to save in the pool.
• max_retries (int) – The maximum number of retries each connection should attempt. Note, this applies only to failed connections and timeouts, never to requests where the server returns a response.
• pool_block – Whether the connection pool should block for connections.

下面是一个例子

import requests
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=3)
s.mount('http://', a)

试试这个

a = requests.adapters.HTTPAdapter(pool_connections = N, pool_maxsize = M)

其中 N 和 M 适合您的程序。