python - 由于用户代理问题，AIOHTTP 客户端超时

Question

aiohttp 客户端遇到问题。
问题:我需要通过 https url 下载图像，它与 requests.get() 配合使用效果很好，但由于 aiohttp 超时而失败。

这是失败的示例:

url = "https://www.miamiherald.com/wps/source/images/miamiherald/facebook.jpg"
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'}

async with aiohttp.ClientSession().get(url, headers=headers) as response:
    content = await response.read()

得到:

Fatal read error on socket transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x10fd8be80>
transport: <_SelectorSocketTransport fd=11 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 801, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
TimeoutError: [Errno 60] Operation timed out

同时，请求可以使用相同的 header 正常工作!

url = "https://www.miamiherald.com/wps/source/images/miamiherald/facebook.jpg"
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'}

r = requests.get(url, headers = headers, stream=True)

任何人都可以帮助我使其适用于这个特定的示例吗？

Answer 1

只需将超时参数传递给 cs.get(timeout=...) 或 cs(timeout=...)。这是文档 https://docs.aiohttp.org/en/stable/client_quickstart.html#timeouts

示例:

import asyncio
import aiohttp
from aiohttp.client import ClientTimeout


async def test():
    url = "https://www.google.com/photos/about/static/images/google.svg"
    headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'
    }

    timeout = ClientTimeout(total=0)  # `0` value to disable timeout
    cs = aiohttp.ClientSession()

    async with cs.request('get', url, headers=headers, timeout=timeout) as response:
        print(response.status)
        content = await response.read()

    await cs.close()


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(test())