python - 使用 aiohttp 检测 http 响应编码

Question

我正在尝试学习如何使用 asyncio 构建异步网络爬虫。以下是用于测试框架的粗略爬虫:

import asyncio, aiohttp
from bs4 import BeautifulSoup

@asyncio.coroutine
def fetch(url):
    with (yield from sem):
        print(url)
        response = yield from aiohttp.request('GET',url)
        response = yield from response.read_and_close()
    return response.decode('utf-8')

@asyncio.coroutine
def get_links(url):
    page = yield from fetch(url)
    soup = BeautifulSoup(page)
    links = soup.find_all('a',href=True)
    return [link['href'] for link in links if link['href'].find('www') != -1]

@asyncio.coroutine
def crawler(seed, depth, max_depth=3):
    while True:
        if depth > max_depth:
            break
        links = yield from get_links(seed)
        depth+=1
        coros = [asyncio.Task(crawler(link,depth)) for link in links]
        yield from asyncio.gather(*coros)

sem = asyncio.Semaphore(5)
loop = asyncio.get_event_loop()
loop.run_until_complete(crawler("http://www.bloomberg.com",0))

虽然 asyncio 似乎有很好的文档记录，但 aiohttp 似乎只有很少的文档，所以我正在努力自己解决一些问题。

首先，我们有没有办法检测页面响应的编码？其次，我们可以请求连接在 session 中保持事件状态吗？还是像 requests 中那样默认为 True？

Answer 1

您可以查看 response.headers['Content-Type'] 或使用 chardet 库来处理格式错误的 HTTP 响应。响应体是bytes字符串。

对于 keep-alive 连接，您应该使用 connector，例如:

connector = aiohttp.TCPConnector(share_cookies=True)

response1 = yield from aiohttp.request('get', url1, connector=connector)
body1 = yield from response1.read_and_close()
response2 = aiohttp.request('get', url2, connector=connector)
body2 = yield from response2.read_and_close()