python - 在 Python 类中使用 asyncio 和 aiohttp

Question

我一直在试图弄清楚如何在类中使用 asyncio 和 aiohttp。如果我只是尝试在没有类的情况下运行脚本(只需按原样使用函数)，一切都会正常工作。一旦我将所有函数放入一个类中并尝试在 Main.py 中使用该类，脚本就会锁定而不会出现任何错误。不太确定从这里到哪里去，我猜我必须以不同的方式设置我的类(class)才能使其正常工作。如果有人知道为什么这不起作用，如果您分享我做错的事情，我将不胜感激。感谢您抽出时间。

获取.py

import asyncio
from aiohttp import ClientSession

class Fetch:
 def __init__(self, proxy=None):
  self.proxy = proxy
  self.headers =  {'user-agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'}

 def set_headers(self, headers):
  if not headers:
   headers = self.headers
  return headers

 def set_proxy(self, proxy):
  if proxy:
   p = proxy
  else:
   p = self.proxy
  return "http://{}".format(p)

 async def get_fetch(self, session, url, headers=None, proxy=None, params=None, timeout=9):
  array = []
  while True:
   try:
    async with session.get(url, headers=self.set_headers(headers), proxy=self.set_proxy(proxy), params=params, timeout=timeout) as r:
     print (r.status)
     if r.status == 200:
      obj = await r.read()
      array.append(obj)
      break
   except:
    pass
  return array

 async def get_bound(self, sem, session, url):
  async with sem:
   array = await self.get_fetch(session, url)
   return array

 async def get_run(self, urls, semaphores=400):
  tasks = []
  sem = asyncio.Semaphore(semaphores)

  async with ClientSession() as session:
   for url in urls:
    task = asyncio.ensure_future(self.get_bound(sem, session, url))
    tasks.append(task)

  responses = await asyncio.gather(*tasks)
  return responses

 def get(self, urls):
  loop = asyncio.get_event_loop()
  future = asyncio.ensure_future(self.get_run(urls))
  array = loop.run_until_complete(future)
  loop.close()
  return [ent for sublist in array for ent in sublist]

主.py

from Browser import Fetch
from bs4 import BeautifulSoup

proxy = 'xxx.xxx.xxx.xxx:xxxxx'
fetch = Fetch(proxy)

if __name__ == '__main__':
 urls = ['http://ip4.me','http://ip4.me','http://ip4.me']
 array = fetch.get(urls)
 for obj in array:
  soup = BeautifulSoup(obj, 'html.parser')
  for ip in soup.select('tr +  tr td font'):
   print(ip.get_text())

Answer 1

您的缩进错误。

async with ClientSession() as session:
    for url in urls:
        task = asyncio.ensure_future(self.get_bound(sem, session, url))
        tasks.append(task)

responses = await asyncio.gather(*tasks)
return responses

将最后两行放回 with block 中。

您的代码类似于 https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html 。在此引用中，await requests 和相关语句完全位于 with block 内，否则您的代码会让 ClientSession 实例超出范围(并在 http 调用返回之前关闭底层 session 。

顺便说一句，请考虑您的代码的标准缩进样式。单个空格使得很难发现这些简单的错误。