twitter - Tweepy 错误 104 : Connection aborted

Question

我正在尝试使用 Tweepy 抓取一些推文，但在数百次请求后连接崩溃并出现以下错误:tweepy.error.TweepError: 发送请求失败:('Connection aborted.', error("(104, 'ECONNRESET')",))

我的代码是这样的:

  for status in tweepy.Cursor(api.search,
                           q="",
                           count=100,
                           include_entities=True,
                           monitor_rate_limit=True, 
                           wait_on_rate_limit=True,
                           wait_on_rate_limit_notify = True,
                           retry_count = 5, #retry 5 times
                           retry_delay = 5, #seconds to wait for retry
                           geocode ="34.0207489,-118.6926066,100mi", # los angeles
                           until=until_date,
                           lang="en").items():

      try:
        towrite = json.dumps(status._json)
        output.write(towrite + "\n")
      except Exception, e:
        log.error(e)
      c+=1
      if c % 10000 == 0:  # 100 requests, sleep
        time.sleep(900) # sleep 15 min

我可以使用 try/except 捕获错误，但我无法从崩溃点重新启动游标。有谁知道如何解决此错误，或从上次已知状态重新启动光标？

谢谢!

Answer 1

Tweepy 文档说请求/15 分钟窗口(用户身份验证)为 180，但显然 sleep 时间过长会影响连接可靠性(在某些请求之后)，因此如果您每 5 秒运行一次请求，一切似乎都正常:

   for status in tweepy.Cursor(api.search,
                       q="",
                       count=100,
                       include_entities=True,
                       monitor_rate_limit=True, 
                       wait_on_rate_limit=True,
                       wait_on_rate_limit_notify = True,
                       retry_count = 5, #retry 5 times
                       retry_delay = 5, #seconds to wait for retry
                       geocode ="34.0207489,-118.6926066,100mi", # los angeles
                       until=until_date,
                       lang="en").items():

  try:
    towrite = json.dumps(status._json)
    output.write(towrite + "\n")
  except Exception, e:
    log.error(e)
  c+=1
  if c % 100 == 0:  # first request completed, sleep 5 sec
    time.sleep(5)