Python Mechanize 连接失败问题

Question

我正在运行一个爬虫，它在 MAC 和 Ubuntu 系统上运行良好。当我在 Windows 中运行它时，它会产生以下堆栈跟踪:

  File "build\bdist.win32\egg\mechanize\_mechanize.py", line 203, in open
    return self._mech_open(url, data, timeout=timeout)
  File "build\bdist.win32\egg\mechanize\_mechanize.py", line 230, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "build\bdist.win32\egg\mechanize\_opener.py", line 193, in open
    response = urlopen(self, req, data)
  File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 344, in _open
    '_open', req)
  File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 332, in _call_chain
    result = func(*args)
  File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 1170, in https_open
    return self.do_open(conn_factory, req)
  File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 1118, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>

Answer 1

我非常怀疑问题与您的系统有关。要解决此问题，我建议:

1. 检查您的 Mechanize 版本。旧版本可能会以不同方式处理请求。
2. 调查 set_handle_robots 和 addheaders。问题可能与每个 IP 的请求过多有关。 (我怀疑这是真正的问题。)
3. 检查错误处理。正如上面的评论中所述，您无论如何都应该添加适当的错误处理。

如果上述解决方案均无效，请提供导致错误的代码。