python - 从 url 下载 .html 文件时出现超时错误

Question

从 url 下载 html 页面时出现以下错误。

Error: raise URLError(err) urllib2.URLError: <urlopen error [Errno
 10060] A connection attempt failed because the connected party did not
 properly respond after a period of time or established connection
 failed because connected host has failed to respond>

代码:

import urllib2 
hdr = {'User-Agent': 'Mozilla/5.0'}

for i,site in enumerate(urls[index]):
    print (site)
    req = urllib2.Request(site, headers=hdr)
    page = urllib2.build_opener(urllib2.HTTPCookieProcessor).open(req)
    page_content = page.read()
    with open(path_current+'/'+str(i)+'.html', 'w') as fid:
        fid.write(page_content)

我认为这可能是由于某些代理设置或更改超时造成的，但我不确定。请帮忙，我手动检查了网址似乎可以正常打开。

Answer 1

好吧，由于大多数情况下您不会遇到这种情况，我可以推断您的网络可能很慢。尝试通过以下方式设置超时:

req = urllib2.Request(site, headers=hdr)
timeout_in_sec = 360
page = urllib2.build_opener(urllib2.HTTPCookieProcessor).open(req, timeout=timeout_in_sec)
page_content = page.read()