Python 请求 : ignore exceptions and errors while connecting to proxy server

Question

我用 Python 编写了我的第一个程序。

#This program casts votes in online poll using different proxy servers for each request.
#It works, but some proxy servers cause errors crashing the whole thing.
#To avoid that, I would like it to skip those servers and ignore the errors.

import requests
import time

#Votes to be cast
votes = 5 

#Makes proxy list
f=open('proxy2.txt')
lines=f.read().splitlines()
f.close()

#Vote counter
i = 1 

#Proxy list counter
j = 0 

while (i<=votes):

#Tests and moves to next proxy if there was a problem.
    try:
        r = requests.get('http://www.google.com')
    except requests.exceptions.RequestException:
        j = j + 1

#Headers copied from my browser. Some of them cause errors. Could you tell me why?
    headers =   {
                    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
                    #'Accept-Encoding': 'gzip, deflate',
                    #'Accept-Language': 'pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4',
                    #'Cache-Control': 'max-age=0',
                    #'Connection': 'keep-alive',
                    #'Content-Length': '101',
                    'Content-Type': 'application/x-www-form-urlencoded',
                    #'Host': 'www.mylomza.pl',
                    #'Origin': 'http://www.mylomza.pl',
                    #'Referer': 'http://www.mylomza.pl/home/lomza/item/11780-wybierz-miss-%C5%82ks-i-portalu-mylomzapl-video-i-foto.html',
                    #'Upgrade-Insecure-Requests': '1',
                    #'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
                }

    proxies =   {
                    'http': 'http://'+lines[j] #31.207.0.99:3128
                }

    r = requests.get('http://www.mylomza.pl/home/lomza/item/11780-wybierz-miss-%C5%82ks-i-portalu-mylomzapl-video-i-foto.html', headers=headers, proxies=proxies, timeout=10)

#The funny part - form, that I have to post, requires some kind of ID and this is my way of getting it :P Feel free to suggest an alternative way.
    userid = r.text[(22222-32):22222]

    print('Voter', userid, 'registered.')

    data = {
                'voteid': '141',
                'task_button': 'Głosuj',
                'option': 'com_poll',
                'task': 'vote',
                'id': '25',
                userid: '1'
               }

    r = requests.post('http://www.mylomza.pl/home/lomza/item/index.php', headers=headers, cookies=r.cookies, data=data, proxies=proxies, timeout=10)

    print('Vote nr', i, 'cast from', lines[i])

    i = i + 1
    j = j + 1

    time.sleep(1)

我需要的是让它处理异常和错误。

#Tests and moves to next proxy if there was a problem.
try:
    r = requests.get('http://www.google.com')
except requests.exceptions.RequestException:
    j = j + 1

除此之外，我还可以使用另一种方法来实现这一目标:

#The funny part - form, that I have to post, requires some kind of ID and this is my way of getting it :P Feel free to suggest an alternative way.
    userid = r.text[(22222-32):22222]

有时我的方法不起作用(如下例)。第一次投票通过，第二次没有通过，然后全部崩溃。

Voter 53bf55490ebd07d9c190787c5c6ca44c registered.
Vote nr 1 cast from 111.23.6.161:80
Voter  registered.
Vote nr 2 cast from 94.141.102.203:8080
Traceback (most recent call last):
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connection.py", line 142, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\util\connection.py", line 91, in create_connection
    raise err
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\util\connection.py", line 81, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 362, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 1128, in _send_request
    self.endheaders(body)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 911, in _send_output
    self.send(msg)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 854, in send
    self.connect()
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connection.py", line 167, in connect
    conn = self._new_conn()
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connection.py", line 147, in _new_conn
    (self.host, self.timeout))
requests.packages.urllib3.exceptions.ConnectTimeoutError: (<requests.packages.urllib3.connection.HTTPConnection object at 0x03612730>, 'Connection to 94.141.102.203 timed out. (connect timeout=10)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\adapters.py", line 403, in send
    timeout=timeout
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 623, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\util\retry.py", line 281, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='94.141.102.203', port=8080): Max retries exceeded with url: http://www.mylomza.pl/home/lomza/item/11780-wybierz-miss-%C5%82ks-i-portalu-mylomzapl-video-i-foto.html (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x03612730>, 'Connection to 94.141.102.203 timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\PollVoter.py", line 50, in <module>
    r = requests.get('http://www.mylomza.pl/home/lomza/item/11780-wybierz-miss-%C5%82ks-i-portalu-mylomzapl-video-i-foto.html', headers=headers, proxies=proxies, timeout=10)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\api.py", line 71, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\Adrian\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\adapters.py", line 459, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='94.141.102.203', port=8080): Max retries exceeded with url: http://www.mylomza.pl/home/lomza/item/11780-wybierz-miss-%C5%82ks-i-portalu-mylomzapl-video-i-foto.html (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x03612730>, 'Connection to 94.141.102.203 timed out. (connect timeout=10)'))

Answer 1

看起来您正在用太多请求淹没服务器，这就是为什么您会收到其他错误，例如 requests.packages.urllib3.exceptions.MaxRetryError，因为服务器可能会限制您在给定时间内可以建立的连接数。您可以尝试处理输出中列出的所有异常，也可以尝试减少对您请求的 url 的尝试。

[编辑] 或者，如果您想暴力破解并处理所有错误和异常，请尝试以下操作

except:
    j = j + 1

[编辑:]您可以尝试 https: 以及 http:

[编辑] 找到这个:

If the remote server is very slow, you can tell Requests to wait forever for a response, by passing None as a timeout value and then retrieving a cup of coffee.

r = requests.get('https://github.com', timeout=None)