python-3.x - Selenium - urllib.error.URLError :

Question

注意:我花了一个多小时试图解决这个问题，但没有找到适合我的解决方案。

最后证明这是一个非常简单的错误，但我想我会提出这个问题，以便万一其他人有同样的问题可以快速找到解决方案。

---

问题

我试图用以下代码抓取一个网站:

phantomjs_path = '/Users/xxx/xxx/phantomjs-2.1.1-macosx/bin/phantomjs'

driver = webdriver.PhantomJS(executable_path=phantomjs_path)

driver.set_window_size(1024, 768) #optional

driver.get(url)

# wait
element = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.CLASS_NAME, "flightrow")))

response = driver.find_element_by_css_selector('table[class="flighttable"]')

driver.quit()

html = response.get_attribute('outerHTML') #pass from webdrive object to string

出现以下错误:

Traceback (most recent call last):


File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 849, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 711, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 702, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "my_script.py", line 1251, in <module>
    MyObject.script_main()
  File "my_script.py", line 1232, in script_main
    self.parse_js(url)
  File "my_script.py", line 1202, in parse_js
    print('response:', response.text)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/selenium/webdriver/remote/webelement.py", line 68, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/selenium/webdriver/remote/webelement.py", line 461, in _execute
    return self._parent.execute(command, params)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 234, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/selenium/webdriver/remote/remote_connection.py", line 401, in execute
    return self._request(command_info[0], url, body=data)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/selenium/webdriver/remote/remote_connection.py", line 471, in _request
    resp = opener.open(request, timeout=self._timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1282, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1256, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 61] Connection refused>

在 Chrome 浏览器中手动加载 url 是有效的。

无论如何，我尝试将 url 从 https 切换到 http，但我仍然遇到同样的错误。

此外，在前一天我没有收到任何错误，所以我认为这不会是防火墙的问题，正如我在其他一些问题中读到的那样。

查看解决方案的答案...

Answer 1

事实证明，显然我已经向上移动了 driver.quit() 行，因此在调用“get_atribute”时出现了错误。

解决方案

只需将 driver.quit() 向下移动:

driver = webdriver.PhantomJS(executable_path=phantomjs_path)

driver.set_window_size(1024, 768) #optional

driver.get(url)

# wait
element = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.CLASS_NAME, "flightrow")))

response = driver.find_element_by_css_selector('table[class="flighttable"]')

html = response.get_attribute('outerHTML') #pass from webdrive object to string

#do not move quite() upwards! even if 'driver' is not specifically called with the command 'get_attribute'
#it will raise an error if driver is closed.
driver.quit()