python - 如何验证 url 是否存在不重定向？

Question

如何验证页面url是否存在而不重定向到未找到的url页面
示例:

import socket
try:
    socket.gethostbyname('www.google.com/imghp')
except socket.gaierror as ex:
    print "Not existe"

它总是返回不存在

Answer 1

您为任务使用了错误的工具!

来自manual :

socket.gethostbyname(hostname)

Translate a host name to IPv4 address format. The IPv4 address is returned as a string, such as '100.50.200.5'. If the host name is an IPv4 address itself it is returned unchanged. See gethostbyname_ex() for a more complete interface. gethostbyname() does not support IPv6 name resolution, and getaddrinfo() should be used instead for IPv4/v6 dual stack support.

该工具用于检查域是否存在，并获取其IP地址:

>>> try:
...     print(socket.gethostbyname('www.google.com'))
... except socket.gaierror as ex:
...     print("Does not exists")
... 
216.58.211.132

您可能想要实际连接到站点并检查是否有页面:

>>> import requests
>>> response = requests.head('http://www.google.com/imghp')
>>> if response.status_code == 404:
...    print("Does not exists")
... else:
...    print("Exists")
...
Exists

.head()方法来自 python-requests仅从网络服务器获取有关页面的信息，而不是页面本身，因此它在网络使用方面非常轻量级。

剧透警告:如果您尝试获取页面内容，请使用 response.content ，你什么也得不到，为此你需要使用 .get()方法。

---

更新#1

您正在检查的网站已损坏，即它不遵循互联网标准。它不是提供 404，而是提供 302 以重定向到状态代码为 200 的“页面不存在”页面:

>>> response = requests.head('http://qamarsoft.com/does_not_exists', allow_redirects=True)
>>> response.status_code
200

要解决这个问题，您需要获取该站点的页面，并检查重定向的 URI 是否在重定向 URL 中包含 404:

>>> response = requests.head('http://qamarsoft.com/does_not_exists'
>>> response.headers['location']
'http://qamarsoft.com/404'

所以测试会变成:

>>> response = requests.head('http://qamarsoft.com/does_not_exists')
>>> if '404' in response.headers['location']:
...     print('Does not exists')
... else:
...     print('Exists')
Exists

---

更新#2

对于第二个URL，你可以在python控制台中自己尝试一下:

>>> import requests
>>> response = requests.head('http://www.***********.ma/does_not_Exists')
>>> if response.status_code == 404:
...    print("Does not exists")
... else:
...    print("Exists")
...
Does not exists
>>> response = requests.head('http://www.***********.ma/annonceur/a/3550/n.php ')
>>> if response.status_code == 404:
...    print("Does not exists")
... else:
...    print("Exists")
...
Exists

---

注意事项

您可能想要安装requests 包:

pip install requests

或者如果你是现代人并且使用 python3:

pip3 install requests