javascript - Web 抓取 duckduckgo，但获取格式错误的链接

Question

我使用 BeautifulSoup 库创建了一个 Python 3 脚本。它的作用是使用以下 url 转到 duckduckgo 搜索引擎:https://duckduckgo.com/?q=searchterm 然后，它将显示给我第一页中的所有网站。

这是代码，它运行良好:

import requests
from bs4 import BeautifulSoup

r = requests.get('https://duckduckgo.com/html/?q=test')
soup = BeautifulSoup(r.text, 'html.parser')
results = soup.find_all('a', attrs={'class':'result__a'})

i = 0
while i < len(results):
    link = results[i]
    url = link['href']
    print(url)
    i = i + 1

问题是，我没有获得格式正确的 URL(例如:https://www.google.com)。相反，我以搜索查询的格式获取所有网址。

这是我在 duckduckgo 上搜索 test 时的意思:

/l/?kh=-1&uddg=https%3A%2F%2Fduckduckgo.com%2Fy.js%3Fu3%3Dhttps%253A%252F%252Fr.search.yahoo.com%252Fcbclk%252FdWU9MEQwQzVENEZDNDU0NDlEMyZ1dD0xNTM4MzE4MTI3MzE5JnVvPTc3NTg0MzM1OTYxMTUyJmx0PTImZXM9ZVBGTU9iWUdQUy42cVdRVQ%252D%252D%252FRV%253D2%252FRE%253D1538346927%252FRO%253D10%252FRU%253Dhttps%25253a%25252f%25252fwww.bing.com%25252faclick%25253fld%25253dd3peyDLOVSWraifG78tpZ1GjVUCUzCMDkx%252DfJrFXeY2IfiXIwUmngX%252DYKvZWQ6q7hPHC_3kc%252DzBWS1SE015Or2c3CncFMVc9OjVV5OyB2kJqXdRsOzRnaCGy8gYCPuival0gLe7WCkfk_%252DAVKTWmYxranfh02ficTC7i6oC38n2q9U9KPe%252526u%25253dhttps%2525253a%2525252f%2525252fwww.dotdrugconsortium.com%2525252f%2525253futm_source%2525253dbing%25252526utm_medium%2525253dcpc%25252526utm_campaign%2525253dadcenter%25252526utm_term%2525253ddottest%252526rlid%25253d590f68ae34ff126ed0e3331eebd0c4fb%252FRK%253D2%252FRS%253DeKe3rY19jdg9vb_ayBSboMzPU1g%252D%26ad_provider%3Dyhs%26vqd%3D3%2D12729109948094676568590283448597440227%2D122882305188756590950269013545136161936
/l/?kh=-1&uddg=https%3A%2F%2Fwww.merriam%2Dwebster.com%2Fdictionary%2Ftest
/l/?kh=-1&uddg=https%3A%2F%2Fwww.speedtest.net%2F
/l/?kh=-1&uddg=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FTest
/l/?kh=-1&uddg=https%3A%2F%2Fwww.dictionary.com%2Fbrowse%2Ftest
/l/?kh=-1&uddg=https%3A%2F%2Fwww.thefreedictionary.com%2Ftest
/l/?kh=-1&uddg=https%3A%2F%2Fwww.16personalities.com%2F
/l/?kh=-1&uddg=https%3A%2F%2Fwww.speakeasy.net%2Fspeedtest%2F
/l/?kh=-1&uddg=http%3A%2F%2Fwww.humanmetrics.com%2Fcgi%2Dwin%2Fjtypes2.asp
/l/?kh=-1&uddg=https%3A%2F%2Fwww.typingtest.com%2F%3Fab
/l/?kh=-1&uddg=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FTest_cricket
/l/?kh=-1&uddg=https%3A%2F%2Fged.com%2F
/l/?kh=-1&uddg=http%3A%2F%2Fspeedtest.xfinity.com%2F
/l/?kh=-1&uddg=https%3A%2F%2Fwww.16personalities.com%2Ffree%2Dpersonality%2Dtest
/l/?kh=-1&uddg=https%3A%2F%2Fwww.merriam%2Dwebster.com%2Fthesaurus%2Ftest
/l/?kh=-1&uddg=http%3A%2F%2Ftest%2Dipv6.com%2F
/l/?kh=-1&uddg=https%3A%2F%2Fwww.thesaurus.com%2Fbrowse%2Ftest
/l/?kh=-1&uddg=http%3A%2F%2Fspeedtest.att.com%2Fspeedtest%2F
/l/?kh=-1&uddg=http%3A%2F%2Fspeedtest.googlefiber.net%2F
/l/?kh=-1&uddg=http%3A%2F%2Ftest.salesforce.com%2F
/l/?kh=-1&uddg=https%3A%2F%2Fmy.uscis.gov%2Fprep%2Ftest%2Fcivics
/l/?kh=-1&uddg=https%3A%2F%2Fwww.tests.com%2F
/l/?kh=-1&uddg=https%3A%2F%2Fen.wiktionary.org%2Fwiki%2FTest
/l/?kh=-1&uddg=https%3A%2F%2Ftestmy.net%2F
/l/?kh=-1&uddg=https%3A%2F%2Fwww.google.com%2F
/l/?kh=-1&uddg=https%3A%2F%2Fwww.queendom.com%2Ftests%2Findex.htm
/l/?kh=-1&uddg=http%3A%2F%2Fwww.yourdictionary.com%2Ftest
/l/?kh=-1&uddg=http%3A%2F%2Fwww.testout.com%2F
/l/?kh=-1&uddg=https%3A%2F%2Fimplicit.harvard.edu%2Fimplicit%2Ftakeatest.html
/l/?kh=-1&uddg=http%3A%2F%2Fwww.act.org%2Fcontent%2Fact%2Fen%2Fproducts%2Dand%2Dservices%2Fthe%2Dact.html
/l/?kh=-1&uddg=https%3A%2F%2Fwww.ets.org%2Fgre%2F

我想知道是否有办法以标准格式显示所有这些 url。

编辑:这不是我的其他主题的重复，因为在上一个主题中我被告知库 PyCurl 不会得到我想要的东西(它无法捕获 url 中的 javascript 代码) .这里我的代码可以正常工作，但我得到的输出不是我所期望的。

Answer 1

Python 的 urllib.parse图书馆可以为您提供以下帮助:

from bs4 import BeautifulSoup
import urllib.parse
import requests

r = requests.get('https://duckduckgo.com/html/?q=test')
soup = BeautifulSoup(r.text, 'html.parser')
results = soup.find_all('a', attrs={'class':'result__a'}, href=True)

for link in results:
    url = link['href']
    o = urllib.parse.urlparse(url)
    d = urllib.parse.parse_qs(o.query)
    print(d['uddg'][0])

这会显示一些开始的东西:

http://www.speedtest.net/
https://www.merriam-webster.com/dictionary/test
https://en.wikipedia.org/wiki/Test
https://www.thefreedictionary.com/test
https://www.dictionary.com/browse/test

第一次使用urlparse()获取路径组件。从这里获取 query 字符串并将其传递给 parse_qs()进一步处理它。然后，您可以使用 uddg 名称提取链接。