python - 提取 Youtube 搜索结果时出现问题

Question

我是 Python 的新手，已经学习了使用 bs4 进行网页抓取的基础知识。在这里我试图提取 Youtube 搜索结果的所有链接，它不像在其他网站上那样工作。我分析了搜索结果的 html 数据，搜索结果的链接在 id 为“video title”的 anchor 标记中，但该标记没有出现在我的 bs4 解析的 html 文档中

from bs4 import BeautifulSoup as bs
import requests
name=input("Enter video name ")
url='https://www.youtube.com/results?search_query='+name
searched=requests.get(url)
soup=bs(searched.text,'html.parser')
aid=soup.find_all('a',{'id':'video-title'})
print(aid)

我希望输出包含所有搜索结果。我还没有学过其他包，如果可能的话，我想在 bs4 中做这个。

Answer 1

获取 YouTube 搜索结果数据的所有这些麻烦只是浪费时间和精力。

为什么不试试这些选项

• YouTube Data API
• Selenium + Headless chrome

也就是说，对于前 20 个结果，您可以从源中的 JavaScript 内容中获取数据。答案如下。

在理解生成的 json 大约 1 小时后，它仍然无法进行某些查询。YouTube 是一个非常复杂的网站。响应可能因位置、浏览器、搜索查询等而异。

我们正在从源代码中的这个脚本标签中提取数据。

代码:

from bs4 import BeautifulSoup as bs
import requests
import re
import json
headers={
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36'
}
name=input("Enter video name: ")
url='https://www.youtube.com/results?search_query=hello'+name
searched=requests.get(url,headers=headers)
soup=bs(searched.text,'html.parser')
aid=soup.find('script',string=re.compile('ytInitialData'))
extracted_josn_text=aid.text.split(';')[0].replace('window["ytInitialData"] =','').strip()
video_results=json.loads(extracted_josn_text)
#print(item_section=video_results["contents"]["twoColumnSearchResultsRenderer"]["primaryContents"]["sectionListRenderer"]["contents"][1])
item_section=video_results["contents"]["twoColumnSearchResultsRenderer"]["primaryContents"]["sectionListRenderer"]["contents"][0]["itemSectionRenderer"]["contents"]

for item in item_section:
    try:
        video_info=item["videoRenderer"]
        title=video_info["title"]["simpleText"]
        url=video_info["navigationEndpoint"]["commandMetadata"]["webCommandMetadata"]["url"]
        print('Title:',title)
        print('Url:',url, end="\n----------\n")
    except KeyError:
            pass

输出:

Enter video name: hello
Title: New Punjabi Songs 2017-Hello Hello(Ful Song)-Prince Narula-Yuvika Chaudhary-Latest Punjabi Song 2017
Url: /watch?v=mv326-zVpAQ
----------
Title: Alan Walker - The Spectre
Url: /watch?v=wJnBTPUQS5A
----------
Title: Hello Hello (Full HD) - Rajvir Jawanda | MixSingh | Josan Bros | New Punjabi Songs 2018
Url: /watch?v=xydupjQSj44
----------
Title: Bachchan - Hello Hello - Kannada Movie Full Song Video | Sudeep | Bhavana | V Harikrishna
Url: /watch?v=oLMMgoug4Uk
----------
Title: Hello Hello latest 2017 16 june punjabi song
Url: /watch?v=MqCSsPXw8QU
----------
Title: 👋  Hello Hello 👋  | + More Kids Songs | Super Simple Songs
Url: /watch?v=saDkICxEdgY
----------
Title: 'Gallan Goodiyaan' Full VIDEO Song | Dil Dhadakne Do | T-Series
Url: /watch?v=jCEdTq3j-0U
----------
Title: Hello Hello Gippy Grewal Feat. Dr. Zeus Full Song HD | Latest Punjabi Song 2013
Url: /watch?v=IRW2O4QZhgs
----------
Title: Hello Hello | Pataakha | Malaika Arora | Vishal Bhardwaj & Rekha Bhardwaj | Gulzar | Ganesh Acharya
Url: /watch?v=RxBAitQLSLA
----------
Title: Hello Hello (Lyrical Audio) Prince Narula ft. Yuvika Chaudhary | Punjabi Lyrical Audio 2017 | WHM
Url: /watch?v=v8VIsIvhDoQ
----------
Title: Hello Hello Full Video Song || Bhale Bhale Magadivoi || Nani, Lavanya Tripathi
Url: /watch?v=y3FI02OO_kU
----------
Title: Hello hello gaad bahe dhufee na egaa  (new comedy hhhhhh)
Url: /watch?v=DuRrcTo4rgg
----------
Title: Proper Patola - Official Video | Namaste England | Arjun | Parineeti | Badshah | Diljit | Aastha
Url: /watch?v=YmXJp4RtBCM
----------
Title: Official Video: Nikle Currant Song | Jassi Gill | Neha Kakkar | Sukh-E Muzical Doctorz | Jaani
Url: /watch?v=uBaqgt5V0mU
----------
Title: Insane (Full Song)  Sukhe - Jaani - Arvindr Khaira - White Hill Music - Latest Punjabi Song 2018
Url: /watch?v=mKpPhVVF8So
----------
Title: Radha bole HELLO HELLO-cartoon song mix with step up 2
Url: /watch?v=TFCTgNCzrck
----------
Title: Hello Song | CoCoMelon Nursery Rhymes & Kids Songs
Url: /watch?v=fxVMqaViVaA
----------
Title: Bachchan - Hello Hello Unplugged Version | Sudeep | Bhavana | V Harikrishna
Url: /watch?v=lvH3kTGJeEQ
----------
Title: Hello Hello! Can You Clap Your Hands? | Original  Kids Song | Super Simple Songs
Url: /watch?v=fN1Cyr0ZK9M
----------

您可以尝试的最后一件事是模拟 youtube 本身使用的 API

即。 POST 请求到

https://www.youtube.com/results?search_query=yoursearchtext

它有很多 cookie 和 session 值作为参数发送。您可能需要模拟所有这些。您可能需要使用 Requests session objects这样做。