python - 从网站获取不同参与者的姓名时遇到问题

Question

我试图从网页中仅获取不同参与者的姓名，但是当我运行下面的脚本时，它什么也没获取。它也不会抛出任何错误。我知道内容是高度动态的，并且在页面源代码中不可用，但仍然有多种方法可以获取它们，例如使用开发工具中可用的脚本标签或公共(public) api。我试图找到所有这些，但没有任何线索继续前进。

Website address

These are the names in the image我想捕获它。当页面打开时，应自动选择左侧面板中的Esports选项卡。

这是我迄今为止尝试过的:

import requests
from bs4 import BeautifulSoup

url = 'find the link above'

res = requests.get(url,headers={"User-Agent":"Mozilla/5.0"})
soup = BeautifulSoup(res.text,'lxml')
for items in soup.select(".gll-MarketGroup_Wrapper"):
    data = [item.text for item in items.select("[class$='_NameContainer']")]
    print(data)

如何使用请求仅从该网站获取名称？

Answer 1

我尝试使用网站上的一个奇怪的 api 端点，我必须使用正则表达式来解析它。但是，如果您过于频繁地请求端点，有时需要多次尝试。如果您找不到更好的方法来解决此问题，那么这应该可以正常工作。抱歉，这不是 100%

代码

import requests
from bs4 import BeautifulSoup
import re

def attempt_get_api(attempt):
    headers = {'User-Agent': 'Mozilla/5.0'}
    url = 'https://www.bet365.com.au/SportsBook.API/web?zid=0&pd=%23AC%23B151%23C1%23D50%23E2%23F163%23&cid=13&ctid=13'
    r = requests.get(url, headers=headers, allow_redirects=False)
    text = r.text
    #print(text)
    games = re.finditer(r'NA=(.*?);', text)
    print('Attempt = ' + str(attempt))

    activated=False
    for game in games:
        activated=True
        if 'v' in game.group():
            print(game.group(1))

    if not activated:
        print(r.text)
        if attempt > 5:
            print("Too many attempts, quitting.")
            quit()
        attempt_get_api(attempt+1)

attempt_get_api(1)

输出

Attempt = 1
Trap v Cure
herO v TY
Maru v Solar
Stats v Rogue
Classic v RagnaroK
Dear v Zest
Overwatch - OWL Play-Ins
Philadelphia Fusion v Shanghai Dragons
LNG Academy v SDX Gaming
Gama Dream v Dominus Esports.Y
Team WE Academy v TS Gaming
D7G Esports Club v Invictus Gaming Young
Oh My Dream v Young Miracles
LinGan e-Sports v Bilibili Gaming Junior
Scorpio Game v Suning Gaming-S
Victorious Gaming v FunPlus Phoenix Blaze
87 v Top Esports Challenger
Rogue Warriors Shark v Legend Esport Gaming
Neon Esports v Sunrise
Team Aspirations v Sunrise
Look For Sponsors v Five Brothers
Chiefs Esports Club v MAMMOTH
Overwatch - Contenders
Gen.G esports v Element Mystic
O2 Blast v RunAway
Mindfreak v ORDER
Griffin v SK Telecom T1
J Team v ahq e-Sports club
FunPlus Phoenix v Bilibili Gaming
Top Esports v Royal Never Give Up
Lowkey Esports v Dashing Buffalo
GAM Esports v Team Flash
Lucky Future v Bilibili Gaming
SN Gaming v Team WE
RNGU v Invictus Gaming
Titanar Gaming v LP Esports
G2 Esports v AVANGAR
MiBR v Natus Vincere
FaZe v Renegades
Team Vitality v mousesports
Astralis v CR4ZY
Team Liquid v North
Moops v Brute
eSuba v eXtatus
Royal Youth v SuperMassive
G2 Esports v Fnatic
All Knights v Isurus Gaming
Crest Gaming Act v V3 Esports
Flash v Snow
Fnatic Rising v Excel UK
Vodafone Giants v Origen BCN
KINGZONE DragonX v Afreeca Freecs
MAD Team v G-Rex
Flash Wolves v Hong Kong Attitude
Clutch Gaming v FlyQuest
Flamengo v INTZ
Origen v Splyce