python - 在网页上应用过滤器时如何复制请求？

Question

我正在尝试在 premier league player stats 中应用过滤器时复制请求.我注意到 url 在为 2019/20 赛季应用过滤器时添加了组件“?co=1&se=274”

https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274

代替

https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats

但是在做的时候

requests.get('https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274')

并抓取内容，它被抓取就像没有应用过滤器一样。如何在网页上应用过滤器的地方发出请求？

通过深入研究，我了解到它受 CloudFront 保护，这意味着在发布请求之前所有查询参数都会被删除。有解决办法吗？

这是我抓取数据的方式:

from bs4 import BeautifulSoup as soup
import requests
from tqdm import tqdm
from pprint import pprint
players_url =['https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274']


# this is dict where we store all information:
players = {}
for i in tqdm(players_url):
    player_page = requests.get(i)
    cont = soup(player_page.content, 'lxml')
    time.sleep(2)
    data = dict((k.contents[0].strip(), v.get_text(strip=True)) for k, v in zip(cont.select('.topStat span.stat, .normalStat span.stat'), cont.select('.topStat span.stat > span, .normalStat span.stat > span')))
    clud_ele = cont.find('div', attrs={'class' : 'info'})
    club = {"Club" : clud_ele.get_text(strip=True)}
    position = {"Position": clud_ele.find_next('div', attrs={'class' : 'info'}).get_text(strip=True)}
    data.update(club)
    data.update(position)
    players[cont.select_one('.playerDetails .name').get_text(strip=True)] = data

pprint(players)

在输出中我可以清楚地看到没有应用过滤器，因为本赛季还没有 45 场比赛

{'Virgil van Dijk': {'Accurate long balls': '533',
                     'Aerial battles lost': '207',
                     'Aerial battles won': '589',
                     'Appearances': '122',
                     'Assists': '2',
                     'Big chances created': '11',
                     'Blocked shots': '23',
                     'Clean sheets': '45',

Answer 1

您可以通过复制当您尝试按季节过滤时完成的后台请求来解决这个问题。我使用 requests 库来获取所有玩家的统计数据。

这个过程主要涉及三个url，

1. 获取季节 ID 值。 (例如 274)

https://footballapi.pulselive.com/football/competitions/1/compseasons?page=0&pageSize=100

1. 获取球员姓名和 ids (例如姓名:Virgil van Dijk，ID:5140)

https://footballapi.pulselive.com/football/players

1. 使用玩家 ID (例如 5140)获取玩家统计数据

https://footballapi.pulselive.com/football/stats/player/

完整脚本

import requests
import json

response = requests.get('https://footballapi.pulselive.com/football/competitions/1/compseasons?page=0&pageSize=100').json() # request to obtain the id values and corresponding season 

id = int(response["content"][0]["id"]) # converts current season id which is a decimal point value to interger

players = {} # dictionary to store players data

playersAndStats = {} # dictionary to store player name and associated stats

numEntries = 100

page = 0

# loop to get player name and id
while True: 

  params = (
      ('pageSize', '100'),
      ('compSeasons', str(id)), 
      ('altIds', 'true'),
      ('page', str(page)),
      ('type', 'player'),
      ('id', '-1'),
      ('compSeasonId', str(id)),
  )

  response = requests.get('https://footballapi.pulselive.com/football/players',params=params).json()

  playersData = response["content"]

  for playerData in playersData:
    players[playerData["id"]] = playerData["name"]["display"]

  page += 1

  if page == response["pageInfo"]["numPages"]:
    break

print("Total no. of players :",len(players))

count = 0 
total = len(players)

# loop to get player stats 
for player in players:

  count += 1
  print(count,"/",total)

  params = (
      ('comps', '1'),
      ('compSeasons', str(id)), # setting season id to current season id
  )

  playerId = str(int(player))

  # gets the stat of the player using playerId 
  response = requests.get('https://footballapi.pulselive.com/football/stats/player/'+playerId,params=params).json()

  stats = response["stats"]

  # creating a stat dict for the player
  playersAndStats[players[player]] = {}

  # loop to store each stat associated with the player
  for stat in stats:
    playersAndStats[players[player]][stat["name"].replace("_"," ").capitalize()] = int(stat["value"])

# to store data to a json file 
f = open("data.json","w")

# pretty prints and writes the same to the json file 
f.write(json.dumps(playersAndStats,indent=4, sort_keys=True))
f.close()

print("Saved to data.json")

示例输出

"Aaron Connolly": {
        "Accurate back zone pass": 1,
        "Accurate fwd zone pass": 1,
        "Accurate pass": 2,
        "Aerial lost": 1,
        "Appearances": 1,
        "Attempts conceded ibox": 2,
        "Attempts conceded obox": 1,
        "Backward pass": 1,
        "Ball recovery": 1,
        "Duel lost": 1,
        "Duel won": 0,
        "Final third entries": 1,
        "Goals conceded": 1,
        "Goals conceded ibox": 1,
        "Losses": 1,
        "Mins played": 24,
        "Open play pass": 1,
        "Poss won mid 3rd": 1,
        "Rightside pass": 1,
        "Successful final third passes": 1,
        "Successful open play pass": 1,
        "Total back zone pass": 1,
        "Total final third passes": 1,
        "Total fwd zone pass": 1,
        "Total offside": 2,
        "Total pass": 2,
        "Total sub on": 1,
        "Total tackle": 0,
        "Touches": 3,
        "Won tackle": 0
    }

data.json 文件包含所有玩家的数据。

The stats associated with each player varies according to their playing position.