python - Beautifulsoup 和 Panda - 帮助修改多页代码

Question

所以我编写了这段代码来从网站上抓取数据。

import requests
from bs4 import BeautifulSoup
import re

page = requests.get('https://sofifa.com/shortlist/32931')
soup = BeautifulSoup(page.text, 'html.parser')
dados = soup.find_all('a', href=re.compile("/player/"))
capa = soup.find('article')
capa1 = capa.find('div' , {'class': 'card card-border mb-2 fixed-width'})
time = capa1.find('div' , {'class': 'card-title h5'}).string

records = []
for nomes in dados:
    nome = nomes.string
    records.append((nome))

import pandas as pd
from openpyxl import load_workbook

book = load_workbook('Tabela Pipoco 2019.xlsx')
writer = pd.ExcelWriter('Tabela Pipoco 2019.xlsx', engine='openpyxl')
writer.book = book
writer.sheets = dict((ws.title, ws) for ws in book.worksheets)
df = pd.DataFrame(records, columns=[time])
df.to_excel(writer, "Times", index=False, encoding='utf-8', startcol=0)
writer.save()

问题是我想用 10 个不同的页面来制作这个!由于我缺乏知识，我在jupyter笔记本上编写了10个不同的代码并全部运行。

代码中唯一改变的是这两行:

page = requests.get('https://sofifa.com/shortlist/32931')

和

df.to_excel(writer, "Times", index=False, encoding='utf-8', startcol=0)

因此，在第一行，我们将用特定数字(32931、32882、32589)更改结尾数字，依此类推。

在第二行，更改的是 startcol=，即(0 表示第一页，3 表示第二页，6 表示第三页，依此类推)

如何将内容放在单个代码上？

谢谢大家

Answer 1

When you’ve written the same code 3 times, write a function (c) David Robinson

您可以使用带有两个参数的函数包装代码:url 和 startcol，然后针对不同的输入循环调用此函数。例如:

# Define a function
def your_func_to_avoid_writing_the_same_code_ten_times(url, col):
    page = requests.get(url)  
    # ... missed code for better formatting
    df.to_excel(writer, "Times", index=False, encoding='utf-8', startcol=col)
    writer.save()

url_list = ['https://sofifa.com/shortlist/32931', 'https://sofifa.com/shortlist/32882'] # And so on

# Initialize columns counter
col = 0
# Call the function and update the column
for i in range(len(url_list)): 
    print(your_func_to_avoid_writing_the_same_code_ten_times(url_list[i], col))
    col += 3