python - 循环速度优化

Question

我正在尝试预测体育比赛的结果，因此希望以可以训练模型的方式转换我的数据框。目前，我正在使用 for 循环来遍历所有玩过的游戏，选择游戏的两名玩家并检查他们在实际游戏发生之前如何执行 x 游戏。之后，我想取这些球员之前比赛的统计数据的平均值，并将它们连接在一起。最后，我添加了实际游戏的真实结果，以便我可以根据真实结果训练模型。

现在我遇到了一些速度性能问题，我当前的代码大约需要 9 分钟才能完成 20000 个游戏(大约 200 个变量)。我已经设法从 20 分钟缩短到 9 分钟。

我首先将每个游戏添加到一个数据帧中，后来我将其更改为将每个单独的数据帧添加到一个列表中，并最后在该列表中创建一个大数据帧。我还添加了 if 语句，以确保如果玩家没有玩至少 x 场游戏，循环就会继续。

我预计结果会比 9 分钟快得多。我认为它可以快得多。

希望大家能帮帮我!

import pandas as pd
import numpy as np
import random
import string

letters = list(string.ascii_lowercase)
datelist = pd.date_range(start='1/1/2017', end='1/1/2019')

data = pd.DataFrame({'Date':np.random.choice(datelist,5000),
                     'League': np.random.choice(['LeagueA','LeagueB'], 5000),
                     'Home_player':np.random.choice(letters, 5000),
                     'Away_player':np.random.choice(letters, 5000),
                     'Home_strikes':np.random.randint(1,20,5000),
                     'Home_kicks':np.random.randint(1,20,5000),
                     'Away_strikes':np.random.randint(1,20,5000),
                     'Away_kicks':np.random.randint(1,20,5000),
                     'Winner':np.random.randint(0,2,5000)})

leagues = list(data['League'].unique())
home_columns = [col for col in data if col.startswith('Home')]
away_columns = [col for col in data if col.startswith('Away')]

# Determine to how many last x games to take statistics
total_games = 5 
final_df = []

# Make subframe of league
for league in leagues:
    league_data = data[data.League == league]
    league_data = league_data.sort_values(by='Date').reset_index(drop=True)
    # Pick the last game
    league_data = league_data.head(500)
    for i in range(0,len(league_data)):
        if i < 1:
            league_copy = league_data.sort_values(by='Date').reset_index(drop=True)
        else:
            league_copy = league_data[:-i].reset_index(drop=True)

        # Loop back from the last game
        last_game = league_copy.iloc[-1:].reset_index(drop=True)         

        # Take home and away player
        Home_player = last_game.loc[0,"Home_player"] # Pick home team
        Away_player = last_game.loc[0,'Away_player'] # pick away team

        # # Remove last row so current game is not picked
        df = league_copy[:-1] 

        # Now check the statistics of the games befóre this game was played
        Home = df[df.Home_player == Home_player].tail(total_games) # Pick data from home team           

        # If the player did not play at least x number of games, then continue
        if len(Home) < total_games:
            continue
        else:
            Home = Home[home_columns].reset_index(drop=True) # Pick all columnnames that start with "Home"


        # Do the same for the away team
        Away = df[df.Away_player == Away_player].tail(total_games) # Pick data from home team           

        if len(Away) < total_games:
            continue
        else:
            Away = Away[away_columns].reset_index(drop=True) # Pick all columnnames that start with "Home"


        # Now concat home and away player data
        Home_away = pd.concat([Home, Away], axis=1)
        Home_away.drop(['Away_player','Home_player'],inplace=True,axis=1)

        # Take the mean of all columns
        Home_away = pd.DataFrame(Home_away.mean().to_dict(),index=[0])

        # Now again add home team and away team to dataframe
        Home_away["Home_player"] = Home_player
        Home_away["Away_player"] = Away_player

        winner = last_game.loc[0,"Winner"]
        date = last_game.loc[0,"Date"]
        Home_away['Winner'] = winner
        Home_away['Date'] = date

        final_df.append(Home_away)
final_df = pd.concat(final_df, axis=0)
final_df = final_df[['Date','Home_player','Away_player','Home_kicks','Away_kicks','Home_strikes','Away_strikes','Winner']]

Answer 1

这并不能回答您的问题，但您可以利用包 line_profiler 来查找代码中缓慢的部分。

资源: http://gouthamanbalaraman.com/blog/profiling-python-jupyter-notebooks.html

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     2         1         35.0     35.0      0.0      letters = list(string.ascii_lowercase)
     3         1      11052.0  11052.0      0.0      datelist = pd.date_range(start='1/1/2017', end='1/1/2019')
     4                                           
     5         1       3483.0   3483.0      0.0      data = pd.DataFrame({'Date':np.random.choice(datelist,5000),
     6         1       1464.0   1464.0      0.0                           'League': np.random.choice(['LeagueA','LeagueB'], 5000),
     7         1       2532.0   2532.0      0.0                           'Home_player':np.random.choice(letters, 5000),
     8         1       1019.0   1019.0      0.0                           'Away_player':np.random.choice(letters, 5000),
     9         1        693.0    693.0      0.0                           'Home_strikes':np.random.randint(1,20,5000),
    10         1        682.0    682.0      0.0                           'Home_kicks':np.random.randint(1,20,5000),
    11         1        682.0    682.0      0.0                           'Away_strikes':np.random.randint(1,20,5000),
    12         1        731.0    731.0      0.0                           'Away_kicks':np.random.randint(1,20,5000),
    13         1      40409.0  40409.0      0.0                           'Winner':np.random.randint(0,2,5000)})
    14                                           
    15         1       6560.0   6560.0      0.0      leagues = list(data['League'].unique())
    16         1        439.0    439.0      0.0      home_columns = [col for col in data if col.startswith('Home')]
    17         1        282.0    282.0      0.0      away_columns = [col for col in data if col.startswith('Away')]
    18                                           
    19                                               # Determine to how many last x games to take statistics
    20         1         11.0     11.0      0.0      total_games = 5 
    21         1         12.0     12.0      0.0      final_df = []
    22                                           
    23                                               # Make subframe of league
    24         3         38.0     12.7      0.0      for league in leagues:
    25                                           
    26         2      34381.0  17190.5      0.0          league_data = data[data.League == league]
    27         2      30815.0  15407.5      0.0          league_data = league_data.sort_values(by='Date').reset_index(drop=True)
    28                                                   # Pick the last game
    29         2       5045.0   2522.5      0.0          league_data = league_data.head(500)
    30      1002      14202.0     14.2      0.0          for i in range(0,len(league_data)):
    31      1000      11943.0     11.9      0.0              if i < 1:
    32         2      28407.0  14203.5      0.0                  league_copy = league_data.sort_values(by='Date').reset_index(drop=True)
    33                                                       else:
    34       998    5305364.0   5316.0      4.2                  league_copy = league_data[:-i].reset_index(drop=True)
    35                                           
    36                                                       # Loop back from the last game
    37      1000    4945240.0   4945.2      3.9              last_game = league_copy.iloc[-1:].reset_index(drop=True)         
    38                                           
    39                                                       # Take home and away player
    40      1000    1504055.0   1504.1      1.2              Home_player = last_game.loc[0,"Home_player"] # Pick home team
    41      1000     899081.0    899.1      0.7              Away_player = last_game.loc[0,'Away_player'] # pick away team
    42                                           
    43                                                       # # Remove last row so current game is not picked
    44      1000    2539351.0   2539.4      2.0              df = league_copy[:-1] 
    45                                           
    46                                                       # Now check the statistics of the games befóre this game was played
    47      1000   16428854.0  16428.9     13.0              Home = df[df.Home_player == Home_player].tail(total_games) # Pick data from home team           
    48                                           
    49                                                       # If the player did not play at least x number of games, then continue
    50      1000      49133.0     49.1      0.0              if len(Home) < total_games:
    51       260       2867.0     11.0      0.0                  continue
    52                                                       else:
    53       740   12968016.0  17524.3     10.2                  Home = Home[home_columns].reset_index(drop=True) # Pick all columnnames that start with "Home"
    54                                           
    55                                           
    56                                                       # Do the same for the away team
    57       740   12007650.0  16226.6      9.5              Away = df[df.Away_player == Away_player].tail(total_games) # Pick data from home team           
    58                                           
    59       740      33357.0     45.1      0.0              if len(Away) < total_games:
    60        64        825.0     12.9      0.0                  continue
    61                                                       else:
    62       676   11598741.0  17157.9      9.1                  Away = Away[away_columns].reset_index(drop=True) # Pick all columnnames that start with "Home"
    63                                           
    64                                           
    65                                                       # Now concat home and away player data
    66       676    5114022.0   7565.1      4.0              Home_away = pd.concat([Home, Away], axis=1)
    67       676    9702001.0  14352.1      7.6              Home_away.drop(['Away_player','Home_player'],inplace=True,axis=1)
    68                                           
    69                                                       # Take the mean of all columns
    70       676   12171184.0  18004.7      9.6              Home_away = pd.DataFrame(Home_away.mean().to_dict(),index=[0])
    71                                           
    72                                                       # Now again add home team and away team to dataframe
    73       676    5112558.0   7563.0      4.0              Home_away["Home_player"] = Home_player
    74       676    4880017.0   7219.0      3.8              Home_away["Away_player"] = Away_player
    75                                           
    76       676     791718.0   1171.2      0.6              winner = last_game.loc[0,"Winner"]
    77       676     696925.0   1031.0      0.5              date = last_game.loc[0,"Date"]
    78       676    5142111.0   7606.7      4.1              Home_away['Winner'] = winner
    79       676    9630466.0  14246.3      7.6              Home_away['Date'] = date
    80                                           
    81       676      16125.0     23.9      0.0              final_df.append(Home_away)
    82         1    5088063.0 5088063.0      4.0      final_df = pd.concat(final_df, axis=0)
    83         1      18424.0  18424.0      0.0      final_df = final_df[['Date','Home_player','Away_player','Home_kicks','Away_kicks','Home_strikes','Away_strikes','Winner']]