Python Pandas - 查找和分组异常值

Question

我有一个带有多列的 pd 数据框(为了便于阅读而进行了简化)-每一行都包含一个 id (uuid)、索引和一个或多个功能:

uuid         index           Atrium       Ventricle 
di-abc           0            20.73           26.21
di-abc           1            18.92           25.14
di-efg           7            19.02            0.30 
di-efg           9             1.23            0.51
di-efg           6            21.24           26.02
di-hjk           3            22.10           25.16
di-hjk           6            19.16           25.57

我想要:

查找每个特征的异常值(即“心房”和“心室”列)

按以下格式导出异常值:

outliers = {
    'Atrium' : [
         {'uuid' : 'di-efg', 'index' : 9, 'value' : 1.23},
     ],
     'Ventricle' : [
         {'uuid' : 'di-efg', 'index' : 7, 'value' : 0.30},
         {'uuid' : 'di-efg', 'index' : 9, 'value' : 0.53},
    ]
}

注意事项(处理此问题的奖励积分):

特征(以及列)的数量是动态的

单行可以包含零个、一个、两个或多个异常值

我在双 for 循环之外的两个步骤都遇到了困难。
有没有一种有效的方法来计算这个数据帧的异常值？
这是一种有效但效率不高的方法来捕获我要完成的任务:

# initialize variables:
outliers = {}
features = ['Atrium', 'Ventricle']

# iterate over each feature:
for feature in features:

    # set feature on outlier to empty list:
    outliers[feature] = []
            
    # create a dataframe of outliers for that specific feature:
    outlier_df = df[df[feature] > (df[feature].mean() + df[feature].std())] # can mess with this if needed
    outlier_df = outlier_df[['dicom', 'frame', 'index', feature]]
            
    # iterate through the data frame and find the uuid, index, and feature:
    for index, row in outlier_df.iterrows():

        # append each outlier to the outlier dictionary:
        outliers[feature].append({
             'uuid' : row['uuid'],
             'index' : row['index'],
             'value' : row[feature],
         })

Answer 1

这是通过定义一个函数来解决该问题的一种方法，该函数将输入参数作为列名并以所需格式返回当前列中的所有异常值:

def detect_outliers(col):
    # Define your outlier detection condition here
    mask = (df[col] - df[col].mean()).abs() > df[col].std()
    return df.loc[mask, ['uuid', 'index', col]]\
             .rename(columns={col: 'value'}).to_dict('records')

outliers = {col: detect_outliers(col) for col in features}

替代方法 多一点参与 Pandas 操作，如 stacking , grouping和 aggregation :

# Select only feature columns
feature_df = df.set_index(['uuid', 'index'])[features]

# Define your outlier detection condition
mask = (feature_df - feature_df.mean()).abs() > feature_df.std()

# Prepare outlier dataframe
outlier_df = feature_df[mask].stack().reset_index(level=[0, 1], name='value')
outlier_df['records'] = outlier_df.to_dict('r')

# Get the outliers in the desired format
outliers = outlier_df.groupby(level=0).agg(list)['records'].to_dict()

>>> outliers

{
    'Atrium': [
        {'uuid': 'di-efg', 'index': 9, 'value': 1.23}
    ],
    'Ventricle': [
        {'uuid': 'di-efg', 'index': 7, 'value': 0.3},
        {'uuid': 'di-efg', 'index': 9, 'value': 0.51}
    ]
}