python-3.x - np.argsort 和 pd.nsmallest 给出不同的结果

Question

使用以下示例数据和代码:

import pandas as pd
import numpy as np

np.random.seed(2021)
dates = pd.date_range('20130226', periods=90)
df = pd.DataFrame(np.random.uniform(0, 10, size=(90, 4)), index=dates, columns=['A_values', 'B_values', 'C_values', 'target'])

# function to calculate mape
def mape(y_true, y_pred):
    y_pred = np.array(y_pred)
    return np.mean(np.abs(y_true - y_pred) / np.clip(np.abs(y_true), 1, np.inf),
                   axis=0)*100

preds = df.columns[df.columns.str.endswith('_values')]
k = 2
print(df)

输出:

            A_values  B_values  C_values    target
2013-02-26  6.059783  7.333694  1.389472  3.126731
2013-02-27  9.972433  1.281624  1.789931  7.529254
2013-02-28  6.621605  7.843101  0.968944  0.585713
2013-03-01  9.623960  6.165574  0.866300  5.612724
2013-03-02  6.165247  9.638430  5.743043  3.711608
             ...       ...       ...       ...
2013-05-22  0.589729  6.479978  3.531450  6.872059
2013-05-23  6.279065  3.837670  8.853146  8.209883
2013-05-24  5.533017  5.241127  1.388056  5.355926
2013-05-25  1.596038  4.665995  2.406251  1.971875
2013-05-26  3.269001  1.787529  6.659690  7.545569

[90 rows x 4 columns]

我将计算 mape 并使用两种不同的方法为每个年/月组找到 2 个最低误差值:

方法一:

def grpProc(grp):
    err = mape(grp[preds], grp[['target']])
    print(err)
    sort_args = np.argsort(err, axis=1) < k
    cols = preds[sort_args]
    print(cols)
    print('-'*50)

df.groupby(pd.Grouper(freq='M')).apply(grpProc)

输出:

A_values     54.685258
B_values    212.458242
C_values    161.332752
dtype: float64
Index(['A_values', 'C_values'], dtype='object')
--------------------------------------------------
A_values     77.504315
B_values    128.986127
C_values    118.977186
dtype: float64
Index(['A_values', 'C_values'], dtype='object')
--------------------------------------------------
A_values    132.535352
B_values    150.886936
C_values     94.279492
dtype: float64
Index(['B_values', 'C_values'], dtype='object')
--------------------------------------------------
A_values    150.554314
B_values    114.113724
C_values     92.487276
dtype: float64
Index(['B_values', 'C_values'], dtype='object')
--------------------------------------------------

方法二:

def grpProc(grp):
    err = mape(grp[preds], grp[['target']])
    print(err)
    cols = err.nsmallest(k).index
    print(cols)
    print('-'*50)

df.groupby(pd.Grouper(freq='M')).apply(grpProc)

输出:

A_values     54.685258
B_values    212.458242
C_values    161.332752
dtype: float64
Index(['A_values', 'C_values'], dtype='object')
--------------------------------------------------
A_values     77.504315
B_values    128.986127
C_values    118.977186
dtype: float64
Index(['A_values', 'C_values'], dtype='object')
--------------------------------------------------
A_values    132.535352
B_values    150.886936
C_values     94.279492
dtype: float64
Index(['C_values', 'A_values'], dtype='object')
--------------------------------------------------
A_values    150.554314
B_values    114.113724
C_values     92.487276
dtype: float64
Index(['C_values', 'B_values'], dtype='object')
--------------------------------------------------

如您所见，方法 1 为第三组给出了错误的 2 个最低值，正确的应该是:['C_values', 'A_values']。

A_values    132.535352
B_values    150.886936
C_values     94.279492
dtype: float64
Index(['B_values', 'C_values'], dtype='object')

如果我们使用 np.argsort 而不是 pd.nsmallest，如何让它正确？谢谢。

编辑:

def grpProc(grp):
    err = mape(grp[preds], grp[['target']])
    print(err)
    # sort_args = np.argsort(err, axis=1) < k # incorrect result
    # sort_args = np.argsort(err, axis=1)[:k] # correct result and order of values
    # sort_args  = np.argsort(err).head(k) # correct result and order of values
    sort_args = np.argsort(np.argsort(err, axis=1)) < k # correct result but incorrect order of values
    cols = preds[sort_args]
    print(cols)
    print('-'*50)

df.groupby(pd.Grouper(freq='M')).apply(grpProc)

输出:

A_values     54.685258
B_values    212.458242
C_values    161.332752
dtype: float64
Index(['A_values', 'C_values'], dtype='object')
--------------------------------------------------
A_values     77.504315
B_values    128.986127
C_values    118.977186
dtype: float64
Index(['A_values', 'C_values'], dtype='object')
--------------------------------------------------
A_values    132.535352
B_values    150.886936
C_values     94.279492
dtype: float64
Index(['A_values', 'C_values'], dtype='object')
--------------------------------------------------
A_values    150.554314
B_values    114.113724
C_values     92.487276
dtype: float64
Index(['B_values', 'C_values'], dtype='object')
--------------------------------------------------

Answer 1

np.argsort 正在做位置重新索引

sort_args  = err.iloc[np.argsort(err)].head(2)

你还有 pandas argsort(与 numpy 相同)

err.iloc[err.argsort()].head(2)

更新

sort_args = np.argsort(np.argsort(err, axis=1)) < k