algorithm - Pandas:求解时间序列数据集的最高值阈值

Question

Givens: 我有一组一天的时间序列数据，比如 96 个值。我有一个累积值，比如在给定时间段内有 101 个单位。

问题:我需要找到阈值 X，其中所有高于该阈值的值总和为给定的累积值 101。请参见下图的视觉效果:

• X 值(黑线)是所需的阈值
• 101(曲线下的红色区域)是给定的累计值
• 蓝线是时间序列数据

约束:我必须多次执行此计算(一年中的每一天)，因此最好避免迭代，但不是必需的。

示例数据:

DateTime    Usage_KWH
1/1/2015 0:15   10.32
1/1/2015 0:30   10.56
1/1/2015 0:45   9.84
1/1/2015 1:00   9.36
1/1/2015 1:15   10.32
1/1/2015 1:30   9.6
1/1/2015 1:45   9.6
1/1/2015 2:00   10.32
1/1/2015 2:15   9.84
1/1/2015 2:30   9.6
1/1/2015 2:45   10.08
1/1/2015 3:00   9.36
1/1/2015 3:15   9.84
1/1/2015 3:30   10.32
1/1/2015 3:45   9.84
1/1/2015 4:00   9.84
1/1/2015 4:15   10.08
1/1/2015 4:30   9.6
1/1/2015 4:45   9.6
1/1/2015 5:00   10.8
1/1/2015 5:15   9.6
1/1/2015 5:30   9.84
1/1/2015 5:45   14.76
1/1/2015 6:00   14.4
1/1/2015 6:15   14.76
1/1/2015 6:30   15.12
1/1/2015 6:45   14.4
1/1/2015 7:00   14.4
1/1/2015 7:15   14.04
1/1/2015 7:30   12.96
1/1/2015 7:45   14.04
1/1/2015 8:00   12.6
1/1/2015 8:15   12.96
1/1/2015 8:30   14.04
1/1/2015 8:45   12.96
1/1/2015 9:00   17.28
1/1/2015 9:15   17.28
1/1/2015 9:30   17.76
1/1/2015 9:45   17.28
1/1/2015 10:00  17.76
1/1/2015 10:15  16.8
1/1/2015 10:30  17.28
1/1/2015 10:45  19.68
1/1/2015 11:00  17.28
1/1/2015 11:15  16.8
1/1/2015 11:30  16.8
1/1/2015 11:45  17.28
1/1/2015 12:00  16.8
1/1/2015 12:15  17.28
1/1/2015 12:30  17.28
1/1/2015 12:45  16.8
1/1/2015 13:00  17.28
1/1/2015 13:15  16.8
1/1/2015 13:30  16.8
1/1/2015 13:45  17.28
1/1/2015 14:00  25.92
1/1/2015 14:15  25.2
1/1/2015 14:30  25.2
1/1/2015 14:45  25.2
1/1/2015 15:00  25.2
1/1/2015 15:15  25.92
1/1/2015 15:30  25.2
1/1/2015 15:45  25.92
1/1/2015 16:00  25.92
1/1/2015 16:15  23.76
1/1/2015 16:30  23.76
1/1/2015 16:45  23.76
1/1/2015 17:00  24.48
1/1/2015 17:15  25.92
1/1/2015 17:30  8.88
1/1/2015 17:45  9.12
1/1/2015 18:00  8.88
1/1/2015 18:15  9.6
1/1/2015 18:30  8.88
1/1/2015 18:45  9.12
1/1/2015 19:00  9.12
1/1/2015 19:15  9.6
1/1/2015 19:30  9.12
1/1/2015 19:45  8.88
1/1/2015 20:00  9.12
1/1/2015 20:15  9.36
1/1/2015 20:30  9.12
1/1/2015 20:45  8.88
1/1/2015 21:00  6
1/1/2015 21:15  6
1/1/2015 21:30  6
1/1/2015 21:45  4
1/1/2015 22:00  5
1/1/2015 22:15  6
1/1/2015 22:30  7
1/1/2015 22:45  5
1/1/2015 23:00  7
1/1/2015 23:15  4
1/1/2015 23:30  6
1/1/2015 23:45  5

我糟糕的迭代代码:

time_series_df = pd.DataFrame(time_series_list)

#Iterative approach taking 10 steps
for x in (time_series_df.max, time_series_df.min, -(time_series_df.max)/10):
    #Getting values above an arbitrary threshold
    temp = time_series_df.query('Usage_KWH > @x')
    #If the difference above threshold and aggregate sum for the day are less than given cumulative value then try again
    if time_series_df.sum - temp < 101:
        final_threshold = temp
#print the highest value that did not exceed 101
print('final answer', final_threshold)

额外:我尝试使用 clip_upper、rank、cumsum、quantile 和 nlargest 的变体。我正在使用 Pandas 0.18

Answer 1

此处的技巧是对数据进行排序。这是一种方法。可能会提高速度!

df2           = df.sort_values(['Usage_KWH'], ascending=[False]).reset_index()
df2['KWHcum'] = df2['Usage_KWH'].cumsum()/  (df2.index+1) 
df2["dif"]    = np.round( df2['KWHcum'] - df2['Usage_KWH'], 3)*(df2.index+1)
df2

#        index DateTime  Usage_KWH     KWHcum       dif
# 0   1/1/2015    14:00      25.92  25.920000    0.0000
# 1   1/1/2015    16:00      25.92  25.920000    0.0000
# 2   1/1/2015    15:45      25.92  25.920000    0.0000
# 3   1/1/2015    15:15      25.92  25.920000    0.0000
# 4   1/1/2015    17:15      25.92  25.920000    0.0000
# 5   1/1/2015    14:45      25.20  25.800000    3.6000
# 6   1/1/2015    14:15      25.20  25.714286    3.6001
# 7   1/1/2015    15:30      25.20  25.650000    3.6000
# 8   1/1/2015    14:30      25.20  25.600000    3.6000
# 9   1/1/2015    15:00      25.20  25.560000    3.6000
# 10  1/1/2015    17:00      24.48  25.461818   10.7998
# 11  1/1/2015    16:30      23.76  25.320000   18.7200
# 12  1/1/2015    16:45      23.76  25.200000   18.7200
# 13  1/1/2015    16:15      23.76  25.097143   18.7194
# 14  1/1/2015    10:45      19.68  24.736000   75.8400
# 15  1/1/2015     9:30      17.76  24.300000  104.6400
# 16  1/1/2015    10:00      17.76  23.915294  104.6401
# 17  1/1/2015    11:00      17.28  23.546667  112.8006
# 18  1/1/2015     9:45      17.28  23.216842  112.7992
# 19  1/1/2015    12:30      17.28  22.920000  112.8000
# 20  1/1/2015    10:30      17.28  22.651429  112.7994
# 21  1/1/2015    12:15      17.28  22.407273  112.8006
# 22  1/1/2015    13:00      17.28  22.184348  112.7989
# 23  1/1/2015    11:45      17.28  21.980000  112.8000
# 24  1/1/2015    13:45      17.28  21.792000  112.8000
# 25  1/1/2015     9:00      17.28  21.618462  112.8010
# 26  1/1/2015     9:15      17.28  21.457778  112.8006
# 27  1/1/2015    11:15      16.80  21.291429  125.7592
# 28  1/1/2015    11:30      16.80  21.136552  125.7614
# 29  1/1/2015    10:15      16.80  20.992000  125.7600

 df2           = df2[df2['dif'] < 101]
 print df2['Usage_KWH'].tail(1)    
  #  14    19.68
  # Name: Usage_KWH, dtype: float64

df2           = df2[df2['dif'] < 141]
print df2['Usage_KWH'].tail(1)    
#33    16.8
#Name: Usage_KWH, dtype: float64