python - 使用 pandas 时间序列的过去 n 小时的变化率

Question

我想将列添加到时间索引的 pandas DataFrame 中，其中包含每个现有列在过去 n 小时内的变化率。我已经用下面的代码完成了这个，但是，它对我的​​需要来说太慢了(可能是因为遍历了每列的每个索引？)。

有没有(计算上)更快的方法来做到这一点？

roc_hours = 12
tol = 1e-10 
for c in ts.columns:
    c_roc = c + ' +++ RoC ' + str(roc_hours) + 'h' 
    ts[c_roc] = np.nan
    for i in ts.index[np.isfinite(ts[c])]:
        df = ts[c][i - np.timedelta64(roc_hours, 'h'):i]
        X = (df.index.values - df.index.values.min()).astype('Int64')*2.77778e-13 #hours back
        Y = df.values
        if Y.std() > tol and X.shape[0] > 1:
            fit = np.polyfit(X,Y,1)
            ts[c_roc][i] = fit[0]
        else:
            ts[c_roc][i] = 0

编辑 输入数据帧 ts 是不规则采样的，可以包含 NaN。输入ts的前几行:

+---------------------+-------------------+------+------+--------------------+-------------------+------------------+
|         WCT         |         a         |  b   |  c   |         d          |         e         |        f         |
+---------------------+-------------------+------+------+--------------------+-------------------+------------------+
| 2011-09-04 20:00:00 |                   |      |      |                    |                   |                  |
| 2011-09-04 21:00:00 |                   |      |      |                    |                   |                  |
| 2011-09-04 22:00:00 |                   |      |      |                    |                   |                  |
| 2011-09-04 23:00:00 |                   |      |      |                    |                   |                  |
| 2011-09-05 02:00:00 |        93.0       | 97.0 | 20.0 |       209.0        |        85.0       |       98.0       |
| 2011-09-05 03:00:00 | 74.14285714285714 | 97.0 | 20.0 | 194.14285714285717 | 74.42857142857143 |       98.0       |
| 2011-09-05 04:00:00 |        67.5       | 98.5 | 20.0 |       176.0        |        75.0       |       98.0       |
| 2011-09-05 05:00:00 |        72.0       | 98.5 | 20.0 |       176.0        |        75.0       |       98.0       |
| 2011-09-05 07:00:00 |        80.0       | 93.0 | 19.0 |       186.0        |        71.0       |       97.0       |
| 2011-09-05 08:00:00 |        80.0       | 93.0 | 19.0 |       186.0        |        71.0       |       97.0       |
| 2011-09-05 09:00:00 |        78.5       | 98.0 | 19.0 |       186.0        |        71.0       |       97.0       |
| 2011-09-05 10:00:00 |        73.0       | 98.0 | 19.0 |       186.0        |        71.0       |       97.0       |
| 2011-09-05 11:00:00 |        77.0       | 98.0 | 18.0 |       175.0        |        87.0       | 97.0999984741211 |
| 2011-09-05 12:00:00 |        78.0       | 98.0 | 19.0 |       163.0        |        57.0       | 98.4000015258789 |
| 2011-09-05 15:00:00 |        78.0       | 98.0 | 19.0 |       163.0        |        57.0       | 98.4000015258789 |
+---------------------+-------------------+------+------+--------------------+-------------------+------------------+

编辑2

分析后，瓶颈在切片步骤:df = ts[c][i - np.timedelta64(roc_hours, 'h'):i]。此行提取在 now-roc_hours 和 now 之间加盖时间戳的观察结果。这是非常方便的语法，但占用了大量的计算时间。

Answer 1

在我的数据集上工作，还没有检查你的数据集:

import pandas as pd
from numpy import polyfit
from matplotlib import style
style.use('ggplot')

# ... acquire a dataframe named *water* with a column *value*

WINDOW = 10
ax=water.value.plot()
roll = pd.rolling_mean(water.value, WINDOW)
roll.plot(ax=ax)

def lintrend(df):
    df = df.tolist()
    m, b = polyfit(range(len(df)), df,1)
    return m

linny = pd.rolling_apply(water.value, WINDOW, lintrend)

linny.plot(ax=ax)

在 rolling_apply 将其转换为 numpy.ndarray 之后将 numpy.ndarray 转换为列表似乎并不优雅。有什么建议吗？