python - 如何有效地重新采样 DatetimeIndex

Question

Pandas 在系列/数据帧上有一个 resample 方法，但似乎没有办法单独对 DatetimeIndex 进行重采样？

具体来说，我有一个每日 Datetimeindex，其中可能缺少日期，我想以每小时的频率对其重新采样，但只包括原始每日索引中的天数。

有没有比我下面的尝试更好的方法？

In [56]: daily_index = pd.period_range('01-Jan-2017', '31-Jan-2017', freq='B').asfreq('D')

In [57]: daily_index
Out[57]: 
PeriodIndex(['2017-01-02', '2017-01-03', '2017-01-04', '2017-01-05',
             '2017-01-06', '2017-01-09', '2017-01-10', '2017-01-11',
             '2017-01-12', '2017-01-13', '2017-01-16', '2017-01-17',
             '2017-01-18', '2017-01-19', '2017-01-20', '2017-01-23',
             '2017-01-24', '2017-01-25', '2017-01-26', '2017-01-27',
             '2017-01-30', '2017-01-31'],
            dtype='int64', freq='D')

In [58]: daily_index.shape
Out[58]: (22,)

In [59]: hourly_index = pd.DatetimeIndex([]).union_many(
    ...:     pd.date_range(day.to_timestamp('H','S'), day.to_timestamp('H','E'), freq='H')
    ...:     for day in daily_index
    ...: )

In [60]: hourly_index
Out[60]: 
DatetimeIndex(['2017-01-02 00:00:00', '2017-01-02 01:00:00',
               '2017-01-02 02:00:00', '2017-01-02 03:00:00',
               '2017-01-02 04:00:00', '2017-01-02 05:00:00',
               '2017-01-02 06:00:00', '2017-01-02 07:00:00',
               '2017-01-02 08:00:00', '2017-01-02 09:00:00',
               ...
               '2017-01-31 14:00:00', '2017-01-31 15:00:00',
               '2017-01-31 16:00:00', '2017-01-31 17:00:00',
               '2017-01-31 18:00:00', '2017-01-31 19:00:00',
               '2017-01-31 20:00:00', '2017-01-31 21:00:00',
               '2017-01-31 22:00:00', '2017-01-31 23:00:00'],
              dtype='datetime64[ns]', length=528, freq=None)

In [61]: 22*24
Out[61]: 528

In [62]: %%timeit
    ...: hourly_index = pd.DatetimeIndex([]).union_many(
    ...:     pd.date_range(day.to_timestamp('H','S'), day.to_timestamp('H','E'), freq='H')
    ...:     for day in daily_index
    ...: )
100 loops, best of 3: 13.7 ms per loop

更新:

我对@NTAWolf 的答案进行了细微改动，它具有相似的性能，但不会对输入日期重新排序以防它们未排序

def resample_index(index, freq):
    """Resamples each day in the daily `index` to the specified `freq`.

    Parameters
    ----------
    index : pd.DatetimeIndex
        The daily-frequency index to resample
    freq : str
        A pandas frequency string which should be higher than daily

    Returns
    -------
    pd.DatetimeIndex
        The resampled index

    """
    assert isinstance(index, pd.DatetimeIndex)
    start_date = index.min()
    end_date = index.max() + pd.DateOffset(days=1)
    resampled_index = pd.date_range(start_date, end_date, freq=freq)[:-1]
    series = pd.Series(resampled_index, resampled_index.floor('D'))
    return pd.DatetimeIndex(series.loc[index].values)

In [184]: %%timeit
     ...: hourly_index3 = pd.date_range(daily_index.start_time.min(), 
     ...:                               daily_index.end_time.max() + 1, 
     ...:                               normalize=True, freq='H')
     ...: hourly_index3 = hourly_index3[hourly_index3.floor('D').isin(daily_index.start_time)]
100 loops, best of 3: 2.97 ms per loop

In [185]: %timeit resample_index(daily_index.to_timestamp('D','S'), freq='H')
100 loops, best of 3: 2.93 ms per loop

Answer 1

|             Method              |  Time   | Relative ||---------------------------------|---------|----------|| OP's updated approach           | 1.31 ms |  17.6 %  || Generate daterange, np.in1d     | 1.75 ms |  23.5 %  || Generate daterange, Series.isin | 1.90 ms |  25.5 %  || Resample with dummy Series      | 4.37 ms |  58.7 %  || OP's initial approach           | 7.45 ms | 100.0 %  |

Update 2: Generate daterange, np.in1d

Again, @IanS inspired more optimisation! This is a little less readable, but a bit faster:

%%timeit -r 10
hourly_index4 = pd.date_range(daily_index.start_time.min(), 
                              daily_index.end_time.max() + pd.DateOffset(days=1), 
                              normalize=True, freq='H')
overlap = np.in1d(np.array(hourly_index4.values, dtype='datetime64[D]'),
                  np.array(daily_index.start_time.values, dtype='datetime64[D]'))
hourly_index4 = hourly_index4[overlap]

1000 loops, best of 10: 1.75 ms per loop

在这里，通过将两个系列的值转换为相同的 numpy 日期时间类型(在此过程中降低 hourly_index)来获得加速。将 .values 传递给 numpy 会稍微加快速度。

更新一:生成日期范围，Series.isin

比初始出价更快的方法，受@IanS 方法的启发:每小时为数据中的完整日期范围生成日期范围，并仅选择与数据中现有日期匹配的条目:

%%timeit
hourly_index3 = pd.date_range(daily_index.start_time.min(), 
                              # The following line should use 
                              # +pd.DateOffset(days=1) in place of +1
                              # but is left as is to show the option.
                              daily_index.end_time.max() + 1, 
                              normalize=True, freq='H')
hourly_index3 = hourly_index3[hourly_index3.floor('D').isin(daily_index.start_time)]

100 loops, best of 3: 1.9 ms per loop

这减少了大约 75% 的处理时间。

原始答案:使用虚拟系列重新采样

使用虚拟系列，可以避免循环。在我的电脑上，它减少了大约 40% 的运行时间。

我为您的方法安排了以下时间:

In [14]: %%timeit -o -r 10
   ....: hourly_index = pd.DatetimeIndex([]).union_many(   
   ....:     pd.date_range(day.to_timestamp('H','S'), day.to_timestamp('H','E'), freq='H')
   ....:     for day in daily_index
   ....: )
   ....: 
100 loops, best of 10: 7.45 ms per loop

为了更快的方法:

In [13]: %%timeit -o -r 10
s = pd.Series(0, index=daily_index)
s = s.resample('H').last()
s = s[s.index.start_time.floor('D').isin(daily_index.start_time)]
hourly_index2 = s.index.start_time
   ....: 
100 loops, best of 10: 4.37 ms per loop

请注意，我们并不真正关心系列中的值(value)；这里我只是默认为 int。

表达式 s.index.start_time.floor('D').isin(daily_index.start_time) 为我们提供了一个 bool 向量，其值在 s.index 中匹配 daily_index 中的日期。