python - 有效地 reshape pandas 数据框列中的数组

Question

我有一个问题:

让我们考虑这样的 pandas 数据框:

Width  Height  Bitmap

67     56    <1d numpy array with length 67 * 56>
59     71    <1d numpy array with length 59 * 71>
61     73    <1d numpy array with length 61 * 73>
...    ...   ...

现在，我想将 numpy.reshape() 函数应用于位图列中的每一行。结果，它应该看起来像:

Width  Height  Bitmap

67     56    <2d numpy array with shape 67x56 >
59     71    <2d numpy array with shape 59x71 >
61     73    <2d numpy array with shape 61x73>
...    ...   ...

我有一个如下所示的工作解决方案:

for idx, bitmap in df['bitmap'].iteritems():
    df['bitmap'][idx] = np.reshape(bitmap, (df['width'][idx], df['height'][idx]))

我的位图数据框非常巨大(1,200,000 行)，所以我想有效地应用 np.reshape() 。可能吗？

Answer 1

我会保留循环，但一旦进入循环内部，我会尝试通过预先计算/将宽度和高度值存储在数组中，然后在循环内部访问它们来减少计算量。访问数组应该会更快。此外，我们将修改形状参数，而不是在循环中重新整形。

因此，实现将是 -

def arr1d_2D(df):
    r = df.width.values
    c = df.height.values
    n = df.shape[0]
    for i in range(n):
        df.iloc[i,2].shape = (r[i],c[i])

我们可以使用这里的所有 NumPy 来处理 bitmap 列的基础数据，这应该会快得多 -

def arr1d_2D_allNumPy(df):
    r = df.width.values
    c = df.height.values
    n = df.shape[0]
    b = df['bitmap'].values
    for i in range(n):
        b[i].shape = (r[i],c[i])

示例运行 -

In [9]: df
Out[9]: 
   width  height                                bitmap
0      3       2                    [0, 1, 7, 4, 8, 1]
1      2       2                          [7, 3, 8, 6]
2      2       4              [6, 8, 6, 4, 7, 0, 6, 2]
3      4       3  [8, 6, 5, 2, 2, 2, 4, 3, 3, 3, 1, 8]
4      4       3  [3, 8, 4, 8, 6, 4, 2, 3, 8, 7, 7, 4]

In [10]: arr1d_2D_allNumPy(df)

In [11]: df
Out[11]: 
   width  height                                        bitmap
0      3       2                      [[0, 1], [7, 4], [8, 1]]
1      2       2                              [[7, 3], [8, 6]]
2      2       4                  [[6, 8, 6, 4], [7, 0, 6, 2]]
3      4       3  [[8, 6, 5], [2, 2, 2], [4, 3, 3], [3, 1, 8]]
4      4       3  [[3, 8, 4], [8, 6, 4], [2, 3, 8], [7, 7, 4]]

运行时测试

方法 -

def org_app(df):   # Original approach
    for idx, bitmap in df['bitmap'].iteritems():
        df['bitmap'][idx] = np.reshape(bitmap, (df['width'][idx], \
                                                df['height'][idx]))

时间安排 -

In [43]: # Setup input dataframe and two copies for testing
    ...: a = np.random.randint(1,5,(1000,2))
    ...: df = pd.DataFrame(a, columns=(('width','height')))
    ...: n = df.shape[0]
    ...: randi = np.random.randint
    ...: df['bitmap'] = [randi(0,9,(df.iloc[i,0]*df.iloc[i,1])) for i in range(n)]
    ...: 
    ...: df_copy1 = df.copy()
    ...: df_copy2 = df.copy()
    ...: df_copy3 = df.copy()
    ...: 

In [44]: %timeit org_app(df_copy1)
1 loops, best of 3: 26 s per loop

In [45]: %timeit arr1d_2D(df_copy2)
10 loops, best of 3: 115 ms per loop

In [46]: %timeit arr1d_2D_allNumPy(df_copy3)
1000 loops, best of 3: 475 µs per loop

In [47]: 26000000/475.0  # Speedup with allNumPy version over original
Out[47]: 54736.84210526316

疯狂的50,000x+加速，只是展示了访问数据的更好方法，特别是 pandas 数据帧中的数组数据。