pandas - 过滤后取消堆叠 MultiIndex pandas 数据帧时出现越界错误

Question

我有一个多索引 pandas DataFrame，我对其执行了一些操作(包括删除具有空值的列)，然后尝试取消堆叠...但这会导致索引错误。有任何解决这个问题的方法吗？代码如下:

ds = ds.unstack(level='Symbol')
ds.columns = ds.columns.swaplevel(0, 1)
ds = ds[start:end]
ds = ds[equities]
ds = ds.stack(level='Symbol')
ds.dropna(axis=1, inplace=True) # this line breaks the code
ds = ds.unstack(level='Symbol')
ds.head()

如果没有 dropna 行，代码执行良好，所以这会破坏索引……这对我来说似乎是一个错误。这会引发某些数据帧的错误，但并非所有数据帧都可能仅针对某些情况。任何帮助将不胜感激!

转储错误日志如下:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-341-efb7e680485a> in <module>()
      9 #ds.dropna(axis=1, inplace=True)
     10 
---> 11 ds = ds.unstack(level='Symbol')
     12 
     13 ds.head()

~/.local/lib/python3.5/site-packages/pandas/core/frame.py in unstack(self, level, fill_value)
   4567         """
   4568         from pandas.core.reshape.reshape import unstack
-> 4569         return unstack(self, level, fill_value)
   4570 
   4571     _shared_docs['melt'] = ("""

~/.local/lib/python3.5/site-packages/pandas/core/reshape/reshape.py in unstack(obj, level, fill_value)
    467     if isinstance(obj, DataFrame):
    468         if isinstance(obj.index, MultiIndex):
--> 469             return _unstack_frame(obj, level, fill_value=fill_value)
    470         else:
    471             return obj.T.stack(dropna=False)

~/.local/lib/python3.5/site-packages/pandas/core/reshape/reshape.py in _unstack_frame(obj, level, fill_value)
    480         unstacker = partial(_Unstacker, index=obj.index,
    481                             level=level, fill_value=fill_value)
--> 482         blocks = obj._data.unstack(unstacker)
    483         klass = type(obj)
    484         return klass(blocks)

~/.local/lib/python3.5/site-packages/pandas/core/internals.py in unstack(self, unstacker_func)
   4349         new_columns = new_columns[columns_mask]
   4350 
-> 4351         bm = BlockManager(new_blocks, [new_columns, new_index])
   4352         return bm
   4353 

~/.local/lib/python3.5/site-packages/pandas/core/internals.py in __init__(self, blocks, axes, do_integrity_check, fastpath)
   3035         self._consolidate_check()
   3036 
-> 3037         self._rebuild_blknos_and_blklocs()
   3038 
   3039     def make_empty(self, axes=None):

~/.local/lib/python3.5/site-packages/pandas/core/internals.py in _rebuild_blknos_and_blklocs(self)
   3123         for blkno, blk in enumerate(self.blocks):
   3124             rl = blk.mgr_locs
-> 3125             new_blknos[rl.indexer] = blkno
   3126             new_blklocs[rl.indexer] = np.arange(len(rl))
   3127 

IndexError: index 100352 is out of bounds for axis 1 with size 100352

Answer 1

问题是 dropna 删除了一些行，因此 MultiIndex 的值也被删除了，但是 MultiIndex 默认情况下不会更改。因此需要通过 MultiIndex.remove_unused_levels 从 MultiIndex 中删除这些不必要的值.

ds = ds.stack(level='Symbol')
ds.dropna(axis=1, inplace=True)

ds.index = ds.index.remove_unused_levels()

ds = ds.unstack(level='Symbol')