python - 如何将数据拆分为训练和测试，同时牢记 pandas 中的 groupby 列？

Question

我想按照 20:80 的比例将数据集拆分为测试数据集和训练数据集。但是，在拆分时，我不希望以 1 S_Id 值在训练中具有很少的数据点而在测试中具有其他数据点的方式进行拆分。

我有一个数据集:

S_Id      Datetime               Item      
1         29-06-2018 03:23:00    654
1         29-06-2018 04:01:00    452
1         29-06-2018 04:25:00    101
2         30-06-2018 05:17:00    088
2         30-06-2018 05:43:00    131
3         30-06-2018 10:36:00    013
3         30-06-2018 11:19:00    092

我想像这样整齐地分割:火车:

S_Id      Datetime               Item      
1         29-06-2018 03:23:00    654
1         29-06-2018 04:01:00    452
1         29-06-2018 04:25:00    101
2         30-06-2018 05:17:00    088
2         30-06-2018 05:43:00    131

测试:

S_Id      Datetime               Item 
3         30-06-2018 10:36:00    013
3         30-06-2018 11:19:00    092

所有相同的 S_Id 必须位于一组中。可以通过简单的“groupby”来完成吗？

感谢您的帮助!

Answer 1

我不相信有一个直接的函数可以做到这一点，所以你可以编写一个自定义的函数:

def sample_(we_array, train_size):
    """
     we_array : used as the weight of each unique element on your S_id column, 
     it's normalized to represent a probability

    """
    idx = np.arange(we_array.size) #get the index for each element on the array
    np.random.shuffle(idx) #shuffle it 
    cum = we_array[idx].cumsum()  
    train_idx = idx[cum<train_size]# we take the first elements until we have  
                                   # our desired size
    test_idx = idx[cum>=train_size]
    return train_idx, test_idx

idx = df.S_Id.values
unique, counts = np.unique(idx, return_counts = True) # we access the unique 
                                                      # elements and their cout
probability = counts/counts.sum()
train_idx, test_idx = sample_(probability, 0.8)
train = df[df.S_Id.isin(unique[train_idx])] 
test = df[df.S_Id.isin(unique[test_idx])]