machine-learning - 如何将训练数据集拆分为训练数据集、验证数据集和测试数据集？

Question

我有一个自定义图像数据集及其目标。我在 PyTorch 中创建了一个训练数据集。我想将其分为三个部分:训练、验证和测试。我该怎么做？

Answer 1

一旦您拥有了“主”数据集，您就可以使用 data.Subset拆分它。
这是随机分割的示例

import torch
from torch.utils import data
import random

master = data.Dataset( ... )  # your "master" dataset
n = len(master)  # how many total elements you have
n_test = int( n * .05 )  # number of test/val elements
n_train = n - 2 * n_test
idx = list(range(n))  # indices to all elements
random.shuffle(idx)  # in-place shuffle the indices to facilitate random splitting
train_idx = idx[:n_train]
val_idx = idx[n_train:(n_train + n_test)]
test_idx = idx[(n_train + n_test):]

train_set = data.Subset(master, train_idx)
val_set = data.Subset(master, val_idx)
test_set = data.Subset(master, test_idx)

这也可以使用 data.random_split 来实现:

train_set, val_set, test_set = data.random_split(master, (n_train, n_val, n_test))