python - 使用 CNN 训练海量数据

Question

作为机器学习的新手，我尝试寻找很多用于训练分类器以及如何使数据可读的方法。到目前为止我所知道的是，标签对于分类器来说是最重要的，这是显而易见的。我的问题是，我有巨大的数据集，其中有超过 300000 个图像和标签，在其他文件夹中解释每个图像的侧面和边界框，我还有其他数据信息，这些信息位于其他文件夹中，不在标签文件夹中，而是在杂项文件夹中.mat 文件和每个 .mat 文件都包含图像中汽车的品牌或型号名称。因为，到目前为止，我正在将图像和标签数据转换为 numpy 格式，并将它们附加到training_data中，我应该如何处理杂项数据，以便它也可以使用标签和图像数据进行训练。

我们将非常感谢您的回答。

我将在下面详细解释该文件夹，以便大家对数据有更多了解。我只需要理论答案步骤，如果你们能做到的话。

数据说明

文件夹和文件说明如下:

-image:
  Stores all full car images in the path format 'make_id/model_id/released_year/image_name.jpg'.
-label:
  Stores all labels to the full car images in the path format 'make_id/model_id/released_year/image_name.txt'. Each label file has three lines. The first line is a number which is the viewpoint annotation (-1 - uncertain, 1 - front, 2 - rear, 3 - side, 4 - front-side, 5 - rear-side). The second line is the number of the bounding boxes, which is all '1' in the current release. The third line is the coordinates of the bounding box in the format 'x1 y1 x2 y2' in pixels, where 1 <= x1 < x2 <= image_width, and 1 <= y1 < y2 <= image_height. 
-misc:
  -attributes.txt:
    Each line is the attribute annotation for one model which is in the format 'model_id maximum_speed displacement door_number seat_number type'. For car types, a number from 1~12 corresponds to a specific type, which is described in 'car_type.mat'. Unavailable attributes are denoted by '0' or '0.0'.
  -make_model_name.mat
    Cell array 'make_names' provides the projections from 'make_id' to make names, and cell array 'model_names' provides the projections from 'model_id' to model names.
-part:
  Stores all part images in the path format 'make_id/model_id/released_year/part_id/image_name.jpg'. The correspondance of 'part_id' and part names are: 1 - headlight, 2 - taillight, 3 - fog light, 4 - air intake, 5 - console, 6 - steering wheel, 7 - dashboard, and 8 - gear lever.
-train_test_split:
  This folder generally provides all the train/test subsets used in the paper.
  -classification
    Stores the train/test lists for the classification task with full car images in the paper.
  -part:
    Stores the train/test lists for the classification task with car part in the paper.
  -verification:
    'verification_train.txt' is the image list for training the verification models which is also for testing attribute prediction. 'verification_pairs_easy.txt', 'verification_pairs_medium.txt', and 'verification_pairs_hard.txt' are the three sets with different difficulties for testing car verification models. Each line of 'verification_pairs_XXX.txt' is in the format of 'path_to_image_1 path_to_image_2 label' where label is '1' for positive pairs and is '0' for negative pairs.

谢谢。

Answer 1

您可以做的是将杂项数据设为数字。

假设您有 3 种不同型号的汽车:法拉利、特斯拉 和兰博。您可以定义，如果汽车是 Ferrari，则其编号为 0。如果是 Tesla，则编号为 1；如果是 Lambo，则编号为 2。现在，当您加载标签时，还加载杂项文件夹数据并按照上面的定义进行交换:Ferrari = 0, Tesla = 1, Lambo = 2 现在您可以将其附加到标签向量并教网络预测汽车模型的 0、1 或 2。当网络做出预测后，您可以断言预测是否正确(例如，如果神经网络预测 1 则意味着图像中的汽车是 Tesla)。

您可以将此方法应用于misc文件夹中的任何其他非数字特征。这称为嵌入 - 将非数值转换为数值。