python - 在 keras 中使用 TFRecords

Question

我已经将一个图像数据库转换为两个 TFRecords，一个用于训练，另一个用于验证。我想使用这两个文件进行数据输入，使用 keras 训练一个简单的模型，但我收到一个错误，我无法理解与数据形状相关的错误。

这是代码(所有大写变量在别处定义):

def _parse_function(proto):
    f = {
        "x": tf.FixedLenSequenceFeature([IMG_SIZE[0] * IMG_SIZE[1]], tf.float32, default_value=0., allow_missing=True),
        "label": tf.FixedLenSequenceFeature([1], tf.int64, default_value=0, allow_missing=True)
    }
    parsed_features = tf.parse_single_example(proto, f)

    x = tf.reshape(parsed_features['x'] / 255, (IMG_SIZE[0], IMG_SIZE[1], 1))
    y = tf.cast(parsed_features['label'], tf.float32)
    return x, y

def load_dataset(input_path, batch_size, shuffle_buffer):
    dataset = tf.data.TFRecordDataset(input_path)
    dataset = dataset.shuffle(shuffle_buffer).repeat()  # shuffle and repeat
    dataset = dataset.map(_parse_function, num_parallel_calls=16)
    dataset = dataset.batch(batch_size).prefetch(1)  # batch and prefetch

    return dataset.make_one_shot_iterator()

train_iterator = load_dataset(TRAIN_TFRECORDS, BATCH_SIZE, SHUFFLE_BUFFER)
val_iterator = load_dataset(VALIDATION_TFRECORDS, BATCH_SIZE, SHUFFLE_BUFFER)

model = tf.keras.Sequential()
model.add(tf.keras.layers.Flatten(input_shape=(IMG_SIZE[0], IMG_SIZE[1], 1)))
model.add(tf.keras.layers.Dense(1, 'sigmoid'))

model.compile(
    optimizer=tf.train.AdamOptimizer(),
    loss='binary_crossentropy',
    metrics=['accuracy']
)

model.fit(
    train_iterator,
    epochs=N_EPOCHS,
    steps_per_epoch=N_TRAIN // BATCH_SIZE,
    validation_data=val_iterator,
    validation_steps=N_VALIDATION // BATCH_SIZE

)

这是我得到的错误:

tensorflow.python.framework.errors_impl.InvalidArgumentError: data[0].shape = [3] does not start with indices[0].shape = [2]
     [[Node: training/TFOptimizer/gradients/loss/dense_loss/Mean_grad/DynamicStitch = DynamicStitch[N=2, T=DT_INT32, _class=["loc:@training/TFOptimizer/gradients/loss/dense_loss/Mean_grad/floordiv"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/TFOptimizer/gradients/loss/dense_loss/Mean_grad/range, training/TFOptimizer/gradients/loss/dense_loss/Mean_3_grad/Maximum, training/TFOptimizer/gradients/loss/dense_loss/Mean_grad/Shape/_35, training/TFOptimizer/gradients/loss/dense_loss/Mean_3_grad/Maximum/_41)]]

(我知道这里定义的模型不是图像分析的好模型，我只是采用了最简单的可能重现错误的架构)

Answer 1

改变:

"label": tf.FixedLenSequenceFeature([1]...

进入:

"label": tf.FixedLenSequenceFeature([]...

不幸的是，网站上的文档中没有对此进行解释，但是 some explanation可以在github上的FixedLenSequenceFeature的docstring中找到。基本上，如果您的数据由单个维度(+ 一个批处理维度)组成，则不需要指定它。