python - Tensorflow Dataset.from_tensor_slices 花费的时间太长

Question

我有以下代码:

data = np.load("data.npy")
print(data) # Makes sure the array gets loaded in memory
dataset = tf.contrib.data.Dataset.from_tensor_slices((data))

文件 “data.npy” 是 3.3 GB。使用 numpy 读取文件需要几秒钟，但创建 tensorflow 数据集对象的下一行需要很长时间才能执行。这是为什么？它在幕后做什么？

Answer 1

引用此 answer :

np.load of a npz just returns a file loader, not the actual data. It's a 'lazy loader', loading the particular array only when accessed.

这就是它速度快的原因。

编辑 1: 进一步扩展此答案，引用自 tensorflow's documentation :

If all of your input data fit in memory, the simplest way to create a Dataset from them is to convert them to tf.Tensor objects and use Dataset.from_tensor_slices().

This works well for a small dataset, but wastes memory---because the contents of the array will be copied multiple times---and can run into the 2GB limit for the tf.GraphDef protocol buffer.

该链接还展示了如何高效地执行此操作。