python - TF.data 迁移到 dataset.interleave

Question

TensorFlow 每晚:1.15.0-dev20190730

filenames = tf.gfile.Glob(data_files_pattern)
dataset = tf.data.Dataset.from_tensor_slices(filenames).repeat()

def _read_fn(f):
  return tf.data.TFRecordDataset(f)

dataset = dataset.apply(tf.data.experimental.parallel_interleave(
    map_func=_read_fn,
    cycle_length=CYCLE_LENGTH,
    block_length=BLOCK_LENGTH,
    sloppy=True,
    buffer_output_elements=BUFFER_OUTPUT_ELEMENTS,
    prefetch_input_elements=BUFFER_INPUT_ELEMENTS))
dataset = dataset.batch(BATCH_SIZE, drop_remainder=False)
dataset = dataset.prefetch(PREFETCH)
return dataset

我收到以下警告:

WARNING:tensorflow:From sample.py:35: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.
W0909 06:50:51.144233 140600866592512 deprecation.py:323] From sample.py:35: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.

当我迁移以避免警告时，我的读取速度变慢并且 CPU 利用率降低:

filenames = tf.gfile.Glob(data_files_pattern)
dataset = tf.data.Dataset.from_tensor_slices(filenames).repeat()

def _read_fn(f):
   return tf.data.TFRecordDataset(f)

options = tf.data.Options()
options.experimental_deterministic = True
dataset = dataset.interleave(
    map_func=_read_fn,
    cycle_length=CYCLE_LENGTH,
    block_length=BLOCK_LENGTH,      
    num_parallel_calls=tf.data.experimental.AUTOTUNE).with_options(options)
dataset = dataset.batch(BATCH_SIZE, drop_remainder=False)
dataset = dataset.prefetch(PREFETCH)
return dataset

我的迁移正确吗？

Answer 1

问题在于您正在将草率(非确定性)parallel_interleave 与确定性interleave 进行比较。您为 parallel_interleave 设置了 sloppy=True，因此为了正确迁移，您需要设置

options.experimental_deterministic = False

用于交错。