python - Tensorflow Dataset API 读取 csv 转换 tfrecords

Question

使用 Tensorflow 1.3 的 Dataset API 读取 tfrecords 文件时出现错误

2017-09-03 21:33:53.751096: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Name: <unknown>, Key: features, Index: 0.  Number of float values != expected.  Values size: 14 but output shape: []
2017-09-03 21:33:53.751173: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Name: <unknown>, Key: features, Index: 0.  Number of float values != expected.  Values size: 14 but output shape: []
 [[Node: ParseSingleExample/ParseExample/ParseExample = ParseExample[Ndense=2, Nsparse=0, Tdense=[DT_FLOAT, DT_FLOAT], dense_shapes=[[], []], sparse_types=[]](ParseSingleExample/ExpandDims, ParseSingleExample/ParseExample/ParseExample/names, ParseSingleExample/ParseExample/ParseExample/dense_keys_0, ParseSingleExample/ParseExample/ParseExample/dense_keys_1, ParseSingleExample/ParseExample/Reshape, ParseSingleExample/ParseExample/Reshape_1)]]
Traceback (most recent call last):
 File "/home/fan/PycharmProjects/DeepStockMarket/t.py", line 32, in <module>
 print(sess.run(label))
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1118, in _run
 feed_dict_tensor, options, run_metadata)
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1315, in _do_run
options, run_metadata)
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1334, in _do_call
 raise type(e)(node_def, op, message)
 tensorflow.python.framework.errors_impl.InvalidArgumentError: Name: <unknown>, Key: features, Index: 0.  Number of float values != expected.  Values size: 14 but output shape: []
 [[Node: ParseSingleExample/ParseExample/ParseExample = ParseExample[Ndense=2, Nsparse=0, Tdense=[DT_FLOAT, DT_FLOAT], dense_shapes=[[], []], sparse_types=[]](ParseSingleExample/ExpandDims, ParseSingleExample/ParseExample/ParseExample/names, ParseSingleExample/ParseExample/ParseExample/dense_keys_0, ParseSingleExample/ParseExample/ParseExample/dense_keys_1, ParseSingleExample/ParseExample/Reshape, ParseSingleExample/ParseExample/Reshape_1)]]
 [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[?], [?]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](OneShotIterator)]]

tfrecords 文件是使用以下代码从 CSV 文件转换而来的

csv = pd.read_csv("./source/000001/1991/BASIC_000001_1991.csv").values
with tf.python_io.TFRecordWriter("csv.tfrecords") as writer:
    for row in csv:
        features, label = row[1:-1], row[-1]
        example = tf.train.Example()
        example.features.feature["features"].float_list.value.extend(features)
        example.features.feature["label"].float_list.value.append(label)

我正在使用下面的代码来阅读它

def _p_fn(proto):
    f = {
        "features": tf.FixedLenFeature([], tf.float32, default_value=0.0),
        "label": tf.FixedLenFeature([], tf.float32, default_value=0.0)
    }
    parsed_features = tf.parse_single_example(proto, f)
    features = parsed_features["features"]
    label = parsed_features["label"]
    return features, label

f = ["csv.tfrecords"]
dataset = tf.contrib.data.TFRecordDataset(f)
dataset = dataset.map(_p_fn)
dataset = dataset.batch(5)
iterator = dataset.make_one_shot_iterator()
features, label = iterator.get_next()

sess = tf.Session()
print(sess.run(label))

有谁知道哪里出了问题吗？非常感谢

Answer 1

由于您正在使用 FixedLenFeature 和您的 len(feature)>1 您应该清楚地指定形状

 # from the error msg youe feature len = 14
 f = {
    "features": tf.FixedLenFeature([14], tf.float32, default_value=tf.zeros([14])),
    "label": tf.FixedLenFeature([], tf.float32, default_value=0.0)
 }