pandas - Tensorflow:无法为 StringOutputStream 分配大于 kint32max 的缓冲区

Question

我正在尝试在大约 1000 万行和 50 列的数据集上运行 Tensorflow 线性逻辑回归模型。

当我在其中的一个子集(即约 600 万组)上运行相同的模型时，该模型运行良好。但是当它尝试这 1000 万组时，它会抛出以下堆栈跟踪并退出

    [libprotobuf ERROR google/protobuf/io/zero_copy_stream_impl_lite.cc:173] Cannot allocate buffer larger than kint32max for StringOutputStream.
Traceback (most recent call last):
  File "tensorflow_up.py", line 70, in <module>
    m.fit(input_fn=train_input_fn, steps=200)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 280, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 426, in fit
    loss = self._train_model(input_fn=input_fn, hooks=hooks)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 984, in _train_model
    _, loss = mon_sess.run([model_fn_ops.train_op, model_fn_ops.loss])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 462, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 786, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 744, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 883, in run
    feed_dict, options)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 909, in _call_hook_before_run
    request = hook.before_run(run_context)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/basic_session_run_hooks.py", line 340, in before_run
    "graph.pbtxt")
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/graph_io.py", line 67, in write_graph
    file_io.atomic_write_string_to_file(path, str(graph_def))
ValueError: Unable to convert message to str

这是相关的代码片段:

m = tf.contrib.learn.LinearClassifier(feature_columns=[location, master_source, device] + realValues, model_dir='~/')
m.fit(input_fn=train_input_fn, steps=200)


results = m.evaluate(input_fn=eval_input_fn, steps=1)

我在这里遗漏了什么吗？程序运行时，内存统计数据看起来不错。

Answer 1

Installing Tensorflow from sources解决了我的问题。