machine-learning - 如何与 tflearn 进行交叉验证？

Question

我想用 tflearn 进行 k 折交叉验证。因此我想重置网络k次。但是，我认为我需要重置图表(例如 tf.reset_default_graph())，但我不确定，也不知道如何使用 tflearn 执行此操作。

我的代码

对于以下内容，您需要 hasy_tools.py

#!/usr/bin/env python

"""
Trains a simple convnet on the HASY dataset.

Gets to 76.78% test accuracy after 1 epoch.
573 seconds per epoch on a GeForce 940MX GPU.

# WARNING: THIS IS NOT WORKING RIGHT NOW
"""
import os
import hasy_tools as ht
import numpy as np

import tensorflow as tf
import tflearn
from tflearn.layers.core import input_data, fully_connected, dropout
from tflearn.layers.conv import conv_2d, max_pool_2d
from tflearn.layers.estimator import regression

batch_size = 128
nb_epoch = 1

# input image dimensions
img_rows, img_cols = 32, 32

accuracies = []

for fold in range(1, 4):
    tf.reset_default_graph()

    # Load data
    dataset_path = os.path.join(os.path.expanduser("~"), 'hasy')
    hasy_data = ht.load_data(fold=fold,
                             normalize=True,
                             one_hot=True,
                             dataset_path=dataset_path,
                             flatten=False)
    train_x = hasy_data['train']['X'][:1000]
    train_y = hasy_data['train']['y'][:1000]
    test_x = hasy_data['test']['X']
    test_y = hasy_data['test']['y']

    # Define model
    network = input_data(shape=[None, img_rows, img_cols, 1], name='input')
    network = conv_2d(network, 32, 3, activation='prelu')
    network = conv_2d(network, 64, 3, activation='prelu')
    network = max_pool_2d(network, 2)
    network = dropout(network, keep_prob=0.25)
    network = fully_connected(network, 1024, activation='tanh')
    network = dropout(network, keep_prob=0.5)
    network = fully_connected(network, 369, activation='softmax')

    # Train model
    network = regression(network, optimizer='adam', learning_rate=0.001,
                         loss='categorical_crossentropy', name='target')
    model = tflearn.DNN(network, tensorboard_verbose=0)
    model.fit({'input': train_x}, {'target': train_y}, n_epoch=nb_epoch,
              validation_set=({'input': test_x}, {'target': test_y}),
              snapshot_step=100, show_metric=True, run_id='convnet_mnist',
              batch_size=batch_size)

    # Serialize model
    model.save('cv-model-fold-%i.tflearn' % fold)

    # Evaluate model
    score = model.evaluate(test_x, test_y)
    print('Test accuarcy: %0.4f%%' % (score[0] * 100))
    accuracies.append(score[0])

accuracies = np.array(accuracies)
print(("CV Accuracy. mean={mean:0.2f}%%\t ({min:0.2f}%% - {max:0.2f}%%)"
       ).format(mean=accuracies.mean()  * 100,
                min=accuracies.min() * 100,
                max=accuracies.max() * 100))

错误

仅使用一次折叠运行代码效果很好，但使用多次折叠时我得到:

I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x5556c9100 of size 33554432
I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x5576c9100 of size 66481920
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x506d7cc00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x506d7d100 of size 62720
I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x507573700 of size 798208
I tensorflow/core/common_runtime/bfc_allocator.cc:693]      Summary of in-use Chunks by size: 
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 345 Chunks of size 256 totalling 86.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1024 totalling 1.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 23 Chunks of size 1280 totalling 28.8KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 21 Chunks of size 1536 totalling 31.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 20 Chunks of size 4096 totalling 80.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 16 Chunks of size 73728 totalling 1.12MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 20 Chunks of size 131072 totalling 2.50MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 188928 totalling 184.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 17 Chunks of size 262144 totalling 4.25MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 446464 totalling 872.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 515584 totalling 503.5KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 524288 totalling 512.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 16 Chunks of size 1511424 totalling 23.06MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 16777216 totalling 96.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 23026176 totalling 21.96MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 33554432 totalling 192.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 66481920 totalling 63.40MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 15 Chunks of size 67108864 totalling 960.00MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 67183872 totalling 64.07MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 1.40GiB
I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats: 
Limit:                  1500971008
InUse:                  1500109824
MaxInUse:               1500109824
NumAllocs:                   43767
MaxAllocSize:            844062464

W tensorflow/core/common_runtime/bfc_allocator.cc:274] **************************************************************************************************xx
W tensorflow/core/common_runtime/bfc_allocator.cc:275] Ran out of memory trying to allocate 8.00MiB.  See logs for memory state.
W tensorflow/core/framework/op_kernel.cc:975] Resource exhausted: OOM when allocating tensor with shape[128,16,16,64]
--
Traceback (most recent call last):
  File "tflearn_hasy_cv.py", line 60, in <module>
    batch_size=batch_size)
  File "/home/moose/GitHub/tflearn/tflearn/models/dnn.py", line 188, in fit
    run_id=run_id)
  File "/home/moose/GitHub/tflearn/tflearn/helpers/trainer.py", line 277, in fit
    show_metric)
  File "/home/moose/GitHub/tflearn/tflearn/helpers/trainer.py", line 684, in _train
    feed_batch)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 766, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 964, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1014, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1034, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[128,16,16,64]
     [[Node: MaxPool2D/MaxPool = MaxPool[T=DT_FLOAT, data_format="NHWC", ksize=[1, 2, 2, 1], padding="SAME", strides=[1, 2, 2, 1], _device="/job:localhost/replica:0/task:0/gpu:0"](Conv2D_1/PReLU/add)]]
     [[Node: Crossentropy/Mean/_19 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_920_Crossentropy/Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'MaxPool2D/MaxPool', defined at:
  File "tflearn_hasy_cv.py", line 47, in <module>
    network = max_pool_2d(network, 2)
  File "/home/moose/GitHub/tflearn/tflearn/layers/conv.py", line 363, in max_pool_2d
    inference = tf.nn.max_pool(incoming, kernel, strides, padding)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 1617, in max_pool
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 1598, in _max_pool
    data_format=data_format, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
    self._traceback = _extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[128,16,16,64]
     [[Node: MaxPool2D/MaxPool = MaxPool[T=DT_FLOAT, data_format="NHWC", ksize=[1, 2, 2, 1], padding="SAME", strides=[1, 2, 2, 1], _device="/job:localhost/replica:0/task:0/gpu:0"](Conv2D_1/PReLU/add)]]
     [[Node: Crossentropy/Mean/_19 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_920_Crossentropy/Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

编辑:我在 Keras 中遇到了同样的问题。

Answer 1

我在构建神经网络之前、for 循环之前输入了代码 tf.reset_default_graph()。

您还应该在 for 循环之外定义模型，就在它之前。只有训练应该在 kfolds 循环中。