python - 无法解决，最终确定 GeneratorDataset 迭代器 : Failed precondition: Python interpreter state is not initialized 时发生错误

Question

无法解决此错误:

tensorflow.python.framework.errors_impl.InvalidArgumentError:  logits and labels must have the same first dimension, got logits shape [4,1] and labels shape [12]
     [[node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at /Users/astro/pythonprojects/covid_chest_xray_image_classification/main.py:86) ]] [Op:__inference_train_function_978]

Function call stack:
train_function

2020-07-17 01:50:11.552216: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
     [[{{node PyFunc}}]]

Process finished with exit code 1

我正在尝试根据三个类别对图像进行分类，
如果有帮助，这是我的代码:

# Classification of cases from Chest-xray images
import os
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.image import imread
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, MaxPool2D, Dropout, Flatten
from tensorflow.keras.callbacks import EarlyStopping

data_dir = 'Covid19-dataset'

# Creating paths
train_path = 'Covid19-dataset/train'
test_path = 'Covid19-dataset/test'

# Displaying one image
normal_xray_path = train_path + '/Normal/01.jpeg'
normal_xray_path_arr = imread(normal_xray_path)

# print(len(os.listdir(os.path.join(train_path, 'Viral Pneumonia'))))
# number of images for Normal, Covid, Viral Pneumonia = 70,111,70

# Dimensionality study
dim1 = []
dim2 = []
test_covid_path = test_path + '/Covid'
for image_name in os.listdir(os.path.join(test_path, 'Covid')):
    img = imread(os.path.join(test_covid_path, image_name))
    # print(img.shape)
    '''debug for tuple length 2'''
    # if len(img.shape)==2:
    #     print(image_name)
    d1, d2, colors = img.shape
    dim1.append(d1)
    dim2.append(d2)
# print(np.mean(dim1), np.mean(dim2)) = 728.2 782.6

# Keeping dimensions of images same
image_shape = (540, 583, 3)

img_gen = ImageDataGenerator(rotation_range=5, width_shift_range=0.1, height_shift_range=0.1,
                             rescale=1 / 255, shear_range=0.1, zoom_range=0.1)

img_gen.flow_from_directory(test_path)
# plt.imshow(normal_xray_path_arr)
# plt.show()
# plt.imshow(img_gen.random_transform(normal_xray_path_arr))
# plt.show()

model = Sequential()
model.add(Conv2D(filters=32, kernel_size=(3, 3), input_shape=image_shape, activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Conv2D(filters=64, kernel_size=(3, 3), input_shape=image_shape, activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Conv2D(filters=64, kernel_size=(3, 3), input_shape=image_shape, activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Conv2D(filters=128, kernel_size=(3, 3), input_shape=image_shape, activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))

model.add(Dense(1, activation='softmax'))

model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
early_stop = EarlyStopping(monitor='val_loss', patience=2)

# Settings
batch_size = 4

train_image_gen = img_gen.flow_from_directory(train_path, target_size=image_shape[:2],
                                              color_mode='rgb',
                                              batch_size=batch_size,
                                              class_mode='categorical')
test_image_gen = img_gen.flow_from_directory(test_path, target_size=image_shape[:2],
                                             color_mode='rgb',
                                             batch_size=batch_size,
                                             class_mode='categorical',
                                             shuffle=False)
model.summary()

# Result index
# print(train_image_gen.class_indices) ={'Covid': 0, 'Normal': 1, 'Viral Pneumonia': 2}

results = model.fit_generator(train_image_gen, epochs=30, validation_data=test_image_gen,
                              callbacks=[early_stop])

输出:

C:\Users\astro\AppData\Local\Programs\Python\Python38\python.exe C:/Users/astro/pythonprojects/covid_chest_xray_image_classification/main.py
2020-07-17 01:50:05.579136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Found 57 images belonging to 3 classes.
2020-07-17 01:50:07.487490: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-07-17 01:50:07.513725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.515GHz coreCount: 14 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 178.84GiB/s
2020-07-17 01:50:07.514010: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-07-17 01:50:07.517418: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-07-17 01:50:07.520744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-07-17 01:50:07.521789: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-07-17 01:50:07.525789: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-07-17 01:50:07.527902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-07-17 01:50:07.535988: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-07-17 01:50:07.536194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-07-17 01:50:07.536719: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-07-17 01:50:07.543508: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1195281a180 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-17 01:50:07.543717: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-07-17 01:50:07.543983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.515GHz coreCount: 14 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 178.84GiB/s
2020-07-17 01:50:07.544272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-07-17 01:50:07.544416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-07-17 01:50:07.544560: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-07-17 01:50:07.544703: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-07-17 01:50:07.544846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-07-17 01:50:07.544997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-07-17 01:50:07.545141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-07-17 01:50:07.545307: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-07-17 01:50:08.078548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-17 01:50:08.078710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2020-07-17 01:50:08.078803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2020-07-17 01:50:08.079063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2917 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-07-17 01:50:08.081913: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1197b02d2c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-07-17 01:50:08.082107: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1650, Compute Capability 7.5
Found 209 images belonging to 3 classes.
WARNING:tensorflow:From C:/Users/astro/pythonprojects/covid_chest_xray_image_classification/main.py:86: Model.fit_generator (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
Please use Model.fit, which supports generators.
Found 57 images belonging to 3 classes.
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 538, 581, 32)      896       
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 269, 290, 32)      0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 267, 288, 64)      18496     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 133, 144, 64)      0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 131, 142, 64)      36928     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 65, 71, 64)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 63, 69, 128)       73856     
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 31, 34, 128)       0         
_________________________________________________________________
flatten (Flatten)            (None, 134912)            0         
_________________________________________________________________
dense (Dense)                (None, 128)               17268864  
_________________________________________________________________
dropout (Dropout)            (None, 128)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 129       
=================================================================
Total params: 17,399,169
Trainable params: 17,399,169
Non-trainable params: 0
_________________________________________________________________
Epoch 1/30
2020-07-17 01:50:09.168943: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-07-17 01:50:09.706730: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-07-17 01:50:10.841108: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation. 
Modify $PATH to customize ptxas location.
This message will be only logged once.
Traceback (most recent call last):
  File "C:/Users/astro/pythonprojects/covid_chest_xray_image_classification/main.py", line 86, in <module>
    results = model.fit_generator(train_image_gen, epochs=30, validation_data=test_image_gen,
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1465, in fit_generator
    return self.fit(
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\keras\engine\training.py", line 66, in _method_wrapper
    return method(self, *args, **kwargs)
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\keras\engine\training.py", line 848, in fit
    tmp_logs = train_function(iterator)
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\def_function.py", line 644, in _call
    return self._stateless_fn(*args, **kwds)
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\function.py", line 2420, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\function.py", line 1661, in _filtered_call
    return self._call_flat(
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\function.py", line 1745, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\function.py", line 593, in call
    outputs = execute.execute(
  File "C:\Users\astro\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError:  logits and labels must have the same first dimension, got logits shape [4,1] and labels shape [12]
     [[node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at /Users/astro/pythonprojects/covid_chest_xray_image_classification/main.py:86) ]] [Op:__inference_train_function_978]

Function call stack:
train_function

2020-07-17 01:50:11.552216: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
     [[{{node PyFunc}}]]

Process finished with exit code 1

错误还包括:

tensorflow.python.framework.errors_impl.InvalidArgumentError:  logits and labels must have the same first dimension, got logits shape [4,1] and labels shape [12]

我不知道如何解决这个问题

Answer 1

您需要指定 input_shape仅适用于第一个 Conv2D 层。实际上，input_shape到下一个 Conv2D 层将不同于 image_shape .这些层看起来像:

model.add(Conv2D(filters=32, kernel_size=(3, 3), input_shape=image_shape, activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))
model.add(Conv2D(filters=128, kernel_size=(3, 3), activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2)))