python - tensorflow.keras.preprocessing.text.Tokenizer.texts_to_sequences 的 Numpy 数组给出了奇怪的输出，list([2]) 而不是 [[2]]

Question

Numpy 数组 tensorflow.keras.preprocessing.text.Tokenizer.texts_to_sequences 为训练标签提供奇怪的输出，如下所示:

(training_label_list[0:10]) = [list([1]) list([1]) list([1]) list([1]) list([1]) list([1]) list([1]) list([1]) list([1]) list([1])]

但是正在打印验证标签的普通数组，

(validation_label_list[0:10]) = [[16]
 [16]
 [16]
 [16]
 [16]
 [16]
 [16]
 [16]
 [16]
 [16]]

换句话说，type(training_label_list[0]) = <class 'list'>但是

type(validation_label_list[0]) =  <class 'numpy.ndarray'>

因此，在使用 Keras Model.fit 训练模型时，它导致以下错误，

ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type list).

这是 Google Colab 的链接，轻松重现错误。

重现错误的完整代码如下:

!pip install tensorflow==2.1

# For Preprocessing the Text => To Tokenize the Text
from tensorflow.keras.preprocessing.text import Tokenizer
# If the Two Articles are of different length, pad_sequences will make the length equal
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Package for performing Numerical Operations
import numpy as np

Unique_Labels_List = ['India', 'USA', 'Australia', 'Germany', 'Bhutan', 'Nepal', 'New Zealand', 'Israel', 'Canada', 'France', 'Ireland', 'Poland', 'Egypt', 'Greece', 'China', 'Spain', 'Mexico']


Train_Labels = Unique_Labels_List[0:14]
#print('Train Labels = {}'.format(Train_Labels))

Val_Labels =  Unique_Labels_List[14:]
#print('Val_Labels = {}'.format(Val_Labels))

No_Of_Train_Items = [248, 200, 200, 218, 248, 248, 249, 247, 220, 200, 200, 211, 224, 209]
No_Val_Items = [212, 200, 219]

T_L = []
for Each_Label, Item in zip(Train_Labels, No_Of_Train_Items):
    T_L.append([Each_Label] * Item)

T_L = [item for sublist in T_L for item in sublist]

V_L = []
for Each_Label, Item in zip(Val_Labels, No_Val_Items):
    V_L.append([Each_Label] * Item)

V_L = [item for sublist in V_L for item in sublist]


len(T_L)

len(V_L)

label_tokenizer = Tokenizer()

label_tokenizer.fit_on_texts(Unique_Labels_List)

# Since it should be a Numpy Array, we should Convert the Sequences to Numpy Array, for both Training and 
# Test Labels

training_label_list = np.array(label_tokenizer.texts_to_sequences(T_L))

validation_label_list = np.array(label_tokenizer.texts_to_sequences(V_L))

print('(training_label_list[0:10]) = {}'.format((training_label_list[0:10])))
print('(validation_label_list[0:10]) = {}'.format((validation_label_list[0:10])))

print('type(training_label_list[0]) = ', type(training_label_seq[0]))
print('type(validation_label_seq[0]) = ', type(validation_label_seq[0]))

如果有人可以建议我如何以相同的格式获得培训标签和验证标签，我将不胜感激，因为我在这方面花了很多时间。

Answer 1

将 np.array 替换为 np.hstack，如本 Stack Overflow Answer 中所述已经为我解决了这个问题。

现在，正确的输出是

(training_label_seq[0:10]) = [1 1 1 1 1 1 1 1 1 1]
(validation_label_seq[0:10]) = [16 16 16 16 16 16 16 16 16 16]
type(training_label_list[0]) =  <class 'numpy.int64'>
type(validation_label_seq[0]) =  <class 'numpy.int64'>

工作代码的链接在此 Google Colab .

下面提到的是工作代码(以防万一上述链接不起作用):

!pip install tensorflow==2.1

# For Preprocessing the Text => To Tokenize the Text
from tensorflow.keras.preprocessing.text import Tokenizer
# If the Two Articles are of different length, pad_sequences will make the length equal
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Package for performing Numerical Operations
import numpy as np

Unique_Labels_List = ['India', 'USA', 'Australia', 'Germany', 'Bhutan', 'Nepal', 'New Zealand', 'Israel', 'Canada', 'France', 'Ireland', 'Poland', 'Egypt', 'Greece', 'China', 'Spain', 'Mexico']


Train_Labels = Unique_Labels_List[0:14]
#print('Train Labels = {}'.format(Train_Labels))

Val_Labels =  Unique_Labels_List[14:]
#print('Val_Labels = {}'.format(Val_Labels))

No_Of_Train_Items = [248, 200, 200, 218, 248, 248, 249, 247, 220, 200, 200, 211, 224, 209]
No_Val_Items = [212, 200, 219]

T_L = []
for Each_Label, Item in zip(Train_Labels, No_Of_Train_Items):
    T_L.append([Each_Label] * Item)

T_L = [item for sublist in T_L for item in sublist]

V_L = []
for Each_Label, Item in zip(Val_Labels, No_Val_Items):
    V_L.append([Each_Label] * Item)

V_L = [item for sublist in V_L for item in sublist]


len(T_L)

len(V_L)

label_tokenizer = Tokenizer()

label_tokenizer.fit_on_texts(Unique_Labels_List)

# Since it should be a Numpy Array, we should Convert the Sequences to Numpy Array, for both Training and 
# Test Labels

training_label_list = np.hstack(label_tokenizer.texts_to_sequences(T_L))

validation_label_list = np.hstack(label_tokenizer.texts_to_sequences(V_L))

print('(training_label_list[0:10]) = {}'.format((training_label_list[0:10])))
print('(validation_label_list[0:10]) = {}'.format((validation_label_list[0:10])))

print('type(training_label_list[0]) = ', type(training_label_seq[0]))
print('type(validation_label_seq[0]) = ', type(validation_label_seq[0]))