python - Sklearn 中的 T-SNE 给出 NaN 或 Inf 数据错误

Question

我正在尝试在降维数据上从 sklearn 运行 t-sne。

首先我有一个 tfidf 矩阵。这是相同的代码。

def tf_vectorizer(文档): """提取每个文档的词频向量

"""

from sklearn.feature_extraction.text import TfidfVectorizer
print("Extracting tfidf features for clustering...\n")
tf_vec = TfidfVectorizer(max_df=0.95, min_df=2,norm='l2').fit(docs)

print("Tf-idf features extracted!!\n")
return tf_vec

然后我使用 TruncatedSVD 将维度从 11K 减少到 500。这是

def reduce_dimensions(tfidf_data,n_components):
    """ This function will reduce the dimension of the dataset"""


    from sklearn.decomposition import TruncatedSVD
    svd=TruncatedSVD(n_components=n_components,random_state=42)

    svd_reduced_data=svd.fit_transform(tfidf_data)
    svd_reduced_data=svd_reduced_data.astype('float')
    #print("Explained Variance of all components {}".format(svd.explained_variance_ratio_))
    print("Total variance explained {}".format(svd.explained_variance_ratio_.sum()))

    return svd_reduced_data

我检查了 svd_reduced_data 的输出以检查是否存在 NaN 或 Inf。

np.isnan(svd_reduced_data).sum()
0

这表明该数据中没有缺失值。现在我将这个 500 维的数据传递给 tsne 以将其减少到 2 维，如下所示:

  from sklearn.manifold import TSNE
   tsne=TSNE(n_components=n_components,n_iter=300,random_state=42)
   tsne_reduced_data=tsne.fit_transform(svd_reduced_data)

我收到此错误:

/Users/anaconda/envs/dl/lib/python3.5/site-packages/scipy/linalg/misc.py in norm(a, ord, axis, keepdims)
    127     """
    128     # Differs from numpy only in non-finite handling and the use of blas.
--> 129     a = np.asarray_chkfinite(a)
    130 
    131     # Only use optimized norms if axis and keepdims are not specified.

/Users/anaconda/envs/dl/lib/python3.5/site-packages/numpy/lib/function_base.py in asarray_chkfinite(a, dtype, order)
   1231     if a.dtype.char in typecodes['AllFloat'] and not np.isfinite(a).all():
   1232         raise ValueError(
-> 1233             "array must not contain infs or NaNs")
   1234     return a
   1235 

ValueError: array must not contain infs or NaNs

不知道为什么当基础数据没有 NaN 时会出现此错误。有什么帮助吗？

Answer 1

您还需要检查np.isinf()以确保不存在无限的功能。