python-3.x - 使大型数据集的 CountVectorizer 更快

Question

您好，我只想根据标题对电影进行聚类。我的函数对我的数据非常有用，但我有一个大问题，我的样本是 150.000 部电影，它非常慢，实际上花了 3 天时间来聚类所有电影

过程:

根据长度对电影标题进行排序

用 countvectorizer 转换电影并计算每个电影的相似度(对于每个聚类电影我每次都适合矢量化器并且我转换目标电影)

def product_similarity( clustered_movie, target_movie ):

'''
Calculates the title distance of 2 movies based on title
'''
# fitted vectorizer is a dictionary with fitted movies if wee dont fit to 
# vectorizer the movie it fits and save it to dictionary

if clustered_movie in fitted_vectorizer: 
    vectorizer = fitted_vectorizer[clustered_movie]

    a = vectorizer.transform([clustered_movie]).toarray()
    b = vectorizer.transform( [target_movie] ).toarray()
    similarity = cosine_similarity( a, b )

else:
    clustered_movie = re.sub("[0-9]|[^\w']|[_]", " ",clustered_product )

    vectorizer = CountVectorizer(stop_words=None)
    vectorizer = vectorizer.fit([clustered_movie])

    fitted_vectorizer[clustered_movie] = vectorizer

    a = vectorizer.transform([clustered_movie]).toarray()
    b = vectorizer.transform( [target_movie] ).toarray()
    similarity = cosine_similarity( a, b )

return similarity[0][0]

Answer 1

在所有标题上安装一次 CountVectorizer。保存模型。然后使用拟合模型进行转换。