python - 在 Python 中使用 Kmode 和大 csv 文件

Question

我需要一些帮助来解决我遇到的问题。我有一个大 csv 文件 (6239292, 5)，想要执行无监督机器学习技术 (kmodes)。我的代码是这样的:

import numpy as np
import pandas as pd
print("initialising")


syms = np.genfromtxt('foo.csv', delimiter = ';', dtype=str, skip_header=1, invalid_raise=False)[:, 0:]

print(syms.shape)

X = np.genfromtxt('foo.csv',dtype=object, delimiter=';', invalid_raise=False, skip_header=1)[:, 1:]

X[1:, 0] = X[1:, 0].astype(float) 

from kmodes.kprototypes import KPrototypes

print("Imported successfully")
kproto = KPrototypes(n_clusters=6, init='random', n_init=2, verbose=2)
clusters = kproto.fit_predict(X, categorical=[2,1,3,])

由于文件的大小，它需要很长时间。有什么技巧可以用来减少时间吗？预先感谢您!

Answer 1

您可以选择前 n 行，例如:

read_csv(..., nrows=999999)

或者跳过一些行，然后选择接下来的 n 行:

read_csv(..., skiprows=1000000, nrows=999999)

由于 Central Limit Theorem，您的结果应该不会有问题

The Central Limit Theorem (CLT) is a statistical theory states that given a sufficiently large sample size from a population with a finite level of variance, the mean of all samples from the same population will be approximately equal to the mean of the population.