python - 如何在python中预处理非常大的数据

Question

我有几个文件，每个 100 MB。这些文件的格式如下所示:

0  1  2  5  8  67  9  122
1  4  5  2  5  8
0  2  1  5  6
.....

(注意实际文件中没有添加对齐空格，每个元素只有一个空格分隔，添加对齐是为了美观)

每行中的第一个元素是它的二元分类，该行的其余部分是值为 1 的特征的索引。例如，第三行表示该行的第二个，第一个，第五个和第六个特征为1，其余为零。

我尝试从每个文件中读取每一行，并使用 sparse.coo_matrix 创建一个像这样的稀疏矩阵:

for train in train_files:  
    with open(train) as f:
        row = []
        col = []
        for index, line in enumerate(f):
            record = line.rstrip().split(' ')
            row = row+[index]*(len(record)-4)
            col = col+record[4:]
        row = np.array(row)
        col = np.array(col)
        data = np.array([1]*len(row))
        mtx = sparse.coo_matrix((data, (row, col)), shape=(n_row, max_feature))
        mmwrite(train+'trans',mtx)

但这花了很长时间才完成。晚上开始看数据，睡着了让电脑运行，醒来发现第一个文件还没读完!

处理此类数据的更好方法是什么？

Answer 1

我认为这会比您的方法快一点，因为它不会逐行读取文件。您可以使用一个文件的一小部分尝试此代码，并与您的代码进行比较。
此代码还需要提前知道功能编号。如果我们不知道特征编号，则需要另一行已被注释掉的代码。

import pandas as pd
from scipy.sparse import lil_matrix
from functools import partial


def writeMx(result, row):
    # zero-based matrix requires the feature number minus 1
    col_ind = row.dropna().values - 1
    # Assign values without duplicating row index and values
    result[row.name, col_ind] = 1


def fileToMx(f):
    # number of features
    col_n = 136
    df = pd.read_csv(f, names=list(range(0,col_n+2)),sep=' ')
    # This is the label of the binary classification
    label = df.pop(0)
    # Or get the feature number by the line below
    # But it would not be the same across different files
    # col_n = df.max().max()
    # Number of row
    row_n = len(label)
    # Generate feature matrix for one file
    result = lil_matrix((row_n, col_n))
    # Save features in matrix
    # DataFrame.apply() is usually faster than normal looping
    df.apply(partial(writeMx, result), axis=0)
    return(result)

for train in train_files:
    # result is the sparse matrix you can further save or use
    result = fileToMx(train)
    print(result.shape, result.nnz)
    # The shape of matrix and number of nonzero values
    # ((420, 136), 15)