python - 如何分块迭代两个 Pandas 数据框

Question

对于机器学习任务，我需要处理太大的数据集，无法一次全部放入我的内存中，因此我需要将其分解成 block 。幸运的是，pandas.read_csv 有一个参数 chunk_size，您可以在其中指定要用于分析的数据量，然后使用 for 循环以 block 的形式循环遍历数据集，如下所示:

#This example can be found at http://pandas.pydata.org/pandas-docs/dev/io.html

In [120]: reader = pd.read_table('tmp.sv', sep='|', chunksize=4)

In [121]: reader
<pandas.io.parsers.TextFileReader at 0xaa94ad0>

In [122]: for chunk in reader:
   .....:     print(chunk)
   .....: 
   Unnamed: 0         0         1         2         3
0           0  0.469112 -0.282863 -1.509059 -1.135632
1           1  1.212112 -0.173215  0.119209 -1.044236
2           2 -0.861849 -2.104569 -0.494929  1.071804
3           3  0.721555 -0.706771 -1.039575  0.271860
[4 rows x 5 columns]
   Unnamed: 0         0         1         2         3
0           4 -0.424972  0.567020  0.276232 -1.087401
1           5 -0.673690  0.113648 -1.478427  0.524988
2           6  0.404705  0.577046 -1.715002 -1.039268
3           7 -0.370647 -1.157892 -1.344312  0.844885
[4 rows x 5 columns]
   Unnamed: 0         0        1         2         3
0           8  1.075770 -0.10905  1.643563 -1.469388
1           9  0.357021 -0.67460 -1.776904 -0.968914
[2 rows x 5 columns]. 

但我的机器学习算法需要 for 循环中的训练集和测试集才能对数据 block 进行预测，但我不知道该怎么做。我基本上是在寻找这个:

#pseudo code 

result = []
train = pd.read('train_set',chunksize = some_number)

test = pd.read('test_set',chunksize = some_number)
for chunk in train and test:
    result.append(do_machine_learning(train,test))
save_result(result)

更新:所以我尝试了 Any Hayden 的解决方案，但是当我尝试访问数据的特定部分时，它给了我一个新的错误:

print("getting train set")
train = pd.read_csv(os.path.join(dir,"Train.csv"),chunksize = 200000)
print("getting test set")
test = pd.read_csv(os.path.join(dir,"Test.csv"),chunksize = 200000)
result = []
for chunk in train:
    print("transforming train,test,labels into numpy arrays")
    labels = np.array(train)[:,3]
    train = np.array(train)[:,2]
    test = np.array(test)[:,2]

    print("getting estimator and predictions")
    result.append(stochastic_gradient(train,test))
    print("got everything")
result = np.array(result)

回溯:

Traceback (most recent call last):
  File "C:\Users\Ano\workspace\final_submission\src\rf.py", line 38, in <module>
    main()
  File "C:\Users\Ano\workspace\final_submission\src\rf.py", line 18, in main
    labels = np.array(train)[:,3]
IndexError: 0-d arrays can only use a single () or a list of newaxes (and a single ...) as an index

Answer 1

在 for 循环中，您可以访问当前范围内的变量:

In [11]: a = [1, 2, 3]

In [12]: b = 4

In [13]: for L in a:  # no need to "and b"
             print L, b
1 4
2 4
3 4

小心，这意味着在 for 循环中赋值会覆盖变量:

In [14]: for b in a:
             print b
1
2
3

In [15]: b
Out[15]: 3

要同时遍历两个可迭代对象，请使用 zip:

In [21]: c = [4, 5, 6]

In [22]: zip(a, c)
Out[22]: [(1, 4), (2, 5), (3, 6)]

在 python 2 中这是一个列表，因此在内存中计算(在 python 3 中不是这样)。您可以使用 izip，它是迭代器的帮凶。

In [23]: from itertools import izip  # in python 3, just use zip

In [24]: for La, Lc in izip(a, c):
             print La, Lb
1 4
2 5
3 6