python - Pandas :如何在使用 read_csv 时获取行读取状态？

Question

我正在加载一个非常大的 csv 文件，比如 1000 万条记录，使用 pandas 和 read_csv 方法，我想知道是否有办法显示进度该加载的内容，例如:

100,000 lines read
150,000 lines read

谢谢。

Answer 1

要像这样显示进度:

Completed 1 %
Completed 2 % 
... 
Completed 99 % 
Completed 100 %

你可以试试这个:

import os, pandas
filename = "VeryLong.csv"
lines_number = sum(1 for line in open(filename))
lines_in_chunk = 500 # I don't know what size is better
counter = 0
completed = 0
reader = pandas.read_csv(filename, chunksize=lines_in_chunk)
for chunk in reader:
    # < ... reading the chunk somehow... >
    # showing progress:
    counter += lines_in_chunk
    new_completed = int(round(float(counter)/lines_number * 100))
    if (new_completed > completed): 
        completed = new_completed
        print "Completed", completed, "%"