python - Dask 数据帧 : Get row count?

Question

简单的问题:我在 dask 中有一个数据框，包含大约 3 亿条记录。我需要知道数据框包含的确切行数。有没有简单的方法可以做到这一点？

当我尝试运行 dataframe.x.count().compute() 时，它看起来像是试图将整个数据加载到 RAM 中，但 RAM 中没有空间并且崩溃了。

Answer 1

# ensure small enough block size for the graph to fit in your memory
ddf = dask.dataframe.read_csv('*.csv', blocksize="10MB") 
ddf.shape[0].compute()

来自documentation :

blocksize <str, int or None>Optional Number of bytes by which to cut uplarger files. Default value is computed based on available physicalmemory and the number of cores, up to a maximum of 64MB. Can be anumber like 64000000` or a string like ``"64MB". If None, a singleblock is used for each file.