python - 读取 block 中的 csv 文件时出现内存不足错误-6ren

python - 读取 block 中的 csv 文件时出现内存不足错误

转载作者：太空狗更新时间：2023-10-29 21:54:57

我正在处理一个 2.5 GB 的 csv 文件。 2.5 GB 的表如下所示:

columns=[ka,kb_1,kb_2,timeofEvent,timeInterval]
0:'3M' '2345' '2345' '2014-10-5',3000
1:'3M' '2958' '2152' '2015-3-22',5000
2:'GE' '2183' '2183' '2012-12-31',515
3:'3M' '2958' '2958' '2015-3-10',395
4:'GE' '2183' '2285' '2015-4-19',1925
5:'GE' '2598' '2598' '2015-3-17',1915

我想对 ka 和 kb_1 进行分组以获得如下结果:

columns=[ka,kb,errorNum,errorRate,totalNum of records]
'3M','2345',0,0%,1
'3M','2958',1,50%,2
'GE','2183',1,50%,2
'GE','2598',0,0%,1

(错误记录的定义:当kb_1 != kb_2时，对应的记录被视为异常记录)

我的电脑是 ubuntu 12.04，有 16 GB 内存 和 free -m 返回

             total       used       free     shared    buffers     cached
Mem:        112809      14476      98333          0        128      10823
-/+ buffers/cache:       3524     109285
Swap:    

    0          0          0

我的 python 文件名为 bigData.py

import pandas as pd
import numpy as np

import sys,traceback,os
cksize=98333 # or 1024, either chunk size didn't work at all
try:
    dfs = pd.DataFrame()
    reader=pd.read_table('data/petaJoined.csv', chunksize=cksize)  

    for chunk in reader:#when executed this line,error occur!
        pass
        #temp=tb_createTopRankTable(chunk)
        #dfs.append(temp)
        #df=tb_createTopRankTable(dfs)
   except:
    traceback.print_exc(file=sys.stdout)

ipdb> pd.__version__
'0.16.0'

我使用以下命令来监控内存使用情况:

top 
ps -C python -o %cpu,%mem,cmd

由于崩溃大约需要 2 秒，所以我可以看到 mem 的使用率在某个时候达到了 90%，并且 CPU使用率达到100%

当我执行 python bigData.py 时，会生成以下错误:

/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module dateutil was already imported from /usr/local/lib/python2.7/dist-packages/dateutil/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
  from pkg_resources import resource_stream
/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module pytz was already imported from /usr/local/lib/python2.7/dist-packages/pytz/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
  from pkg_resources import resource_stream
Traceback (most recent call last):
  File "bigData.py", line 10, in <module>
    for chunk in reader:
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 691, in __iter__
    yield self.read(self.chunksize)
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 715, in read
    ret = self._engine.read(nrows)
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1164, in read
    data = self._reader.read(nrows)
  File "pandas/parser.pyx", line 758, in pandas.parser.TextReader.read (pandas/parser.c:7411)
  File "pandas/parser.pyx", line 792, in pandas.parser.TextReader._read_low_memory (pandas/parser.c:7819)
  File "pandas/parser.pyx", line 833, in pandas.parser.TextReader._read_rows (pandas/parser.c:8268)
  File "pandas/parser.pyx", line 820, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:8142)
  File "pandas/parser.pyx", line 1758, in pandas.parser.raise_parser_error (pandas/parser.c:20728)
CParserError: Error tokenizing data. C error: out of memory
Segmentation fault (core dumped)

或

     /usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module dateutil was already imported from /usr/local/lib/python2.7/dist-packages/dateutil/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
      from pkg_resources import resource_stream
    /usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module pytz was already imported from /usr/local/lib/python2.7/dist-packages/pytz/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
      from pkg_resources import resource_stream
    Traceback (most recent call last):
      File "bigData.py", line 10, in <module>
        for chunk in reader:
      File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 691, in __iter__
        yield self.read(self.chunksize)
      File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 715, in read
        ret = self._engine.read(nrows)
      File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1164, in read
        data = self._reader.read(nrows)
      File "pandas/parser.pyx", line 758, in pandas.parser.TextReader.read (pandas/parser.c:7411)
      File "pandas/parser.pyx", line 792, in pandas.parser.TextReader._read_low_memory (pandas/parser.c:7819)
      File "pandas/parser.pyx", line 833, in pandas.parser.TextReader._read_rows (pandas/parser.c:8268)
      File "pandas/parser.pyx", line 820, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:8142)
      File "pandas/parser.pyx", line 1758, in pandas.parser.raise_parser_error (pandas/parser.c:20728)
    CParserError: Error tokenizing data. C error: out of memory
    *** glibc detected *** python: free(): invalid pointer: 0x00007f750d2a4c0e ***
    ====== Backtrace: ========
    /lib/x86_64-linux-gnu/libc.so.6(+0x7db26)[0x7f7511529b26]
    /usr/local/lib/python2.7/dist-packages/pandas/parser.so(+0x4d5a1)[0x7f750d29d5a1]
    /usr/local/lib/python2.7/dist-packages/pandas/parser.so(parser_cleanup+0x15)[0x7f750d29de45]
    /usr/local/lib/python2.7/dist-packages/pandas/parser.so(parser_free+0x9)[0x7f750d29e039]
    /usr/local/lib/python2.7/dist-packages/pandas/parser.so(+0xb43e)[0x7f750d25b43e]
   ....
    python(PyDict_SetItem+0x49)[0x577749]
    python(_PyModule_Clear+0x149)[0x4cafb9]
    python(PyImport_Cleanup+0x477)[0x4cb4f7]
    python(Py_Finalize+0x18e)[0x549f0e]
    python(Py_Main+0x3bc)[0x56b56c]
    /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f75114cd76d]
    python[0x41bb11]
    ======= Memory map: ========
    00400000-00670000 r-xp 00000000 08:01 26612                              /usr/bin/python2.7
    0086f000-00870000 r--p 0026f000 08:01 26612                              /usr/b.......
    008d9000-008eb000 rw-p 00000000 00:00 0 
    01ddb000-036f7000 rw-p 00000000 00:00 0                                  [heap]
    7f748c179000-7f74cc17a000 rw-p 00000000 00:00 0 
    7f7504000000-7f7504021000 rw-p 00000000 00:00 0 
    7f7504021000-7f7508000000 ---p 00000000 00:00 0 
    7f750bf83000-7f750c285000 rw-p 00000000 00:00 0 
    7f750c285000-7f750c586000 rw-p 00000000 00:00 0 
    7f750c586000-7f750c707000 rw-p 00000000 00:00 0 
    7f750c707000-7f750c711000 r-xp 00000000 08:01 533205                     /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
    7f750c711000-7f750c911000 ---p 0000a000 08:01 533205                     /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
    7f750c911000-7f750c912000 r--p 0000a000 08:01 533205                     /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
    7f750c912000-7f750c913000 rw-p 0000b000 08:01 533205                     /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
    7f750c913000-7f750c914000 rw-p 00000000 00:00 0 
    7f750c914000-7f750c918000 r-xp 00000000 08:01 2331                       /lib/x86_64-linux-gnu/libuuid.so.1.3.0
    7f750c918000-7f750cb17000 ---p 00004000 08:01 2331                       /lib/x86_64-linux-gnu/libuuid.so.1.3.0
    7f750cb17000-7f750cb18000 r--p 00003000 08:01 2331                       /lib/x86_64-linux-gnu/libuuid.so.1.3.0
    7f750cb18000-7f750cb19000 rw-p 00004000 08:01 2331                       /lib/x86_64-linux-gnu/libuuid.so.1.3.0
    7f750cb19000-7f750cb34000 r-xp 00000000 08:01 533071                     /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
    7f750cb34000-7f750cd33000 ---p 0001b000 08:01 533071                     /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
    7f750cd33000-7f750cd34000 r--p 0001a000 08:01 533071                     /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
    7f750cd34000-7f750cd38000 rw-p 0001b000 08:01 533071                     /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
    7f750cd38000-7f750d039000 rw-p 00000000 00:00 0 
    7f750d039000-7f750d04e000 r-xp 00000000 08:01 533070                     /usr/local/lib/python2.7/dist-packages/pandas/json.so
    7f750d04e000-7f750d24e000 ---p 00015000 08:01 533070                     /usr/local/lib/python2.7/dist-packages/pandas/json.so
    7f750d24e000-7f750d24f000 r--p 00015000 08:01 533070                     /usr/local/lib/python2.7/dist-packages/pandas/json.so
    7f750d24f000-7f750d250000 rw-p 00016000 08:01 533070                     /usr/local/lib/python2.7/dist-packages/pandas/json.so
    7f750d250000-7f750d2a9000 r-xp 00000000 08:01 533270                     /usr/local/lib/python2.7/dist-packages/pandas/parser.so
    7f750d2a9000-7f750d4a8000 ---p 00059000 08:01 533270                     /usr/local/lib/python2.7/dist-packages/pandas/parser.so
    7f750d4a8000-7f750d4a9000 r--p 00058000 08:01 533270                     /usr/local/lib/python2.7/dist-packages/pandas/parser.so
    7f750d4a9000-7f750d4af000 rw-p 00059000 08:01 533270                     /usr/local/lib/python2.7/dist-packages/pandas/parser.so
    7f750d4af000-7f750d591000 r-xp 00000000 08:01 49584                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
    7f750d591000-7f750d790000 ---p 000e2000 08:01 49584                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
    7f750d790000-7f750d798000 r--p 000e1000 08:01 49584                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
    7f750d798000-7f750d79a000 rw-p 000e9000 08:01 49584                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
    7f750d79a000-7f750d7af000 rw-p 00000000 00:00 0 
    7f750d7af000-7f750d7f1000 r-xp 00000000 08:01 530477                     /usr/lib/pyshared/python2.7/matplotlib/_path.so
    7f750d7f1000-7f750d9f1000 ---p 00042000 08:01 530477                     /usr/lib/pyshared/python2.7/matplotlib/_path.so
    7f750d9f1000-7f750d9f3000 r--p 00042000 08:01 530477                     /usr/lib/pyshared/python2.7/matplotlib/_path.so
    7f750d9f3000-7f750d9f4000 rw-p 00044000 08:01 530477                     /usr/lib/pyshared/python2.7/matplotlib/_path.so
    7f750d9f4000-7f750da2d000 r-xp 00000000 08:01 533269                     /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
    7f750da2d000-7f750dc2c000 ---p 00039000 08:01 533269                     /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
    7f750dc2c000-7f750dc2d000 r--p 00038000 08:01 533269                     /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
    7f750dc2d000-7f750dc31000 rw-p 00039000 08:01 533269                     /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
    7f750dc31000-7f750dc7d000 r-xp 00000000 08:01 533447                     /usr/local/lib/python2.7/dist-packages/pandas/_period.so
    7f750dc7d000-7f750de7c000 ---p 0004c000 08:01 533447                     /usr/local/lib/python2.7/dist-packages/pandas/_period.so
    7f750de7c000-7f750de7d000 r--p 0004b000 08:01 533447                     /usr/local/lib/python2.7/dist-packages/pandas/_period.so
    7f750de7d000-7f750de86000 rw-p 0004c000 08:01 533447                     /usr/local/lib/python2.7/dist-packages/pandas/_period.so
    7f750de86000-7f750de87000 rw-p 00000000 00:00 0 
    7f750de87000-7f750debc000 r-xp 00000000 08:01 533203                     /usr/local/lib/python2.7/dist-packages/pandas/index.so
    7f750debc000-7f750e0bb000 ---p 00035000 08:01 533203                     /usr/local/lib/python2.7/dist-packages/pandas/index.so
    7f750e0bb000-7f750e0bc000 r--p 00034000 08:01 533203                     /usr/local/lib/python2.7/dist-packages/pandas/index.so
    7f750e0bc000-7f750e0c0000 rw-p 00035000 08:01 533203                     /usr/local/lib/python2.7/dist-packages/pandas/index.so
    7f750e0c0000-7f750e295000 r-xp 00000000 08:01 533278                     /usr/local/lib/python2.7/dist-packages/pandas/algos.so
    7f750e295000-7f750e494000 ---p 001d5000 08:01 533278                     /usr/local/lib/python2.7/dist-packages/pandas/algos.so
    7f750e494000-7f750e495000 r--p 001d4000 08:01 533278                     /usr/local/lib/python2.7/dist-packages/pandas/algos.so
    7f750e495000-7f750e4a9000 rw-p 001d5000 08:01 533278                     /usr/local/lib/python2.7/dist-packages/pandas/algos.so
    7f750e4a9000-7f750e4ac000 rw-p 00000000 00:00 0 
    7f750e4ac000-7f750e4b2000 r-xp 00000000 08:01 48831                      /usr/lib/python2.7/lib-dynload/_csv.so
    7f750e4b2000-7f750e6b1000 ---p 00006000 08:01 48831                      /usr/lib/python2.7/lib-dynload/_csv.so
    7f750e6b1000-7f750e6b2000 r--p 00005000 08:01 48831                      /usr/lib/python2.7/lib-dynload/_csv.so
    7f750e6b2000-7f750e6b4000 rw-p 00006000 08:01 48831                      /usr/lib/python2.7/lib-dynload/_csv.so
    7f750e6b4000-7f750e782000 r-xp 00000000 08:01 533449                     /usr/local/lib/python2.7/dist-packages/pandas/lib.so
    7f750e782000-7f750e981000 ---p 000ce000 08:01 533449                     /usr/local/lib/python2.7/dist-packages/pandas/lib.so
    7f750e981000-7f750e982000 r--p 000cd000 08:01 533449                     /usr/local/lib/python2.7/dist-packages/pandas/lib.so
    7f750e982000-7f750e990000 rw-p 000ce000 08:01 533449                     /usr/local/lib/python2.7/dist-packages/pandas/lib.so
    7f750e990000-7f750e992000 rw-p 00000000 00:00 0 
    7f750e992000-7f750ea8f000 r-xp 00000000 08:01 533271                     /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
    7f750ea8f000-7f750ec8e000 ---p 000fd000 08:01 533271                     /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
    7f750ec8e000-7f750ec8f000 r--p 000fc000 08:01 533271                     /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
    7f750ec8f000-7f750eca1000 rw-p 000fd000 08:01 533271                     /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
    7f750eca1000-7f750eca4000 rw-p 00000000 00:00 0 
    7f750eca4000-7f750ecc5000 r-xp 00000000 08:01 48837                      /usr/lib/python2.7/lib-dynload/_ctypes.so
    7f750ecc5000-7f750eec4000 ---p 00021000 08:01 48837                      /usr/lib/python2.7/lib-dynload/_ctypes.so
    7f750eec4000-7f750eec5000 r--p 00020000 08:01 48837                      /usr/lib/python2.7/lib-dynload/_ctypes.so
    7f750eec5000-7f750eec9000 rw-p 00021000 08:01 48837                      /usr/lib/python2.7/lib-dynload/_ctypes.so
    7f750eec9000-7f750eeca000 rw-p 00000000 00:00 0 
    7f750eeca000-7f750ef24000 r-xp 00000000 08:01 532046                     /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
    7f750ef24000-7f750f123000 ---p 0005a000 08:01 532046                     /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
    7f750f123000-7f750f124000 r--p 00059000 08:01 532046                     /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
    7f750f124000-7f750f15c000 rw-p 0005a000 08:01 532046                     /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
    7f750f15c000-7f750f15d000 rw-p 00000000 00:00 0 
    7f750f15d000-7f750f166000 r-xp 00000000 08:01 532085                     /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
    7f750f166000-7f750f365000 ---p 00009000 08:01 532085                     /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
    7f750f365000-7f750f366000 r--p 00008000 08:01 532085                     /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
    7f750f366000-7f750f367000 rw-p 00009000 08:01 532085                     /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
    7f750f367000-7f750f368000 r-xp 00000000 08:01 48818                      /usr/lib/python2.7/lib-dynload/future_builtins.so
    7f750f368000-7f750f567000 ---p 00001000 08:01 48818                      /usr/lib/python2.7/lib-dynload/future_builtins.so
    7f750f567000-7f750f568000 r--p 00000000 08:01 48818                      /usr/lib/python2.7/lib-dynload/future_builtins.so
    7f750f568000-7f750f569000 rw-p 00001000 08:01 48818                      /usr/lib/python2.7/lib-dynload/future_builtins.so
    7f750f569000-7f750f588000 r-xp 00000000 08:01 48815                      /usr/lib/python2.7/lib-dynload/_io.so
    7f750f588000-7f750f787000 ---p 0001f000 08:01 48815                      /usr/lib/python2.7/lib-dynload/_io.so
    7f750f787000-7f750f788000 r--p 0001e000 08:01 48815                      /usr/lib/python2.7/lib-dynload/_io.so
    7f750f788000-7f750f791000 rw-p 0001f000 08:01 48815                      /usr/lib/python2.7/lib-dynload/_io.so
    7f750f791000-7f750f907000 r-xp 00000000 08:01 532132                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
    7f750f907000-7f750fb06000 ---p 00176000 08:01 532132                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
    7f750fb06000-7f750fb07000 r--p 00175000 08:01 532132                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
    7f750fb07000-7f750fb08000 rw-p 00176000 08:01 532132                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
    7f750fb08000-7f750fba4000 rw-p 00000000 00:00 0 
    7f750fba4000-7f750fd01000 r-xp 00000000 08:01 532128                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
    7f750fd01000-7f750ff00000 ---p 0015d000 08:01 532128                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
    7f750ff00000-7f750ff01000 r--p 0015c000 08:01 532128                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
    7f750ff01000-7f750ff02000 rw-p 0015d000 08:01 532128                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
    7f750ff02000-7f750ff9d000 rw-p 00000000 00:00 0 
    7f750ff9d000-7f750ffa3000 r-xp 00000000 08:01 532067                     /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
    7f750ffa3000-7f75101a2000 ---p 00006000 08:01 532067                     /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
    7f75101a2000-7f75101a3000 r--p 00005000 08:01 532067                     /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
    7f75101a3000-7f75101a4000 rw-p 00006000 08:01 532067                     /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
    7f7510265000-7f751028f000 r-xp 00000000 08:01 532108                     /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
    7f751028f000-7f751048e000 ---p 0002a000 08:01 532108                     /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
    7f751048e000-7f751048f000 r--p 00029000 08:01 532108                     /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
    7f751048f000-7f7510491000 rw-p 0002a000 08:01 532108                     /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
    7f7510491000-7f75104d2000 rw-p 00000000 00:00 0 
    7f75104d2000-7f75104d5000 r-xp 00000000 08:01 48833                      /usr/lib/python2.7/lib-dynload/_heapq.so
    7f75104d5000-7f75106d4000 ---p 00003000 08:01 48833                      /usr/lib/python2.7/lib-dynload/_heapq.so
    7f75106d4000-7f75106d5000 r--p 00002000 08:01 48833                      /usr/lib/python2.7/lib-dynload/_heapq.so
    7f75106d5000-7f75106d7000 rw-p 00003000 08:01 48833                      /usr/lib/python2.7/lib-dynload/_heapq.so
    7f75106d7000-7f751073e000 r-xp 00000000 08:01 532118                     /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
    7f751073e000-7f751093d000 ---p 00067000 08:01 532118                     /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
    7f751093d000-7f751093e000 r--p 00066000 08:01 532118                     /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
    7f751093e000-7f7510942000 rw-p 00067000 08:01 532118                     /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
    7f7510942000-7f7510944000 rw-p 00000000 00:00 0 
    7f7510944000-7f7510958000 r-xp 00000000 08:01 48804                      /usr/lib/python2.7/lib-dynload/datetime.so
    7f7510958000-7f7510b57000 ---p 00014000 08:01 48804                      /usr/lib/python2.7/lib-dynload/datetime.so
    7f7510b57000-7f7510b58000 r--p 00013000 08:01 48804                      /usr/lib/python2.7/lib-dynload/datetime.so
    7f7510b58000-7f7510b5c000 rw-p 00014000 08:01 48804                      /usr/lib/python2.7/lib-dynload/datetime.so
    7f7510b5c000-7f7510caf000 r-xp 00000000 08:01 532106                     /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
    7f7510caf000-7f7510eae000 ---p 00153000 08:01 532106                     /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
    7f7510eae000-7f7510eb0000 r--p 00152000 08:01 532106                     /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
    7f7510eb0000-7f7510ebd000 rw-p 00154000 08:01 532106                     /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
    7f7510ebd000-7f7510ecf000 rw-p 00000000 00:00 0 
    7f7510ecf000-7f7510f08000 r-xp 00000000 08:01 533450                     /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
    7f7510f08000-7f7511107000 ---p 00039000 08:01 533450                     /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
    7f7511107000-7f7511108000 r--p 00038000 08:01 533450                     /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
    7f7511108000-7f751110c000 rw-p 00039000 08:01 533450                     /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
    7f751110c000-7f751110d000 rw-p 00000000 00:00 0 
    7f751110d000-7f7511296000 r--p 00000000 08:01 58562                      /usr/lib/locale/locale-archive
    7f7511296000-7f75112ab000 r-xp 00000000 08:01 2312                       /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f75112ab000-7f75114aa000 ---p 00015000 08:01 2312                       /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f75114aa000-7f75114ab000 r--p 00014000 08:01 2312                       /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f75114ab000-7f75114ac000 rw-p 00015000 08:01 2312                       /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f75114ac000-7f7511660000 r-xp 00000000 08:01 2327                       /lib/x86_64-linux-gnu/libc-2.15.so
    7f7511660000-7f751185f000 ---p 001b4000 08:01 2327                       /lib/x86_64-linux-gnu/libc-2.15.so
    7f751185f000-7f7511863000 r--p 001b3000 08:01 2327                       /lib/x86_64-linux-gnu/libc-2.15.so
    7f7511863000-7f7511865000 rw-p 001b7000 08:01 2327                       /lib/x86_64-linux-gnu/libc-2.15.so
    7f7511865000-7f751186a000 rw-p 00000000 00:00 0 
    7f751186a000-7f7511965000 r-xp 00000000 08:01 2400                       /lib/x86_64-linux-gnu/libm-2.15.so
    7f7511965000-7f7511b64000 ---p 000fb000 08:01 2400                       /lib/x86_64-linux-gnu/libm-2.15.so
    7f7511b64000-7f7511b65000 r--p 000fa000 08:01 2400                       /lib/x86_64-linux-gnu/libm-2.15.so
    7f7511b65000-7f7511b66000 rw-p 000fb000 08:01 2400                       /lib/x86_64-linux-gnu/libm-2.15.so
    7f7511b66000-7f7511b7c000 r-xp 00000000 08:01 2288                       /lib/x86_64-linux-gnu/libz.so.1.2.3.4
    7f7511b7c000-7f7511d7b000 ---p 00016000 08:01 2288                       /lib/x86_64-linux-gnu/libz.so.1.2.3.4
    7f7511d7b000-7f7511d7c000 r--p 00015000 08:01 2288                       /lib/x86_64-linux-gnu/libz.so.1.2.3.4
    7f7511d7c000-7f7511d7d000 rw-p 00016000 08:01 2288                       /lib/x86_64-linux-gnu/libz.so.1.2.3.4
    7f7511d7d000-7f7511f2f000 r-xp 00000000 08:01 2279                       /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    7f7511f2f000-7f751212e000 ---p 001b2000 08:01 2279                       /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    7f751212e000-7f7512149000 r--p 001b1000 08:01 2279                       /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    7f7512149000-7f7512154000 rw-p 001cc000 08:01 2279                       /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    7f7512154000-7f7512158000 rw-p 00000000 00:00 0 
    7f7512158000-7f75121ac000 r-xp 00000000 08:01 2393                       /lib/x86_64-linux-gnu/libssl.so.1.0.0
    7f75121ac000-7f75123ac000 ---p 00054000 08:01 2393                       /lib/x86_64-linux-gnu/libssl.so.1.0.0
    7f75123ac000-7f75123af000 r--p 00054000 08:01 2393                       /lib/x86_64-linux-gnu/libssl.so.1.0.0
    7f75123af000-7f75123b6000 rw-p 00057000 08:01 2393                       /lib/x86_64-linux-gnu/libssl.so.1.0.0
    7f75123b6000-7f75123b8000 r-xp 00000000 08:01 2283                       /lib/x86_64-linux-gnu/libutil-2.15.so
    7f75123b8000-7f75125b7000 ---p 00002000 08:01 2283                       /lib/x86_64-linux-gnu/libutil-2.15.so
    7f75125b7000-7f75125b8000 r--p 00001000 08:01 2283                       /lib/x86_64-linux-gnu/libutil-2.15.so
    7f75125b8000-7f75125b9000 rw-p 00002000 08:01 2283                       /lib/x86_64-linux-gnu/libutil-2.15.so
    7f75125b9000-7f75125bb000 r-xp 00000000 08:01 2406                                            

/lib/x86_64-linux-gnu/ld-2.15.so
    7f7512a2d000-7f7512b31000 rw-p 00000000 00:00 0 
    7f7512b62000-7f7512bea000 rw-p 00000000 00:00 0 
    7f7512bf7000-7f7512bf9000 rw-p 00000000 00:00 0 
    7f7512bf9000-7f7512bfa000 rwxp 00000000 00:00 0 
    7f7512bfa000-7f7512bfc000 rw-p 00000000 00:00 0 
    7f7512bfc000-7f7512bfd000 r--p 00022000 08:01 2260                       /lib/x86_64-linux-gnu/ld-2.15.so
    7f7512bfd000-7f7512bff000 rw-p 00023000 08:01 2260                       /lib/x86_64-linux-gnu/ld-2.15.so
    7ffcf454c000-7ffcf4585000 rw-p 00000000 00:00 0                          [stack]
    7ffcf459b000-7ffcf459d000 r-xp 00000000 00:00 0                          [vdso]
    ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
    Aborted (core dumped)

使用下面的代码，没有内存问题，但是下面的代码可以做什么，我的意思是进行分组和数据聚合

with open("data/petaJoined.csv", "r") as content:
    for line in content:
        #print line
        pass
     #do stuff with line` 
    content.close()

有人知道发生了什么吗？

其实我想达到Pandas read csv out of memory中显示的结果

也许会有解决方案？

请注意，我已经使用了按 block 读取 csv，但仍然存在内存错误

然后，我更改了 block 大小，以另一种方式拥有我的 bigData.py 文件

import pandas as pd
import numpy  as np
import sys, traceback, os
import etl2                                    # my self processing flow
reload(etl2)
def iter_chunks(n,df):
    while True:
        try:
           yield df.get_chunk(n)
        except StopIteration:
            break
cksize=5
try:
    dfs = pd.DataFrame()
    reader=pd.read_table( 'data/petaJoined.csv',
                          chunksize   = cksize,
                          low_memory  = False,
                          iterator    = True
                          )                    # choose as appropriate
    for chunk in iter_chunks(cksize,reader):
        temp=etl2.tb_createTopRankTable(chunk)
        dfs.append(temp)
    df=tb_createTopRankTable(dfs)
    #
    # for chunk in reader:
    #     pass
    # temp=tb_createTopRankTable(chunk)
    # dfs.append(temp)
    # df=tb_createTopRankTable(dfs)
except:
    traceback.print_exc(file=sys.stdout)

运行一段时间后仍然会出现段错误

def tb_createTopRankTable(df):
    try:
        key='name1'
        key2='name2'
        df2 = df.groupby([key,key2])['isError'].agg({ 'errorNum':  'sum','totalParcel': 'count' })
        df2['errorRate'] = df2['errorNum'] / df2['totalParcel']
        return df2

最佳答案

逐行阅读时基于您的代码段。

我假设 kb_2 是错误指示符，

groups={}
with open("data/petaJoined.csv", "r") as large_file:
    for line in large_file:
        arr=line.split('\t')
        #assuming this structure: ka,kb_1,kb_2,timeofEvent,timeInterval
        k=arr[0]+','+arr[1]
        if not (k in groups.keys())
            groups[k]={'record_count':0, 'error_sum': 0}
        groups[k]['record_count']=groups[k]['record_count']+1
        groups[k]['error_sum']=groups[k]['error_sum']+float(arr[2])
for k,v in groups.items:
    print ('{group}: {error_rate}'.format(group=k,error_rate=v['error_sum']/v['record_count']))

这段代码片段将所有分组存储在字典中，并在读取整个文件后计算错误率。

如果组的组合过多，会遇到内存不足的异常。

关于python - 读取 block 中的 csv 文件时出现内存不足错误，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/30255986/

文章推荐： python - 使用扭曲将文件读取到标准输出

文章推荐： c# - PayPal 集成 (IPN) - 真的有可能吗？

文章推荐： c# - 使用 Topshelf 进行集成测试以启动 C# Windows 服务

文章推荐： python - 如何防止 hasattr 检索属性值本身

SQL 查询导致我 sleep 不足
所以我正在为考试复习，并在 SQL 河(或荒地)中撞到了一块大石头我制作了以下表格并插入了以下数据: create table Permissions ( fileName VARCHAR(
JQueryUI 对话框 maxWidth 不足
我有一个使用 maxWidth 定义的 jqueryui 对话框。 $("#myDialog").dialog({ autoOpen: false, width: 'a
c - 如何使用平方根优化c中的循环(完美、丰富、不足)
注意:我遗漏了不相关的代码所以我目前正在研究 CCC 1996 P1，这个问题的全部目的是能够计算一个整数输入是完美数、不足数还是充数。我上面列出的代码可以工作，但是我认为它太慢了。该代码会迭代每个
r - R 中的关联规则 RAM 不足
已关闭。此问题需要 debugging details 。目前不接受答案。编辑问题以包含 desired behavior, a specific problem or error, and the
python - Redis 使用的 RAM 不足
我正在使用 Go 和 Redis 开发 API。问题是RAM使用不足，我找不到问题的根源。 TL;DR 版本有数百/数千个哈希对象。每个 1 KB 的对象(键+值)占用大约 0.5 MB 的 RAM
kubernetes - 由于 CPU 不足，Pod 处于挂起状态
在我的 GCE Kubernetes 集群上，我无法再创建 pod。 Warning FailedScheduling pod (www.caveconditions.com-f1be467e3
kubernetes - Amazon EKS Fargate中的 pod 不足
当我尝试在EKS Fargate群集上安装指标服务器时，它抛出错误: 0/4 nodes are available: 4 Insufficient pods. 按照以下说明从此处安装指标服务器:ht
ios - 为什么 iOS 终止后台应用程序而不是以不同方式处理 RAM 不足？
遍布this document Apple 提到 iOS 在某些情况下会终止应用程序，最常见的原因似乎是释放一些 RAM。这会导致未实现状态恢复的应用程序出现问题——用户正在处理和暂时离开的一些内容可
audio - Google Cloud Speech:配额组 token 不足
尝试处理一个10分钟的音频文件时出现以下错误。我刚刚开始使用Google Cloud产品，所以我是唯一访问此资源的人。我怎么可能超出配额？配额设置为其默认值，我认为我没有任何限制。还有其他原因吗？我
r - 对R中事物类型的全面考察； 'mode' 和 'class' 和 'typeof' 不足
R 语言让我感到困惑。实体有模式和类，但即使这样也不足以完全描述实体。这个answer说 In R every 'object' has a mode and a class. 所以我做了这些实验:
kubernetes - Openshift:没有与以下所有谓词匹配的可用节点::cpu 不足 (173)、MatchNodeSelector (5)
我在 west-1 有一个 Openshift v3 项目。在其中，我有一个运行良好的应用程序，但在 GitHub 提交代码中非常下游的内容后，该应用程序停止工作。问题在于制作 pod: No nod
kubernetes - Openshift:没有与以下所有谓词匹配的可用节点::cpu 不足 (173)、MatchNodeSelector (5)
我在 west-1 有一个 Openshift v3 项目。在其中，我有一个运行良好的应用程序，但在 GitHub 提交代码中非常下游的内容后，该应用程序停止工作。问题在于制作 pod: No nod
wolfram-mathematica - 我可以使用 Stackoverflow API 检查哪些 SO 回答者 sleep 不足？
在 how-do-i-access-the-stackoverflow-api-from-mathematica我概述了如何使用 SO API 让 Mathematica 制作一些有趣的顶级回答者声誉
node.js - 小型 Node.js 应用程序 Pod 的 GKE CPU 不足
所以在 GKE 上，我有一个 Node.js app，每个 pod 使用大约:CPU(cores): 5m, MEMORY: 100Mi 但是我只能为每个 Node 部署 1 个 pod。我使用的是
javascript - 消费者的服务 'AnalyticsDefaultGroup' 的配额 'USER-100s' 和限制 'analyticsreporting.googleapis.com' 的 token 不足
我正在使用 async.eachOfSeries 超过 300 个数组并请求一些 GA api，它工作正常但有时我会收到错误.. UnhandledPromiseRejectionWarning:错误
amazon-s3 - 0/3 个节点可用 : 1 node(s) had taints that the pod didn't tolerate, 2 cpu 不足。 MR3 hive
我正在尝试在 AWS ec2 上托管的 kubernetes 集群上使用 mr3 设置配置单元。当我运行命令 run-hive.sh 时，Hive 服务器启动，并且 master-DAg 被初始化，但
google-cloud-pubsub - 消费者 'administrator' 的服务 'CLIENT_PROJECT-100s' 的配额 'pubsub.googleapis.com' 和限制 'project_number:#' 的 token 不足
创建订阅时有时会出现以下错误: Insufficient tokens for quota 'administrator' and limit 'CLIENT_PROJECT-100s' of ser

太空狗

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

python - 读取 block 中的 csv 文件时出现内存不足错误