个人中心
文章发布
退出
作者热门文章
- c - 在位数组中找到第一个零
- linux - Unix 显示有关匹配两种模式之一的文件的信息
- 正则表达式替换多个文件
- linux - 隐藏来自 xtrace 的命令
25
4
我正在处理一个 2.5 GB 的 csv 文件。 2.5 GB 的表如下所示:
columns=[ka,kb_1,kb_2,timeofEvent,timeInterval]
0:'3M' '2345' '2345' '2014-10-5',3000
1:'3M' '2958' '2152' '2015-3-22',5000
2:'GE' '2183' '2183' '2012-12-31',515
3:'3M' '2958' '2958' '2015-3-10',395
4:'GE' '2183' '2285' '2015-4-19',1925
5:'GE' '2598' '2598' '2015-3-17',1915
我想对 ka 和 kb_1 进行分组以获得如下结果:
columns=[ka,kb,errorNum,errorRate,totalNum of records]
'3M','2345',0,0%,1
'3M','2958',1,50%,2
'GE','2183',1,50%,2
'GE','2598',0,0%,1
(错误记录的定义:当kb_1 != kb_2时，对应的记录被视为异常记录)
我的电脑是 ubuntu 12.04，有 16 GB 内存 和 free -m 返回
total used free shared buffers cached
Mem: 112809 14476 98333 0 128 10823
-/+ buffers/cache: 3524 109285
Swap:
0 0 0
我的 python 文件名为 bigData.py
import pandas as pd
import numpy as np
import sys,traceback,os
cksize=98333 # or 1024, either chunk size didn't work at all
try:
dfs = pd.DataFrame()
reader=pd.read_table('data/petaJoined.csv', chunksize=cksize)
for chunk in reader:#when executed this line,error occur!
pass
#temp=tb_createTopRankTable(chunk)
#dfs.append(temp)
#df=tb_createTopRankTable(dfs)
except:
traceback.print_exc(file=sys.stdout)
ipdb> pd.__version__
'0.16.0'
我使用以下命令来监控内存使用情况:
top
ps -C python -o %cpu,%mem,cmd
由于崩溃大约需要 2 秒，所以我可以看到 mem 的使用率在某个时候达到了 90%，并且 CPU使用率达到100%
当我执行 python bigData.py 时，会生成以下错误:
/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module dateutil was already imported from /usr/local/lib/python2.7/dist-packages/dateutil/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
from pkg_resources import resource_stream
/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module pytz was already imported from /usr/local/lib/python2.7/dist-packages/pytz/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
from pkg_resources import resource_stream
Traceback (most recent call last):
File "bigData.py", line 10, in <module>
for chunk in reader:
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 691, in __iter__
yield self.read(self.chunksize)
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 715, in read
ret = self._engine.read(nrows)
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1164, in read
data = self._reader.read(nrows)
File "pandas/parser.pyx", line 758, in pandas.parser.TextReader.read (pandas/parser.c:7411)
File "pandas/parser.pyx", line 792, in pandas.parser.TextReader._read_low_memory (pandas/parser.c:7819)
File "pandas/parser.pyx", line 833, in pandas.parser.TextReader._read_rows (pandas/parser.c:8268)
File "pandas/parser.pyx", line 820, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:8142)
File "pandas/parser.pyx", line 1758, in pandas.parser.raise_parser_error (pandas/parser.c:20728)
CParserError: Error tokenizing data. C error: out of memory
Segmentation fault (core dumped)
或
/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module dateutil was already imported from /usr/local/lib/python2.7/dist-packages/dateutil/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
from pkg_resources import resource_stream
/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module pytz was already imported from /usr/local/lib/python2.7/dist-packages/pytz/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
from pkg_resources import resource_stream
Traceback (most recent call last):
File "bigData.py", line 10, in <module>
for chunk in reader:
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 691, in __iter__
yield self.read(self.chunksize)
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 715, in read
ret = self._engine.read(nrows)
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1164, in read
data = self._reader.read(nrows)
File "pandas/parser.pyx", line 758, in pandas.parser.TextReader.read (pandas/parser.c:7411)
File "pandas/parser.pyx", line 792, in pandas.parser.TextReader._read_low_memory (pandas/parser.c:7819)
File "pandas/parser.pyx", line 833, in pandas.parser.TextReader._read_rows (pandas/parser.c:8268)
File "pandas/parser.pyx", line 820, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:8142)
File "pandas/parser.pyx", line 1758, in pandas.parser.raise_parser_error (pandas/parser.c:20728)
CParserError: Error tokenizing data. C error: out of memory
*** glibc detected *** python: free(): invalid pointer: 0x00007f750d2a4c0e ***
====== Backtrace: ========
/lib/x86_64-linux-gnu/libc.so.6(+0x7db26)[0x7f7511529b26]
/usr/local/lib/python2.7/dist-packages/pandas/parser.so(+0x4d5a1)[0x7f750d29d5a1]
/usr/local/lib/python2.7/dist-packages/pandas/parser.so(parser_cleanup+0x15)[0x7f750d29de45]
/usr/local/lib/python2.7/dist-packages/pandas/parser.so(parser_free+0x9)[0x7f750d29e039]
/usr/local/lib/python2.7/dist-packages/pandas/parser.so(+0xb43e)[0x7f750d25b43e]
....
python(PyDict_SetItem+0x49)[0x577749]
python(_PyModule_Clear+0x149)[0x4cafb9]
python(PyImport_Cleanup+0x477)[0x4cb4f7]
python(Py_Finalize+0x18e)[0x549f0e]
python(Py_Main+0x3bc)[0x56b56c]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f75114cd76d]
python[0x41bb11]
======= Memory map: ========
00400000-00670000 r-xp 00000000 08:01 26612 /usr/bin/python2.7
0086f000-00870000 r--p 0026f000 08:01 26612 /usr/b.......
008d9000-008eb000 rw-p 00000000 00:00 0
01ddb000-036f7000 rw-p 00000000 00:00 0 [heap]
7f748c179000-7f74cc17a000 rw-p 00000000 00:00 0
7f7504000000-7f7504021000 rw-p 00000000 00:00 0
7f7504021000-7f7508000000 ---p 00000000 00:00 0
7f750bf83000-7f750c285000 rw-p 00000000 00:00 0
7f750c285000-7f750c586000 rw-p 00000000 00:00 0
7f750c586000-7f750c707000 rw-p 00000000 00:00 0
7f750c707000-7f750c711000 r-xp 00000000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
7f750c711000-7f750c911000 ---p 0000a000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
7f750c911000-7f750c912000 r--p 0000a000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
7f750c912000-7f750c913000 rw-p 0000b000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
7f750c913000-7f750c914000 rw-p 00000000 00:00 0
7f750c914000-7f750c918000 r-xp 00000000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f750c918000-7f750cb17000 ---p 00004000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f750cb17000-7f750cb18000 r--p 00003000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f750cb18000-7f750cb19000 rw-p 00004000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f750cb19000-7f750cb34000 r-xp 00000000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
7f750cb34000-7f750cd33000 ---p 0001b000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
7f750cd33000-7f750cd34000 r--p 0001a000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
7f750cd34000-7f750cd38000 rw-p 0001b000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
7f750cd38000-7f750d039000 rw-p 00000000 00:00 0
7f750d039000-7f750d04e000 r-xp 00000000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so
7f750d04e000-7f750d24e000 ---p 00015000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so
7f750d24e000-7f750d24f000 r--p 00015000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so
7f750d24f000-7f750d250000 rw-p 00016000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so
7f750d250000-7f750d2a9000 r-xp 00000000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so
7f750d2a9000-7f750d4a8000 ---p 00059000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so
7f750d4a8000-7f750d4a9000 r--p 00058000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so
7f750d4a9000-7f750d4af000 rw-p 00059000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so
7f750d4af000-7f750d591000 r-xp 00000000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f750d591000-7f750d790000 ---p 000e2000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f750d790000-7f750d798000 r--p 000e1000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f750d798000-7f750d79a000 rw-p 000e9000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7f750d79a000-7f750d7af000 rw-p 00000000 00:00 0
7f750d7af000-7f750d7f1000 r-xp 00000000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so
7f750d7f1000-7f750d9f1000 ---p 00042000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so
7f750d9f1000-7f750d9f3000 r--p 00042000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so
7f750d9f3000-7f750d9f4000 rw-p 00044000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so
7f750d9f4000-7f750da2d000 r-xp 00000000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
7f750da2d000-7f750dc2c000 ---p 00039000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
7f750dc2c000-7f750dc2d000 r--p 00038000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
7f750dc2d000-7f750dc31000 rw-p 00039000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
7f750dc31000-7f750dc7d000 r-xp 00000000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so
7f750dc7d000-7f750de7c000 ---p 0004c000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so
7f750de7c000-7f750de7d000 r--p 0004b000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so
7f750de7d000-7f750de86000 rw-p 0004c000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so
7f750de86000-7f750de87000 rw-p 00000000 00:00 0
7f750de87000-7f750debc000 r-xp 00000000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so
7f750debc000-7f750e0bb000 ---p 00035000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so
7f750e0bb000-7f750e0bc000 r--p 00034000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so
7f750e0bc000-7f750e0c0000 rw-p 00035000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so
7f750e0c0000-7f750e295000 r-xp 00000000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so
7f750e295000-7f750e494000 ---p 001d5000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so
7f750e494000-7f750e495000 r--p 001d4000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so
7f750e495000-7f750e4a9000 rw-p 001d5000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so
7f750e4a9000-7f750e4ac000 rw-p 00000000 00:00 0
7f750e4ac000-7f750e4b2000 r-xp 00000000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so
7f750e4b2000-7f750e6b1000 ---p 00006000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so
7f750e6b1000-7f750e6b2000 r--p 00005000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so
7f750e6b2000-7f750e6b4000 rw-p 00006000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so
7f750e6b4000-7f750e782000 r-xp 00000000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so
7f750e782000-7f750e981000 ---p 000ce000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so
7f750e981000-7f750e982000 r--p 000cd000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so
7f750e982000-7f750e990000 rw-p 000ce000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so
7f750e990000-7f750e992000 rw-p 00000000 00:00 0
7f750e992000-7f750ea8f000 r-xp 00000000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
7f750ea8f000-7f750ec8e000 ---p 000fd000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
7f750ec8e000-7f750ec8f000 r--p 000fc000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
7f750ec8f000-7f750eca1000 rw-p 000fd000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
7f750eca1000-7f750eca4000 rw-p 00000000 00:00 0
7f750eca4000-7f750ecc5000 r-xp 00000000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so
7f750ecc5000-7f750eec4000 ---p 00021000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so
7f750eec4000-7f750eec5000 r--p 00020000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so
7f750eec5000-7f750eec9000 rw-p 00021000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so
7f750eec9000-7f750eeca000 rw-p 00000000 00:00 0
7f750eeca000-7f750ef24000 r-xp 00000000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
7f750ef24000-7f750f123000 ---p 0005a000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
7f750f123000-7f750f124000 r--p 00059000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
7f750f124000-7f750f15c000 rw-p 0005a000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
7f750f15c000-7f750f15d000 rw-p 00000000 00:00 0
7f750f15d000-7f750f166000 r-xp 00000000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
7f750f166000-7f750f365000 ---p 00009000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
7f750f365000-7f750f366000 r--p 00008000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
7f750f366000-7f750f367000 rw-p 00009000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
7f750f367000-7f750f368000 r-xp 00000000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so
7f750f368000-7f750f567000 ---p 00001000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so
7f750f567000-7f750f568000 r--p 00000000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so
7f750f568000-7f750f569000 rw-p 00001000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so
7f750f569000-7f750f588000 r-xp 00000000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so
7f750f588000-7f750f787000 ---p 0001f000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so
7f750f787000-7f750f788000 r--p 0001e000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so
7f750f788000-7f750f791000 rw-p 0001f000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so
7f750f791000-7f750f907000 r-xp 00000000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
7f750f907000-7f750fb06000 ---p 00176000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
7f750fb06000-7f750fb07000 r--p 00175000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
7f750fb07000-7f750fb08000 rw-p 00176000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
7f750fb08000-7f750fba4000 rw-p 00000000 00:00 0
7f750fba4000-7f750fd01000 r-xp 00000000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
7f750fd01000-7f750ff00000 ---p 0015d000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
7f750ff00000-7f750ff01000 r--p 0015c000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
7f750ff01000-7f750ff02000 rw-p 0015d000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
7f750ff02000-7f750ff9d000 rw-p 00000000 00:00 0
7f750ff9d000-7f750ffa3000 r-xp 00000000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
7f750ffa3000-7f75101a2000 ---p 00006000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
7f75101a2000-7f75101a3000 r--p 00005000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
7f75101a3000-7f75101a4000 rw-p 00006000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
7f7510265000-7f751028f000 r-xp 00000000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
7f751028f000-7f751048e000 ---p 0002a000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
7f751048e000-7f751048f000 r--p 00029000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
7f751048f000-7f7510491000 rw-p 0002a000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
7f7510491000-7f75104d2000 rw-p 00000000 00:00 0
7f75104d2000-7f75104d5000 r-xp 00000000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so
7f75104d5000-7f75106d4000 ---p 00003000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so
7f75106d4000-7f75106d5000 r--p 00002000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so
7f75106d5000-7f75106d7000 rw-p 00003000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so
7f75106d7000-7f751073e000 r-xp 00000000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
7f751073e000-7f751093d000 ---p 00067000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
7f751093d000-7f751093e000 r--p 00066000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
7f751093e000-7f7510942000 rw-p 00067000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
7f7510942000-7f7510944000 rw-p 00000000 00:00 0
7f7510944000-7f7510958000 r-xp 00000000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so
7f7510958000-7f7510b57000 ---p 00014000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so
7f7510b57000-7f7510b58000 r--p 00013000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so
7f7510b58000-7f7510b5c000 rw-p 00014000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so
7f7510b5c000-7f7510caf000 r-xp 00000000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
7f7510caf000-7f7510eae000 ---p 00153000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
7f7510eae000-7f7510eb0000 r--p 00152000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
7f7510eb0000-7f7510ebd000 rw-p 00154000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
7f7510ebd000-7f7510ecf000 rw-p 00000000 00:00 0
7f7510ecf000-7f7510f08000 r-xp 00000000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
7f7510f08000-7f7511107000 ---p 00039000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
7f7511107000-7f7511108000 r--p 00038000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
7f7511108000-7f751110c000 rw-p 00039000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
7f751110c000-7f751110d000 rw-p 00000000 00:00 0
7f751110d000-7f7511296000 r--p 00000000 08:01 58562 /usr/lib/locale/locale-archive
7f7511296000-7f75112ab000 r-xp 00000000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f75112ab000-7f75114aa000 ---p 00015000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f75114aa000-7f75114ab000 r--p 00014000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f75114ab000-7f75114ac000 rw-p 00015000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f75114ac000-7f7511660000 r-xp 00000000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so
7f7511660000-7f751185f000 ---p 001b4000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so
7f751185f000-7f7511863000 r--p 001b3000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so
7f7511863000-7f7511865000 rw-p 001b7000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so
7f7511865000-7f751186a000 rw-p 00000000 00:00 0
7f751186a000-7f7511965000 r-xp 00000000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so
7f7511965000-7f7511b64000 ---p 000fb000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so
7f7511b64000-7f7511b65000 r--p 000fa000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so
7f7511b65000-7f7511b66000 rw-p 000fb000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so
7f7511b66000-7f7511b7c000 r-xp 00000000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f7511b7c000-7f7511d7b000 ---p 00016000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f7511d7b000-7f7511d7c000 r--p 00015000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f7511d7c000-7f7511d7d000 rw-p 00016000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f7511d7d000-7f7511f2f000 r-xp 00000000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f7511f2f000-7f751212e000 ---p 001b2000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f751212e000-7f7512149000 r--p 001b1000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f7512149000-7f7512154000 rw-p 001cc000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f7512154000-7f7512158000 rw-p 00000000 00:00 0
7f7512158000-7f75121ac000 r-xp 00000000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f75121ac000-7f75123ac000 ---p 00054000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f75123ac000-7f75123af000 r--p 00054000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f75123af000-7f75123b6000 rw-p 00057000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f75123b6000-7f75123b8000 r-xp 00000000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so
7f75123b8000-7f75125b7000 ---p 00002000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so
7f75125b7000-7f75125b8000 r--p 00001000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so
7f75125b8000-7f75125b9000 rw-p 00002000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so
7f75125b9000-7f75125bb000 r-xp 00000000 08:01 2406
/lib/x86_64-linux-gnu/ld-2.15.so
7f7512a2d000-7f7512b31000 rw-p 00000000 00:00 0
7f7512b62000-7f7512bea000 rw-p 00000000 00:00 0
7f7512bf7000-7f7512bf9000 rw-p 00000000 00:00 0
7f7512bf9000-7f7512bfa000 rwxp 00000000 00:00 0
7f7512bfa000-7f7512bfc000 rw-p 00000000 00:00 0
7f7512bfc000-7f7512bfd000 r--p 00022000 08:01 2260 /lib/x86_64-linux-gnu/ld-2.15.so
7f7512bfd000-7f7512bff000 rw-p 00023000 08:01 2260 /lib/x86_64-linux-gnu/ld-2.15.so
7ffcf454c000-7ffcf4585000 rw-p 00000000 00:00 0 [stack]
7ffcf459b000-7ffcf459d000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted (core dumped)
使用下面的代码，没有内存问题，但是下面的代码可以做什么，我的意思是进行分组和数据聚合
with open("data/petaJoined.csv", "r") as content:
for line in content:
#print line
pass
#do stuff with line`
content.close()
有人知道发生了什么吗？
其实我想达到Pandas read csv out of memory中显示的结果
也许会有解决方案？
请注意，我已经使用了按 block 读取 csv，但仍然存在内存错误
然后，我更改了 block 大小，以另一种方式拥有我的 bigData.py 文件
import pandas as pd
import numpy as np
import sys, traceback, os
import etl2 # my self processing flow
reload(etl2)
def iter_chunks(n,df):
while True:
try:
yield df.get_chunk(n)
except StopIteration:
break
cksize=5
try:
dfs = pd.DataFrame()
reader=pd.read_table( 'data/petaJoined.csv',
chunksize = cksize,
low_memory = False,
iterator = True
) # choose as appropriate
for chunk in iter_chunks(cksize,reader):
temp=etl2.tb_createTopRankTable(chunk)
dfs.append(temp)
df=tb_createTopRankTable(dfs)
#
# for chunk in reader:
# pass
# temp=tb_createTopRankTable(chunk)
# dfs.append(temp)
# df=tb_createTopRankTable(dfs)
except:
traceback.print_exc(file=sys.stdout)
运行一段时间后仍然会出现段错误
def tb_createTopRankTable(df):
try:
key='name1'
key2='name2'
df2 = df.groupby([key,key2])['isError'].agg({ 'errorNum': 'sum','totalParcel': 'count' })
df2['errorRate'] = df2['errorNum'] / df2['totalParcel']
return df2
最佳答案
逐行阅读时基于您的代码段。
我假设 kb_2 是错误指示符，
groups={}
with open("data/petaJoined.csv", "r") as large_file:
for line in large_file:
arr=line.split('\t')
#assuming this structure: ka,kb_1,kb_2,timeofEvent,timeInterval
k=arr[0]+','+arr[1]
if not (k in groups.keys())
groups[k]={'record_count':0, 'error_sum': 0}
groups[k]['record_count']=groups[k]['record_count']+1
groups[k]['error_sum']=groups[k]['error_sum']+float(arr[2])
for k,v in groups.items:
print ('{group}: {error_rate}'.format(group=k,error_rate=v['error_sum']/v['record_count']))
这段代码片段将所有分组存储在字典中，并在读取整个文件后计算错误率。
如果组的组合过多，会遇到内存不足的异常。
关于python - 读取 block 中的 csv 文件时出现内存不足错误，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/30255986/
25
4
0
我的应用程序从一个有 5 个选项卡的选项卡栏 Controller 开始。一开始，第一个出现了它的名字，但其他四个没有名字，直到我点击它们。然后根据用户使用的语言显示名称。如何在选项卡栏出现之前设置选
我有嵌套数组 json 对象(第 1 层、第 2 层和第 3 层)。我的问题是数据表没有出现。任何相关的 CDN 均已导入。该表仅显示部分。我引用了很多网站，但都没有解决我的问题。 之前我使用标准表来
我正在尝试设置要显示的 Parse PFLoginViewController。这是我的一个 View Controller 的类。 import UIKit import Parse import
我遇到了这个问题，我绘制的对象没有出现在 GUI 中。我知道它正在被处理，因为数据被推送到日志文件。但是，图形没有出现。 这是我的一些代码: public static void main(Strin
我有一个树状图，其中包含出现这样的词...... TreeMap occurrence = new TreeMap (); 字符串 = 单词 整数 = 出现次数。 我如何获得最大出现次数 - 整数，
因此，我提示用户输入变量。如果变量小于 0 且大于 10。如果用户输入 10，我想要求用户再次输入数字。我问时间的时候输入4，它说你输入错误。但在第二次尝试时效果很好。例如:如果我输入 25，它会打印
我已经用 css overflow 属性做了一个例子。在这个例子中我遇到了一个溢出滚动的问题。滚动条出现了，但没有工作意味着每当将光标移动到滚动条时，在这个滚动条不活动的时间。我对此一无所知，所以请帮
我现在正在做一个元素。当您单击一个元素时，会出现以下信息，我想知道如何在您单击下一个元素而不重新单击同一元素时使其消失....例如，我的元素中有披萨，我想单击肉披萨看到浇头然后点击奶酪披萨看到浇头和肉
我有一个路由器模块，它将主题与正则表达式进行比较，并将出现的事件与一致的键掩码链接起来。 (它是一个简单的 url 路由过滤，如 symfony http://symfony.com/doc/curr
这个问题在这里已经有了答案: 9年前关闭。 Possible Duplicate: mysql_fetch_array() expects parameter 1 to be resource, bo
我在底部有一个带有工具栏的 View ，我正在使用 NavigationLink 导航到该 View 。但是当 View 出现时，工具栏显示得有点太低了。大约半秒钟后，它突然跳到位。它只会在应用程序启
我试图在我的应用程序上为背景音乐添加一个 AVAudioPlayer，我正在主屏幕上启动播放器，尝试在应用程序打开时开始播放但出现意外行为... 它播放并立即不断创建新玩家并播放这些玩家，因此同时播放
这是获取一个数字，获取其阶乘并将其加倍，但是由于基本情况，如果您输入 0，它会给出 2 作为答案，因此为了绕过它，我使用了 if 语句，但收到错误输入“if”时解析错误。如果你们能提供帮助，我真的很感
暂停期间抛出异常 android.os.DeadObjectException 在 android.os.BinderProxy.transactNative( native 方法) 在 androi
我已经为猜词游戏编写了一些代码。它从用户输入中读取字符并在单词中搜索该字符；根据字符是否在单词中，程序返回并控制一些变量。 代码如下: import java.util.Random; import
我是自动化领域的新手。这是我的简单 TestNG 登录代码，当我以 TestNG 身份运行该代码时，它会出现 java.lang.NullPointerException，双击它会突出显示我导航到 U
我是c#程序员，我习惯了c#的封装语法和其他东西。但是现在，由于某些原因，我应该用java写一些东西，我现在正在练习java一天!我要创建一个为我自己创建一个虚拟项目，以便让自己更熟悉 Java 的
我正在使用 Intellij，我的源类是 main.com.coding，我的资源文件是 main.com.testing。我将 spring.xml 文件放入资源文件中。 我的测试类位于 test.
我想要我的tests folder separate到我的应用程序代码。我的项目结构是这样的 myproject/ myproject/ myproject.py moduleon
这个问题已经有答案了: What is a NullPointerException, and how do I fix it? (12 个回答) 已关闭 6 年前。 因此，我尝试比较 2 个值，一个
我是一名优秀的程序员,十分优秀！