python - 为什么我的 Python 循环要消耗所有内存？

Question

我想在某个时间生成并保留一组元组。但我发现如果有足够的时间，该程序似乎会消耗所有内存。

我尝试过两种方法。一是删除新生成的变量，二是gc.collect()。但他们都不起作用。如果我只是生成而不保留元组，程序将消耗有限的内存。

生成并保留:gk.py

import gc
import time
from memory_profiler import profile
from random import sample
from sys import getsizeof


@profile
def loop(limit):
    t = time.time()
    i = 0
    A = set()
    while True:
        i += 1
        duration = time.time() - t
        a = tuple(sorted(sample(range(200), 100)))
        A.add(a)
        if not i % int(1e4):
            print('step {:.2e}...'.format(i))
        if duration > limit:
            print('done')
            break
        # method 1: delete the variables
#        del duration, a
        # method 2: use gc
#        gc.collect()
    memory = getsizeof(t) + getsizeof(i) + getsizeof(duration) + \
             getsizeof(a) + getsizeof(limit) + getsizeof(A)
    print('memory consumed: {:.2e}MB'.format(memory/2**20))
    pass


def main():
    limit = 300
    loop(limit)
    pass


if __name__ == '__main__':
    print('running...')
    main()

生成但不保留:gnk.py

import time
from memory_profiler import profile
from random import sample
from sys import getsizeof


@profile
def loop(limit):
    t = time.time()
    i = 0
    while True:
        i += 1
        duration = time.time() - t
        a = tuple(sorted(sample(range(200), 100)))
        if not i % int(1e4):
            print('step {:.2e}...'.format(i))
        if duration > limit:
            print('done')
            break
    memory = getsizeof(t) + getsizeof(i) + getsizeof(duration) + \
             getsizeof(a) + getsizeof(limit)
    print('memory consumed: {:.2e}MB'.format(memory/2**20))
    pass


def main():
    limit = 300
    loop(limit)
    pass


if __name__ == '__main__':
    print('running...')
    main()

在cmd/shell中使用“mprof”(需要模块memory_profiler)来检查内存使用情况

mprof run my_file.py
mprof plot

gk.py 的结果

memory consumed: 4.00e+00MB
Filename: gk.py

Line #    Mem usage    Increment   Line Contents
================================================
    12     32.9 MiB     32.9 MiB   @profile
    13                             def loop(limit):
    14     32.9 MiB      0.0 MiB       t = time.time()
    15     32.9 MiB      0.0 MiB       i = 0
    16     32.9 MiB      0.0 MiB       A = set()
    17     32.9 MiB      0.0 MiB       while True:
    18    115.8 MiB      0.0 MiB           i += 1
    19    115.8 MiB      0.0 MiB           duration = time.time() - t
    20    115.8 MiB      0.3 MiB           a = tuple(sorted(sample(range(200), 100)))
    21    115.8 MiB      2.0 MiB           A.add(a)
    22    115.8 MiB      0.0 MiB           if not i % int(1e4):
    23    111.8 MiB      0.0 MiB               print('step {:.2e}...'.format(i))
    24    115.8 MiB      0.0 MiB           if duration > limit:
    25    115.8 MiB      0.0 MiB               print('done')
    26    115.8 MiB      0.0 MiB               break
    27                                     # method 1: delete the variables
    28                             #        del duration, a
    29                                     # method 2: use gc
    30                             #        gc.collect()
    31                                 memory = getsizeof(t) + getsizeof(i) + getsizeof(duration) + \
    32    115.8 MiB      0.0 MiB                getsizeof(a) + getsizeof(limit) + getsizeof(A)
    33    115.8 MiB      0.0 MiB       print('memory consumed: {:.2e}MB'.format(memory/2**20))
    34    115.8 MiB      0.0 MiB       pass

gnk.py 的结果

memory consumed: 9.08e-04MB
Filename: gnk.py

Line #    Mem usage    Increment   Line Contents
================================================
    11     33.0 MiB     33.0 MiB   @profile
    12                             def loop(limit):
    13     33.0 MiB      0.0 MiB       t = time.time()
    14     33.0 MiB      0.0 MiB       i = 0
    15     33.0 MiB      0.0 MiB       while True:
    16     33.0 MiB      0.0 MiB           i += 1
    17     33.0 MiB      0.0 MiB           duration = time.time() - t
    18     33.0 MiB      0.1 MiB           a = tuple(sorted(sample(range(200), 100)))
    19     33.0 MiB      0.0 MiB           if not i % int(1e4):
    20     33.0 MiB      0.0 MiB               print('step {:.2e}...'.format(i))
    21     33.0 MiB      0.0 MiB           if duration > limit:
    22     33.0 MiB      0.0 MiB               print('done')
    23     33.0 MiB      0.0 MiB               break
    24                                 memory = getsizeof(t) + getsizeof(i) + getsizeof(duration) + \
    25     33.0 MiB      0.0 MiB                getsizeof(a) + getsizeof(limit)
    26     33.0 MiB      0.0 MiB       print('memory consumed: {:.2e}MB'.format(memory/2**20))
    27     33.0 MiB      0.0 MiB       pass

我有两个问题:

1. 两个程序消耗的内存都多于变量占用的内存。 “gk.py”消耗了115.8MB，其变量占用了4.00MB。 “gnk.py”消耗了33.0MB，其变量占用了9.08e-04MB。为什么程序消耗的内存比相应变量占用的内存多？

2. “gk.py”消耗的内存随时间线性增加。 “gnk.py”消耗的内存随着时间的推移而保持不变。为什么会出现这种情况？

如有任何帮助，我们将不胜感激。

Answer 1

鉴于集合的大小不断增加，最终会耗尽所有内存。

估计(来 self 的计算机):

10 seconds of code running ~ 5e4 tuples saved to the set
300 seconds of code running ~ 1.5e6 tuples saved to the set

1 tuple = 100 integers ~ 400bytes

total:

1.5e6 * 400bytes = 6e8bytes = 600MB filled in 300s