对象和进程的Python内存消耗

Question

我写了下面的代码:

from hurry.size import size
from pysize import get_zise
import os
import psutil
def load_objects():
   process = psutil.Process(os.getpid())
   print "start method"
   process = psutil.Process(os.getpid())
   print "process consumes " + size(process.memory_info().rss)
   objects = make_a_call()
   print "total size of objects is " + (get_size(objects))
   print "process consumes " + size(process.memory_info().rss)
   print "exit method"

def main():
    process = psutil.Process(os.getpid())
    print "process consumes " + size(process.memory_info().rss)
    load_objects()
    print "process consumes " + size(process.memory_info().rss)

get_size() 使用 this 返回对象的内存消耗代码。

我得到以下打印件:

process consumes 21M
start method
total size of objects is 20M
process consumes 29M
exit method
process consumes 29M

1. 如果进程仅多消耗 8M，对象为何消耗 20M？
2. 如果我退出一个方法，内存不应该减少回 21，因为垃圾收集器将清除消耗的内存吗？

Answer 1

1. 这很可能是因为您的代码不准确。

这是一个具有相同问题的完整工作(python 2.7)示例(为了简单起见，我稍微更新了原始代码)

from hurry.filesize import size
from pysize import get_size
import os
import psutil


def make_a_call():
    return range(1000000)

def load_objects():
    process = psutil.Process(os.getpid())
    print "start method"
    process = psutil.Process(os.getpid())
    print"process consumes ", size(process.memory_info().rss)
    objects = make_a_call()
    # FIXME
    print "total size of objects is ", size(get_size(objects))
    print "process consumes ", size(process.memory_info().rss)
    print "exit method"

def main():
    process = psutil.Process(os.getpid())
    print "process consumes " + size(process.memory_info().rss)
    load_objects()
    print "process consumes " + size(process.memory_info().rss)


main()

这是输出:

process consumes 7M
start method
process consumes  7M
total size of objects is  30M
process consumes  124M
exit method
process consumes 124M

区别是~100Mb

这是代码的固定版本:

from hurry.filesize import size
from pysize import get_size
import os
import psutil


def make_a_call():
    return range(1000000)

def load_objects():
    process = psutil.Process(os.getpid())
    print "start method"
    process = psutil.Process(os.getpid())
    print"process consumes ", size(process.memory_info().rss)
    objects = make_a_call()
    print "process consumes ", size(process.memory_info().rss)
    print "total size of objects is ", size(get_size(objects))
    print "exit method"

def main():
    process = psutil.Process(os.getpid())
    print "process consumes " + size(process.memory_info().rss)
    load_objects()
    print "process consumes " + size(process.memory_info().rss)


main()

这是更新后的输出:

process consumes 7M
start method
process consumes  7M
process consumes  38M
total size of objects is  30M
exit method
process consumes 124M

你看出区别了吗？您在测量最终进程大小之前计算对象大小，这会导致额外的内存消耗。让我们检查它为什么会发生 - 这是来源 https://github.com/bosswissam/pysize/blob/master/pysize.py :

import sys
import inspect

def get_size(obj, seen=None):
    """Recursively finds size of objects in bytes"""
    size = sys.getsizeof(obj)
    if seen is None:
        seen = set()
    obj_id = id(obj)
    if obj_id in seen:
        return 0
    # Important mark as seen *before* entering recursion to gracefully handle
    # self-referential objects
    seen.add(obj_id)
    if hasattr(obj, '__dict__'):
        for cls in obj.__class__.__mro__:
            if '__dict__' in cls.__dict__:
                d = cls.__dict__['__dict__']
                if inspect.isgetsetdescriptor(d) or inspect.ismemberdescriptor(d):
                    size += get_size(obj.__dict__, seen)
                break
    if isinstance(obj, dict):
        size += sum((get_size(v, seen) for v in obj.values()))
        size += sum((get_size(k, seen) for k in obj.keys()))
    elif hasattr(obj, '__iter__') and not isinstance(obj, (str, bytes, bytearray)):
        size += sum((get_size(i, seen) for i in obj))
    return size

这里发生了很多事情!最值得注意的是，它将所有它看到的对象保存在一个集合中以解决循环引用。如果您删除该行，则在任何一种情况下都不会占用那么多内存。

1. 首先，此行为在很大程度上取决于您使用的是 CPython 还是其他软件。对于 CPython，这可能会发生，因为并不总是能够立即将内存返回给操作系统。

这是一个很好的article关于这个问题，引用:

If you create a large object and delete it again, Python has probably released the memory, but the memory allocators involved don’t necessarily return the memory to the operating system, so it may look as if the Python process uses a lot more virtual memory than it actually uses.