python - GQL 查询对象内存泄漏

Question

我有一个长时间运行的后台进程，可以解析几十万行 CSV。我注意到该进程存在内存泄漏，偶尔会导致任务达到其软内存限制并终止。我已将代码部分缩小为以下代码块:

class BaseModel(db.Model):
    _keyNamespace = 'MyApp.Models'

    @classmethod
    def get_by_item_id(cls, id):
        key = "%s_%d" % (cls._keyNamespace, id)
        item = CacheStrategy.get(key)
        if not item:
            query = cls.gql("WHERE Id = :1", id)
            item = query.get()
            del query

        return item

我已将其精简到最基本的部分，但它仍然导致查询对象保留在内存中。示例 GC 引用转储包含在注释的末尾，显示每 200 个订单批处理步骤后 Query 和 Query_Filter 计数增加 200。如果我摆脱查询调用，这当然会消失。

我的问题是，为什么这是泄漏的查询引用，我如何让它遵守删除并删除查询引用？

我试过将其设为实例方法(没有区别)。下面的引用计数跟踪:

INFO     2011-10-17 16:29:39,158 orderparser.py:151] Putting a 200 unit batch of orders, 0.335000 seconds from start
DEBUG    2011-10-17 16:29:40,315 memleaker.py:20] Top Mem Leaks
DEBUG    2011-10-17 16:29:40,334 memleaker.py:22]     356306 Property
DEBUG    2011-10-17 16:29:40,334 memleaker.py:22]     356305 PropertyValue
DEBUG    2011-10-17 16:29:40,334 memleaker.py:22]      74410 Path
DEBUG    2011-10-17 16:29:40,334 memleaker.py:22]      74408 Path_Element
DEBUG    2011-10-17 16:29:40,334 memleaker.py:22]      45127 PropertyValue_ReferenceValue
DEBUG    2011-10-17 16:29:40,334 memleaker.py:22]      45127 PropertyValue_ReferenceValuePathElement
DEBUG    2011-10-17 16:29:40,334 memleaker.py:22]      43822 Reference
DEBUG    2011-10-17 16:29:40,335 memleaker.py:22]      30595 EntityProto
DEBUG    2011-10-17 16:29:40,335 memleaker.py:22]        320 ProtocolMessage
DEBUG    2011-10-17 16:29:40,335 memleaker.py:22]        217 Query
DEBUG    2011-10-17 16:29:40,335 memleaker.py:22]        209 Query_Filter
DEBUG    2011-10-17 16:29:40,335 memleaker.py:22]         55 NOT_PROVIDED
DEBUG    2011-10-17 16:29:40,335 memleaker.py:22]         34 Index_Property
DEBUG    2011-10-17 16:29:40,335 memleaker.py:22]         28 ExtendableProtocolMessage
DEBUG    2011-10-17 16:29:40,336 memleaker.py:22]         18 CompositeIndex
INFO     2011-10-17 16:29:40,644 orderparser.py:151] Putting a 200 unit batch of orders, 1.821000 seconds from start
DEBUG    2011-10-17 16:29:41,930 memleaker.py:20] Top Mem Leaks
DEBUG    2011-10-17 16:29:41,948 memleaker.py:22]     356506 Property
DEBUG    2011-10-17 16:29:41,948 memleaker.py:22]     356505 PropertyValue
DEBUG    2011-10-17 16:29:41,948 memleaker.py:22]      74410 Path
DEBUG    2011-10-17 16:29:41,948 memleaker.py:22]      74408 Path_Element
DEBUG    2011-10-17 16:29:41,948 memleaker.py:22]      45127 PropertyValue_ReferenceValue
DEBUG    2011-10-17 16:29:41,948 memleaker.py:22]      45127 PropertyValue_ReferenceValuePathElement
DEBUG    2011-10-17 16:29:41,948 memleaker.py:22]      43822 Reference
DEBUG    2011-10-17 16:29:41,951 memleaker.py:22]      30595 EntityProto
DEBUG    2011-10-17 16:29:41,951 memleaker.py:22]        417 Query
DEBUG    2011-10-17 16:29:41,951 memleaker.py:22]        409 Query_Filter
DEBUG    2011-10-17 16:29:41,951 memleaker.py:22]        320 ProtocolMessage
DEBUG    2011-10-17 16:29:41,951 memleaker.py:22]         55 NOT_PROVIDED
DEBUG    2011-10-17 16:29:41,951 memleaker.py:22]         34 Index_Property
DEBUG    2011-10-17 16:29:41,951 memleaker.py:22]         28 ExtendableProtocolMessage
DEBUG    2011-10-17 16:29:41,953 memleaker.py:22]         18 CompositeIndex
INFO     2011-10-17 16:29:42,276 orderparser.py:151] Putting a 200 unit batch of orders, 3.450000 seconds from start
DEBUG    2011-10-17 16:29:43,565 memleaker.py:20] Top Mem Leaks
DEBUG    2011-10-17 16:29:43,585 memleaker.py:22]     356706 Property
DEBUG    2011-10-17 16:29:43,585 memleaker.py:22]     356705 PropertyValue
DEBUG    2011-10-17 16:29:43,585 memleaker.py:22]      74410 Path
DEBUG    2011-10-17 16:29:43,585 memleaker.py:22]      74408 Path_Element
DEBUG    2011-10-17 16:29:43,585 memleaker.py:22]      45127 PropertyValue_ReferenceValue
DEBUG    2011-10-17 16:29:43,585 memleaker.py:22]      45127 PropertyValue_ReferenceValuePathElement
DEBUG    2011-10-17 16:29:43,585 memleaker.py:22]      43822 Reference
DEBUG    2011-10-17 16:29:43,586 memleaker.py:22]      30595 EntityProto
DEBUG    2011-10-17 16:29:43,586 memleaker.py:22]        617 Query
DEBUG    2011-10-17 16:29:43,586 memleaker.py:22]        609 Query_Filter
DEBUG    2011-10-17 16:29:43,586 memleaker.py:22]        320 ProtocolMessage
DEBUG    2011-10-17 16:29:43,586 memleaker.py:22]         55 NOT_PROVIDED
DEBUG    2011-10-17 16:29:43,586 memleaker.py:22]         34 Index_Property
DEBUG    2011-10-17 16:29:43,586 memleaker.py:22]         28 ExtendableProtocolMessage
DEBUG    2011-10-17 16:29:43,588 memleaker.py:22]         18 CompositeIndex

Answer 1

我无法使用您的引用计数代码和下面的一个小片段(在 shell.appspot.com 或新应用程序上)重现此内容:

from google.appengine.ext import db
import logging
import sys
import types

def get_refcounts():
    d = {}
    # collect all classes
    for m in sys.modules.values():
        for sym in dir(m):
            o = getattr (m, sym)
            if type(o) is types.ClassType:
                d[o] = sys.getrefcount (o)
    # sort by refcount
    pairs = map (lambda x: (x[1],x[0]), d.items())
    pairs.sort()
    pairs.reverse()
    return pairs

def print_top(num = 15):
    print 'Top Mem Leaks'
    for n, c in get_refcounts()[:num]:
        print '%10d %s' % (n, c.__name__)

class TestModel(db.Model):
  id = db.IntegerProperty()


print_top()

q = TestModel.gql("WHERE id = :1", 1)
item = q.get()
del q

print_top()

您的环境中似乎有某些东西持有对已执行查询的引用。您使用的是 appstats 还是其他开发或调试工具？你能创建一个最小的再现案例来展示你观察到的行为吗？