python - 应用引擎 : Running out of memory while traversing a table

Question

我有以下代码来循环 RawEmailModel 类型的所有实体并更新计数器:

def update_emails(cursor=None, stats = {}):
    BATCH_SIZE = 100
    if not cursor:
        # Start of the job
        pass
    next_cursor = cursor
    more = True
    try:
        while more:
            rawEmails, next_cursor, more = RawEmailModel.query().fetch_page(BATCH_SIZE, start_cursor=next_cursor)
            ndb.get_context().clear_cache()
            for rawEmail in rawEmails:
                try:
                    stats[rawEmail.userId] += 1
                except Exception:
                    stats[rawEmail.userId] = 0              
            logging.debug(stats)
        logging.debug("Done counting")
    except Exception as e:
        logging.error(e)

我正在根据我在https://stackoverflow.com/a/12108891/2448805中读到的内容清除ndb缓存但是，我仍然收到错误消息，提示我内存不足:

20:21:55.240 {u'104211720960924551911': 45622, u'105605183894399744988': 0, u'114651439835375426353': 2, u'112308898027744263560': 667, u'112185522275060884315': 804}

F 20:22:01.389 Exceeded soft private memory limit of 128 MB with 153 MB after servicing 14 requests total

W 20:22:01.390 While handling this request, the process that handled this request was found to be using too much memory and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application.

我不明白为什么当我不断清除循环顶部的缓存时，内存仍然不足？谢谢!

Answer 1

看起来您有大量的 RawEmailModel 条目，并且您的统计字典正在增长并达到内存限制。您的 ndb.get_context().clear_cache() 不会在这里帮助您。

您可能必须想出另一个模型来保存计数，例如 RawEmailCounterModel 以 userId 和total_count 作为字段，并不断从 while 循环更新它，而不是使用您的统计字典来进行计数。

至少这会帮助您解决内存不足的问题。但这可能性能不佳。