python - Google App Engine NDB 数据库损坏了吗？

Question

我不断收到随机错误，例如:

suspended generator _get_tasklet(context.py:329) raised ProtocolBufferDecodeError(corrupted)

或

suspended generator put(context.py:796) raised ValueError(Expecting , delimiter: line 1 column 440 (char 440))

或

suspended generator put(context.py:796) raised ValueError(Invalid \escape: line 1 column 18002 (char 18002))

或

suspended generator _get_tasklet(context.py:329) raised ProtocolBufferDecodeError(truncated)

几天前一切正常，我没有做任何更改。当我重新启动我的应用程序时，一切正常大约五分钟，直到我得到一个

suspended generator _get_tasklet(context.py:329) raised ProtocolBufferDecodeError(corrupted)

在那之后，我在每个数据库放置或获取时都遇到了其他错误之一。导致错误的表和代码每次都不同。我不知道从哪里开始，因为错误每次都出现在新的地方。这些只是常规的数据库放置和获取，例如

ndbstate = NdbStateJ.get_by_id(self.screen_name)

或

ndbstate.put()

Google 搜索无法为我指明任何特定方向。有任何想法吗？错误

Expecting , delimiter: line 1 column 440 (char 440)

可能是因为某些表中的某些字段类型是JSON。但是为什么突然呢？

所以也许我在某处没有正确转义，比如使用 r'{...}'，但是如果某处有错误的条目，如果我无法查询，我该如何解决？为什么它会破坏所有查询的整个表？为什么它是随机的。每次都不是同一个查询。

这是一个表格的例子

class NdbStateJ(ndb.Model):
    last_id = ndb.IntegerProperty()
    last_search_id = ndb.IntegerProperty()
    last_geo_id = ndb.IntegerProperty()
    mytweet_num = ndb.IntegerProperty()
    mentions_processed = ndb.JsonProperty()
    previous_follower_responses = ndb.JsonProperty()
    my_tweets_tweeted = ndb.JsonProperty()
    responses_already_used = ndb.JsonProperty()
    num_followed_by_cyborg = ndb.IntegerProperty(default=0)
    num_did_not_follow_back = ndb.IntegerProperty(default=0)
    language_model_vector = ndb.FloatProperty(repeated=True)
    follow_wait_counter = ndb.IntegerProperty(default=0)

下面是一个创建表的例子

ndbstate = NdbStateJ(id=screen_name,
last_id = 37397357946732541,
last_geo_id = 37397357946732541,
last_search_id = 0,
mytweet_num = 0,
mentions_processed = [],
previous_follower_responses = [],
my_tweets_tweeted = [],
responses_already_used= [],
language_model_vector = [])
ndbstate.put()

Answer 1

这是数据库中格式错误的 JSON 导致的问题。我不知道为什么问题突然开始到处发生；也许谷歌方面发生了一些变化，或者我没有充分检查，新用户能够输入格式错误的数据。谁知道。

为了修复它，我从 https://stackoverflow.com/users/1011633/nizz 中获得灵感回应App Engine return JSON from JsonProperty , https://stackoverflow.com/users/1709587/mark-amery回应How to escape special characters in building a JSON string? , 和 https://stackoverflow.com/users/1639625/tobias-k回应How do I automatically fix an invalid JSON string? .

我将 ndb.JsonProperty() 替换为 ExtendedJsonProperty，其中扩展版本看起来类似于下面的代码。

import json
from google.appengine.ext import ndb 
import logging
logging.getLogger().setLevel(logging.DEBUG)
import re

class ExtendedJsonProperty(ndb.BlobProperty):
    # Inspired by https://stackoverflow.com/questions/18576556/app-engine-return-json-from-jsonproperty
    def _to_base_type(self, value):
        logging.debug('Dumping value '+str(value))
        try:
            return json.dumps(value) 
        except Exception as e:
            logging.warning(('trying to fix error dumping from database: ') +str(e))
            return fix_json(value,json.dumps)

    def _from_base_type(self, value):
        # originally return json.loads(value)
        logging.debug('Loading value '+str(value))
        try:
            return json.loads(value)
        except Exception as e:
            logging.warning(('trying to fix error loading from database: ') +str(e))
            return fix_json(value,json.loads)        

def fix_json(s,json_fun):
    for _i in range(len(s)):
        try:
            result = json_fun(s)   # try to parse...
            return result                    
        except Exception as e:  
            logging.debug('Exception for json loads: '+str(e))          
            if 'delimiter' in str(e):
                # E.g.: "Expecting , delimiter: line 34 column 54 (char 1158)"
                logging.debug('Escaping quote to fix.')
                s = escape_quote(s,e)
            elif 'escape' in str(e):
                # E.g.: "Invalid \escape: line 1 column 9 (char 9)"
                logging.debug('Removing invalid escape to fix.')
                s = remove_invalid_escape(s)
            else:
                break
    return json_fun('{}')

def remove_invalid_escape(value):
    # Inspired by https://stackoverflow.com/questions/19176024/how-to-escape-special-characters-in-building-a-json-string
    return re.sub(r'\\(?!["\\/bfnrt])', '', value)

def escape_quote(s,e):
    # Inspired by https://stackoverflow.com/questions/18514910/how-do-i-automatically-fix-an-invalid-json-string
    # "Expecting , delimiter: line 34 column 54 (char 1158)"
    # position of unexpected character after '"'
    unexp = int(re.findall(r'\(char (\d+)\)', str(e))[0])
    # position of unescaped '"' before that
    unesc = s.rfind(r'"', 0, unexp)
    s = s[:unesc] + r'\"' + s[unesc+1:]
    # position of corresponding closing '"' (+2 for inserted '\')
    closg = s.find(r'"', unesc + 2)
    if closg + 2 < len(s):
        print closg, len(s)
        s = s[:closg] + r'\"' + s[closg+1:]
    return s