python - 在测试期间，Pickle 无法将对象存储在 django locmem 缓存中？

Question

有些事情让我有点困惑......

>>> from django.core.cache import get_cache
>>>
>>> cache = get_cache('django.core.cache.backends.locmem.LocMemCache')
>>>
>>> # Set the 'content' cache key to a string
>>> cache.set('content', 'a string')
>>> cache.get('content')
'a string'
>>>
>>> class TestObj(object):
...     pass
>>>
>>> a = TestObj()
>>> cache.set('content', a)
>>>
>>> # cache hasn't updated...
>>> cache.get('content')
'a string'
>>>
>>> cache.set('content', 1)
>>> # this is fine however..
>>> cache.get('content')
1
>>>

好的，所以缓存出于某种原因不接受对象。

# in locmem.py, set() method
try:
    pickled = pickle.dumps(new_value, pickle.HIGHEST_PROTOCOL)
    self._cache[key] = pickled
except pickle.PickleError:
    pass

这就是为什么，它显然会遇到 PickleError

>>> import pickle
>>> pickled = pickle.dumps(a, pickle.HIGHEST_PROTOCOL)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 396, in save_reduce
    save(cls)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <class 'TestObj'>: it's not found as __builtin__.TestObj

没问题，但为什么会这样？它在 python 控制台中工作得很好，但在 django shell 中却不行？

# Works fine in python shell...
>>> import pickle  
>>> class TestObj(object):
...     pass
... 
>>> testobj = TestObj()   
>>> pickled = pickle.dumps(testobj, pickle.HIGHEST_PROTOCOL)
>>> pickled
'\x80\x02c__main__\nTestObj\nq\x00)\x81q\x01}q\x02b.'
>>>

出现这个问题是因为我试图在缓存中存储一​​个 Mock() 对象以进行测试。不确定我是否以错误的方式解决这个问题......

Answer 1

问题在于 pickle 通过引用序列化类，那么您是否可以使用更好的序列化程序通过序列化类定义而不是通过引用来序列化类？然后你会 pickle 一个模拟对象，然后它会 pickle 类源代码，然后你就可以将它传递给 django 缓存。我是 dill 的作者，这是一个更好的序列化器……也是 klepto 的作者，这是一个缓存包……这正是我用来存储的SQL 表中、磁盘上或内存缓存中的任何对象。

基本上(不是尝试这个，而是根据我自己的缓存包的经验猜测它可以工作)，它应该像这样工作:

>>> from django.core.cache import get_cache
>>> import dill
>>>
>>> cache = get_cache('django.core.cache.backends.locmem.LocMemCache')
>>>
>>> # Set the 'content' cache key to a string
>>> cache.set('content', dill.dumps('a string'))
>>> dill.loads(cache.get('content'))
'a string'
>>>
>>> class TestObj(object):
...     pass
>>>
>>> a = TestObj()
>>> cache.set('content', dill.dumps(a))
>>>
>>> dill.loads(cache.get('content'))
<__main__.TestObj object at 0x10235e510>
>>>
>>> # this is pickling classes w/o using a reference
>>> dill.dumps(a)
'\x80\x02cdill.dill\n_create_type\nq\x00(cdill.dill\n_load_type\nq\x01U\x08TypeTypeq\x02\x85q\x03Rq\x04U\x07TestObjq\x05h\x01U\nObjectTypeq\x06\x85q\x07Rq\x08\x85q\t}q\n(U\r__slotnames__q\x0b]q\x0cU\n__module__q\rU\x08__main__q\x0eU\x07__doc__q\x0fNutq\x10Rq\x11)\x81q\x12}q\x13b.'
>>> # and here's using a reference, which is exactly how pickle does it
>>> dill.dumps(a, byref=True)
'\x80\x02c__main__\nTestObj\nq\x00)\x81q\x01}q\x02b.'

如果您想自己尝试，请在此处获取 dill(和 klepto):https://github.com/uqfoundation