django - 操作失败 : database error when threading in MongoEngine/PyMongo

Question

我有一个函数可以从网站读取数据，对其进行处理，然后将其加载到 MongoDB 中。当我在没有线程的情况下运行它时，它工作正常，但是一旦我设置了只调用这个函数的 celery 任务，我经常会收到以下错误:“OperationFailure:数据库错误:未经授权的 db:dbname 锁定类型:-1”

这有点奇怪，因为如果我在多个终端上运行非 celery 版本，我根本不会收到此错误。

我怀疑这与没有与 Mongo 的开放连接有关，尽管在我的代码中我在每次 Mongo 调用之前打开一个。

具体的异常(exception)情况如下:

Task twitter[a974bfcc-d6ca-4baf-b36f-cae9143ce2d9] raised exception: OperationFailure(u'database error: unauthorized db:data lock type:-1 client:68.193.49.9',)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/celery/execute/trace.py", line 36, in trace
  return cls(states.SUCCESS, retval=fun(*args, **kwargs))
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/celery/app/task/__init__.py", line 232, in __call__
  return self.run(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/celery/app/__init__.py", line 172, in run
  return fun(*args, **kwargs)
File "/djangoblog/network/tasks.py", line 40, in twitter
  n_twitter.GetTweetsTwitter(user)
File "/djangoblog/network/twitter.py", line 255, in GetTweetsTwitter
  id = SaveTweet(user, network, tweet)
File "/djangoblog/network/twitter.py", line 150, in SaveTweet
  if mmo.Moment.objects(user=user.id,source_id=id,network=network.id).count() == 0:
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mongoengine/queryset.py", line 933, in count
  return self._cursor.count(with_limit_and_skip=True)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mongoengine/queryset.py", line 563, in _cursor
  self._cursor_obj = self._collection.find(self._query,
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/mongoengine/queryset.py", line 493, in _collection
  if self._collection_obj.name not in db.collection_names():
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pymongo/database.py", line 361, in collection_names
  names = [r["name"] for r in results]
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pymongo/cursor.py", line 703, in next
  if len(self.__data) or self._refresh():
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pymongo/cursor.py", line 666, in _refresh
  self.__uuid_subtype))
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pymongo/cursor.py", line 628, in __send_message self.__tz_aware)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pymongo/helpers.py", line 101, in _unpack_response error_object["$err"])
OperationFailure: database error: unauthorized db:data lock type:-1 client:68.193.49.9

抱歉格式化，但如果您查看以 mmo.Moment 开头的行，就会在调用之前打开一个连接。

做一些研究，它看起来好像与 PyMongo 中处理线程的方式有关 - http://api.mongodb.org/python/1.5.1/faq.html#how-does-connection-pooling-work-in-pymongo - 我可能需要开始关闭连接，但我希望 MongoEngine 会这样做..

Answer 1

这可能是因为您在启动新连接时没有调用 db.authenticate() 并且在 MongoDB 上使用 auth。

关于线程的关闭，我建议确保您使用连接池并让驱动程序管理池(手动调用 close() 或类似的方法会导致很多痛苦)。

更多信息见the note在 pymongo 文档中关于在多线程环境中使用 authenticate()。