python - 在 Django 应用程序中线程化

Question

我正在尝试创建我称之为 scrobbler 的东西。任务是从队列中读取一个 Delicious 用户，获取他们所有的书签并将它们放入书签队列中。然后应该通过该队列，进行一些解析，然后将数据存储在数据库中。

这显然需要线程，因为大部分时间都花在等待 Delicious 响应，然后等待已添加书签的网站响应并通过某些 API 传递，如果一切都等待它是愚蠢的。

但是，我在处理线程时遇到了问题，并不断收到奇怪的错误，例如未定义数据库表。任何帮助表示赞赏:)

相关代码如下:

# relevant model #
class Bookmark(models.Model):
 account = models.ForeignKey( Delicious )
 url = models.CharField( max_length=4096 )
 tags = models.TextField()
 hash = models.CharField( max_length=32 )
 meta = models.CharField( max_length=32 )

# bookmark queue reading #
def scrobble_bookmark(account):
 try:
  bookmark = Bookmark.objects.all()[0]
 except Bookmark.DoesNotExist:
  return False

 bookmark.delete()

 tags = bookmark.tags.split(' ')
 user = bookmark.account.user

 for concept in Concepts.extract( bookmark.url ):
  for tag in tags:
   Concepts.relate( user, concept['name'], tag )

 return True

def scrobble_bookmarks(account):
 semaphore = Semaphore(10)
 for i in xrange(Bookmark.objects.count()):
  thread = Bookmark_scrobble(account, semaphore)
  thread.start()

class Bookmark_scrobble(Thread):
 def __init__(self, account, semaphore):
  Thread.__init__(self)
  self.account = account
  self.semaphore = semaphore

 def run(self):
  self.semaphore.acquire()
  try:
   scrobble_bookmark(self.account)
  finally:
   self.semaphore.release()

这是我得到的错误:

Exception in thread Thread-65:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner
    self.run()
  File     "/home/swizec/Documents/trees/bookmarklet_server/../bookmarklet_server/Scrobbler/Scrobbler.py", line 60, in run
    scrobble_bookmark(self.account)
  File     "/home/swizec/Documents/trees/bookmarklet_server/../bookmarklet_server/Scrobbler/Scrobbler.py", line 28, in scrobble_bookmark
    bookmark = Bookmark.objects.all()[0]
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 152, in __getitem__
    return list(qs)[0]
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 76, in __len__
self._result_cache.extend(list(self._iter))
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 231, in iterator
    for row in self.query.results_iter():
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 281, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 2373, in execute_sql
cursor.execute(sql, params)
  File "/usr/local/lib/python2.6/dist-packages/django/db/backends/sqlite3/base.py", line 193, in execute
    return Database.Cursor.execute(self, query, params)
OperationalError: no such table: Scrobbler_bookmark

PS:依赖于同一张表的所有其他测试都以优异的成绩通过。

Answer 1

您不能在 Django 的内存数据库(本例中为 sqlite3)中使用线程，请参阅 bug .这可能适用于 PostgreSQL 或 MySQL。

我建议使用 celeryd 之类的东西而不是线程，消息队列比线程更容易使用。