django - 如何修复 Django 的 get_or_create() 并发导致数据重复

Question

我有一个相当复杂的问题导致我的数据库中出现重复记录。

我在 nginx 1.0.5 和 uwsgi(4 名 worker )和 Django 1.4.5 上运行。问题是某些客户端对同一路径发出重复请求，如下面的 nginx 日志所示:

10.205.132.51 - - [26/Aug/2013:16:59:41 -0300] "GET /path/to/ HTTP/1.1" 499 0 "http://mydomain.com.br/path/" "Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20100101 Firefox/21.0"
10.205.132.51 - - [26/Aug/2013:16:59:41 -0300] "GET /path/to/ HTTP/1.1" 200 7372 "http://mydomain.com.br/path/" "Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20100101 Firefox/21.0"

这些请求是同时处理的，在像下面这个 View 这样的情况下，我进入了竞争条件，其中get_or_create 都找不到结果，但都创建了一个新对象:

with transaction.commit_on_success():
    f, created = cls.objects.get_or_create(
        key1=value1,
        key2=value2,
        defaults={...})

在你问之前，不，这两个键在数据库中不是unique_together。

两个请求都从 Django 返回 200 状态代码，但是 nginx 丢弃一个，导致 409 状态代码(冲突)。 transaction.commit_on_success() 部分是尝试减少，但没有解决问题。

我也试过一个基于缓存的锁，使用这个函数:

@contextmanager
def cache_exclusive(name, timeout=10):
    """ found at http://coffeeonthekeyboard.com/simple-out-of-process-lock-with-python-and-memcached-2-985/ """
    key = 'cache_lock:%s' % name
    lock = cache.add(key, True, timeout=timeout)  # Fails if key already exists.
    yield lock  # Tell the inner block if it acquired the lock.
    if lock:  # Only clear the lock if we had it.
        cache.delete(key)

以及具有这种用法的唯一名称:

    with cache_exclusive('key1 and key2') as granted:
        if not granted:
            return
        # do the get_or_create stuff...

但这也没有成功。您对如何处理这些重复请求有什么建议吗？

Answer 1

This method is atomic assuming correct usage, correct database configuration, and correct behavior of the underlying database. However, if uniqueness is not enforced at the database level for the kwargs used in a get_or_create call (see unique or unique_together), this method is prone to a race-condition which can result in multiple rows with the same parameters being inserted simultaneously.

If you are using MySQL, be sure to use the READ COMMITTED isolation level rather than REPEATABLE READ (the default), otherwise you may see cases where get_or_create will raise an IntegrityError but the object won’t appear in a subsequent get() call.

Finally, a word on using get_or_create() in Django views: please make sure to use it only in POST requests unless you have a good reason not to GET requests shouldn’t have any effect on data; use POST whenever a request to a page as a side effect on your data. For more, see Safe methods in the HTTP spec.

https://docs.djangoproject.com/en/dev/ref/models/querysets/#get-or-create