python - 用于有序查询集的 Django 查询集迭代器

Question

我想使用查询集迭代器迭代大型数据集。 Django 为此提供了 iterator()，但每次迭代都会访问数据库。我发现以下代码用于分块迭代 -

  def queryset_iterator(queryset, chunksize=1000):
    '''''
    Iterate over a Django Queryset ordered by the primary key
    This method loads a maximum of chunksize (default: 1000) rows in it's
    memory at the same time while django normally would load all rows in it's
    memory. Using the iterator() method only causes it to not preload all the
    classes.
    Note that the implementation of the iterator
    does not support ordered query sets.
    '''
    pk = 0
    last_pk = queryset.order_by('-pk').values_list('pk', flat=True).first()
    if last_pk is not None:
        queryset = queryset.order_by('pk')
        while pk < last_pk:
            for row in queryset.filter(pk__gt=pk)[:chunksize]:
                pk = row.pk
                yield row
            gc.collect()

这适用于无序查询集。是否有任何解决方案/解决方法可以在有序查询集上执行此操作？

Answer 1

这是我的，有排序功能。

顺便说一下，您使用的迭代器在修改查询集项目时有一个“永远循环”:删除或添加，甚至是一个项目。

下面的迭代器对 last_pk 没有无用的查询

def queryset_iterator(queryset, chunksize=10000, key=None):
    key = [key] if isinstance(key, str) else (key or ['pk'])
    counter = 0
    count = chunksize
    while count == chunksize:
        offset = counter - counter % chunksize
        count = 0
        for item in queryset.all().order_by(*key)[offset:offset + chunksize]:
            count += 1
            yield item
        counter += count
        gc.collect()