python - 删除查询集时Django ValidationError

Question

我正在编写一个 Django 命令来从我的应用程序中删除超过 x 天的数据。

使用以下内容进行过滤:

qs = Data.objects.filter(date_created__lte=timezone.now()-timedelta(days=days_del))


其中 days_del 为整数，date_created 为 DateTimeField。

当尝试打印此查询集或对其调用 .delete() 时，会导致 JSONDecodeError 和 ValidationError。我真的不明白为什么会发生这种情况，也不明白在这种情况下如何阻止它尝试解码 JSON 文件。

请注意，我正在使用 jsonfield pypi 包，并且数据模型有一个 JSONField。

可能是某些数据停滞并导致问题(请参阅回溯)，有没有办法忽略验证并继续删除？

Traceback (most recent call last):
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/jsonfield/fields.py", line 83, in pre_init
    return json.loads(value, **self.load_kwargs)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 464 (char 463)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/webapps/myproj/server/mirrors/management/commands/data_cleanup.py", line 39, in handle
    qs.delete()
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/query.py", line 616, in delete
    collector.collect(del_query)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/deletion.py", line 191, in collect
    reverse_dependency=reverse_dependency)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/deletion.py", line 89, in add
    if not objs:
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/query.py", line 254, in __bool__
    self._fetch_all()
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/query.py", line 1118, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/query.py", line 63, in __iter__
    obj = model_cls.from_db(db, init_list, row[model_fields_start:model_fields_end])
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/base.py", line 583, in from_db
    new = cls(*values)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/base.py", line 502, in __init__
    _setattr(self, field.attname, val)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/jsonfield/subclassing.py", line 43, in __set__
    obj.__dict__[self.field.name] = self.field.pre_init(value, obj)
  File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/jsonfield/fields.py", line 85, in pre_init
    raise ValidationError(_("Enter valid JSON"))
django.core.exceptions.ValidationError: ['Enter valid JSON']

我正在同时删除大量数据，也许有更好的方法来处理这个问题。无论如何，修复旧的陈旧数据不是这里的一个选项。

谢谢

<小时/>

这是命令文件:

from django.core.management.base import BaseCommand, CommandError
from oauth2_provider.models import Application
from django.utils import timezone

import pytz
from datetime import timedelta

from confluence_core.models import Data



class Command(BaseCommand):
    help = 'Delete data older than given days'

    def add_arguments(self, parser):
        parser.add_argument("-d", "--days", type=int, dest='days', required=True, help="Days limit")
        parser.add_argument("-c", "--confirm", action='store_true', dest='confirm', default=False, required=False, help="Confirm before deletion")

    def handle(self, *args, **options):

        days_del = options['days']
        do_delete = False

        qs = Data.objects.filter(date_created__lte=timezone.now()-timedelta(days=days_del))

        if qs.count() > 0:

            if options['confirm']:
                print(f"{qs.count():,} data entries will be deleted.")
                ret = input("Confirm ? (y/n)\n")
                if ret in ['y', 'Y', 'yes']:
                    do_delete = True
            else:
                do_delete = True

            if do_delete is True:
                print(f"Deleting {qs.count():,} data entries...")
                qs.delete()
            else:
                print("Not deleting anything.")

        else:
            print("No data to delete.")

        print("Done.")

Answer 1

鉴于ORM在删除时强制加载查询集，参见:

File "/root/virtualenvs/myproj-prod/lib/python3.6/site-packages/django/db/models/deletion.py", line 89, in add
if not objs:

我能想到的唯一解决方法(不需要 fork 或猴子修补任何东西)是首先更新整个查询集，以便将 jsonfield 设置为有效的内容，即:

qs.update(name_of_the_field={})
qs.delete()

但这并不能阻止数据不一致的其他问题，因此真正的解决方案显然是清理整个数据集。