python - 在 Flask 应用程序中开始使用 Celery 时出现 PicklingError

Question

我希望使用 Celery/RabbitMQ 在 Windows 7 和 Python 3.3 上的 Flask 应用程序中运行常规间隔任务。 Pip 安装:

billiard==3.3.0.10
celery==3.1.6

代码:

### celeryapp.py ###
from celery import Celery
from twend import config, app

def make_celery(app):
    celery = Celery(app.import_name, broker=app.config['CELERY_BROKER_URL'])
    celery.conf.update(app.config)
    TaskBase = celery.Task
    class ContextTask(TaskBase):
        abstract = True
        def __call__(self, *args, **kwargs):
            with app.app_context():
                return TaskBase.__call__(self, *args, **kwargs)
    celery.Task = ContextTask
    return celery

celery_pipe = make_celery(app)
celery_pipe.config_from_object(config)

### tasks.py ###
from celery import task
from twend.celeryapp import celery_pipe as celery

@celery.task()
def add_together(a, b):
    return a + b

### config.py ###
CELERY_BROKER_URL = 'amqp://'
CELERY_RESULT_BACKEND = 'amqp://'
CELERY_ACCEPT_CONTENT = ['json']

当我尝试启动 worker 时，我得到:

ERROR/MainProcess] Unrecoverable error: PicklingError(
"Can't pickle <class 'module'>: attribute lookup builtins.module failed",)
Traceback (most recent call last):
  File "c:\Python33\twend\lib\site-packages\celery\worker\__init__.py", line 212
, in start
    self.blueprint.start(self)
  File "c:\Python33\twend\lib\site-packages\celery\bootsteps.py", line 123, in s
tart
    step.start(parent)
  File "c:\Python33\twend\lib\site-packages\celery\bootsteps.py", line 373, in s
tart
    return self.obj.start()
  File "c:\Python33\twend\lib\site-packages\celery\concurrency\base.py", line 12
7, in start
    self.on_start()
  File "c:\Python33\twend\lib\site-packages\celery\concurrency\prefork.py", line
 112, in on_start
    **self.options)
  File "c:\Python33\twend\lib\site-packages\billiard\pool.py", line 966, in __in
it__
    self._create_worker_process(i)
  File "c:\Python33\twend\lib\site-packages\billiard\pool.py", line 1059, in _cr
eate_worker_process
    w.start()
  File "c:\Python33\twend\lib\site-packages\billiard\process.py", line 139, in s
tart
    self._popen = Popen(self)
  File "c:\Python33\twend\lib\site-packages\billiard\forking.py", line 263, in _
_init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "c:\Python33\twend\lib\site-packages\billiard\_reduction3.py", line 60, i
n dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'module'>: attribute lookup builtins.
module failed

(twend) c:\Python33\twend>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\Python33\twend\lib\site-packages\billiard\forking.py", line 457, in m
ain
    self = load(from_parent)
EOFError

这https://github.com/celery/django-celery/issues/228似乎是一个类似的问题，但已在我使用的 Celery 版本中修复。

我还读到 Windows 对 pickle 有限制，但我不确定实际被 pickle 的是什么。

Answer 1

Celery 使用酸洗来序列化任务和结果。我看到您已经将配置设置为仅接受 json 序列化，因此也许您可以通过添加更多配置行来解决您的问题:

CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'

这样就完全不用pickle了，Celery的序列化全部使用json来完成。这在理论上会导致性能下降，但我通过这种方式解决了我的 Flask 应用程序中的类似问题并且没有发现任何性能问题。 YMMV