python - 无法使用 "PicklingError"在 Windows 10 上启动 Celery worker

Question

我有一个简单的测试代码可以在 Linux 上成功运行，但它无法在我的 Windows 10 x64 计算机上运行。

当我尝试启动一个 celery worker 时，它提示无法恢复的错误:PicklingError。 ( celery 版本:3.1.20)

在我的 celery 配置中，我已将序列化设置为“json”，但它仍然没有任何帮助。

CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']

这是完整的错误信息:

[2016-02-09 15:11:48,532: ERROR/MainProcess] Unrecoverable error: PicklingError("Can't pickle <type 'module'>: it's not found as __builtin__.module",)

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\worker\__init__.py", line 206, in start
    self.blueprint.start(self)   
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\bootsteps.py", line 123, in start
    step.start(parent)
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\bootsteps.py", line 374, in start
    return self.obj.start()
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\concurrency\base.py", line 131, in start
    self.on_start()
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\concurrency\prefork.py", line 117, in on_start
    **self.options)
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 972, in __init__
    self._create_worker_process(i)
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 1068, in _create_worker_process
    w.start()
  File "C:\Python27\lib\site-packages\billiard\process.py", line 137, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\site-packages\billiard\forking.py", line 263, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\site-packages\billiard\py2\reduction.py", line 84, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 562, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 396, in save_reduce
    save(cls)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <type 'module'>: it's not found as __builtin__.module

Answer 1

我遇到了同样的问题。奇怪的是这个问题只存在于 Windows 上，在 Linux 上 Celery 运行没有任何问题。结果我需要将配置模块作为名称而不是对象传递:

app.config_from_object('celeryconfig')

代替

app.config_from_object(celeryconfig)

来自 Celery docs 的解释:

Tip

Using the name of a module is recommended as this means that the module doesn’t need to be serialized when the prefork pool is used. If you’re experiencing configuration pickle errors then please try using the name of a module instead.

显然，当 Celery 作为对象传递时，它需要 pickle 配置，但在 Windows 上失败。如果作为模块名称传递，它就可以工作。感谢@JoyLy 为我指明正确的方向!