celery - 带有 Python 3 的 Librabbitmq 2.0.0 给出了 TypeError : can't pickle memoryview objects

Question

我正在使用 git repo 的最新主分支 https://github.com/celery/librabbitmq和安装 librabbitmq==2.0.0按照自述文件中的说明针对 Python 3.6

使用开发版

您可以通过执行以下操作来克隆存储库:

$ git clone git://github.com/celery/librabbitmq.git

然后通过执行以下操作安装它:

$ cd librabbitmq
$ make install        # or make develop

这工作正常(在操作系统中安装某些用于 c 编译的二进制文件之后)，但是当我制作一个小 a+b 时添加任务并使用 add.delay(2,2) 调用它它失败并出现以下错误。抬头一看，Celery 4 使用 json 作为序列化器，所以很明显不是因为如果 pickle 序列化

从 librabbitmq 更改为 pyamqp 代理工作正常

MacOS 和 Ubuntu 16 中的确切情况相同

[2018-04-30 23:40:02,956: CRITICAL/MainProcess] Unrecoverable error: SystemError(' returned a result with an error set',) Traceback (most recent call last): File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/messaging.py", line 624, in _receive_callback return on_m(message) if on_m else self.receive(decoded, message) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 570, in on_task_received callbacks, File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/strategy.py", line 145, in task_message_handler handle(req) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/worker.py", line 221, in _process_task_sem return self._quick_acquire(self._process_task, req) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/async/semaphore.py", line 62, in acquire callback(*partial_args, **partial_kwargs) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/worker.py", line 226, in _process_task req.execute_using_pool(self.pool) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/request.py", line 531, in execute_using_pool correlation_id=task_id, File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/concurrency/base.py", line 155, in apply_async **options) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/billiard/pool.py", line 1486, in apply_async self._quick_put((TASK, (result._job, None, func, args, kwds))) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 813, in send_job body = dumps(tup, protocol=protocol) TypeError: can't pickle memoryview objects

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/worker.py", line 203, in start self.blueprint.start(self) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start step.start(parent) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/bootsteps.py", line 370, in start return self.obj.start() File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 320, in start blueprint.start(self) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start step.start(parent) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 596, in start c.loop(*c.loop_args()) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/celery/worker/loops.py", line 88, in asynloop next(loop) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/async/hub.py", line 354, in create_loop cb(*cbargs) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/transport/base.py", line 236, in on_readable reader(loop) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/kombu/transport/base.py", line 218, in _read drain_events(timeout=0) File "/Users/somghosh/.virtualenvs/ctdb/lib/python3.6/site-packages/librabbitmq-2.0.0-py3.6-macosx-10.6-intel.egg/librabbitmq/init.py", line 227, in drain_events self._basic_recv(timeout) SystemError: returned a result with an error set

Answer 1

不建议将此库用作 celery 的 rabbitmq 代理。相反，请尝试 py-amqp。这得到了更多的维护和更少的错误。