python-3.x - celery 花 API 结果未返回

Question

调用 Flower API's /api/task/apply/* 只是挂起而没有返回结果，即使任务已根据 Web UI 处理并成功。

重现错误:

# hangs even though task succeeded in logs
curl -d '{"args": [1, 2]}' -H "Content-Type: application/json" -X POST http://0.0.0.0:5000/api/task/apply/tasks.add

即使 Flower UI 将其显示为成功，但通过 ID 检查任务只会将其显示为待处理，而且我还在工作日志中看到了任务结果。

curl -X GET http://0.0.0.0:5000/api/task/result/02fa2656-c92e-44e6-a2af-c04e666e3f89
# {"task-id": "02fa2656-c92e-44e6-a2af-c04e666e3f89", "state": "PENDING"}

我怀疑这个日志行表明了这个问题:celery-worker_1 | [2018-03-06 15:25:15,160: DEBUG/MainProcess] pidbox 收到方法 enable_events() [reply_to:None ticket:None]。

在花侧还有一个 Traceback(请参阅下面的日志)，当我按 Ctrl+C 退出上面的挂起 curl 命令时会发生这种情况。

tasks.py

from celery import Celery

app = Celery('tasks', broker='redis://celery-redis:6379//0', backend='redis://celery-redis:6379//0')


@app.task
def add(a, b):
    return a + b

日志

celery-worker_1  | [2018-03-06 15:24:36,271: DEBUG/MainProcess] pidbox received method active() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'afe22ec9-5cde-330c-9306-c8dea457eecb'} ticket:48fb973b-51a1-4ad4-9482-04e31b19fb14]
celery-worker_1  | [2018-03-06 15:24:36,282: DEBUG/MainProcess] pidbox received method reserved() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'd9911fb2-c681-3fe2-bdb2-248c14bf8351'} ticket:b69f2b6c-b7d7-4d26-a398-970ef22e4280]
celery-worker_1  | [2018-03-06 15:24:36,288: DEBUG/MainProcess] pidbox received method revoked() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '31402263-0a94-3226-89ee-2eb9e0a138b9'} ticket:0a33227a-9b3e-4d9b-bbfd-fd7aa108403b]
celery-worker_1  | [2018-03-06 15:24:36,303: DEBUG/MainProcess] pidbox received method conf(with_defaults=False) [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'bb3879ef-1424-3c02-a3eb-6c33faf985d2'} ticket:6ef872d6-d512-4393-bcdb-9ad8982afdd5]
celery-flower_1  | [W 180306 15:24:37 control:44] 'stats' inspect method failed
celery-flower_1  | [W 180306 15:24:37 control:44] 'active_queues' inspect method failed
celery-flower_1  | [W 180306 15:24:37 control:44] 'registered' inspect method failed
celery-flower_1  | [W 180306 15:24:37 control:44] 'scheduled' inspect method failed
celery-flower_1  | [D 180306 15:24:37 app:62] Successfully updated worker cache
celery-worker_1  | [2018-03-06 15:24:40,160: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-worker_1  | [2018-03-06 15:24:40,160: INFO/MainProcess] Events of group {task} enabled by remote.
celery-worker_1  | [2018-03-06 15:24:45,159: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-worker_1  | [2018-03-06 15:24:50,159: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-worker_1  | [2018-03-06 15:24:55,158: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-worker_1  | [2018-03-06 15:25:00,160: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-worker_1  | [2018-03-06 15:25:05,158: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-flower_1  | [D 180306 15:25:08 tasks:126] Invoking a task 'tasks.add' with '[1, 2]' and '{}'
celery-worker_1  | [2018-03-06 15:25:08,153: INFO/MainProcess] Received task: tasks.add[59ad1f38-dfab-4f09-8dfe-772789b3b630]  
celery-worker_1  | [2018-03-06 15:25:08,154: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fef7f6fa0d0> (args:('tasks.add', '59ad1f38-dfab-4f09-8dfe-772789b3b630', {'lang': 'py', 'task': 'tasks.add', 'id': '59ad1f38-dfab-4f09-8dfe-772789b3b630', 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '59ad1f38-dfab-4f09-8dfe-772789b3b630', 'parent_id': None, 'argsrepr': '[1, 2]', 'kwargsrepr': '{}', 'origin': 'gen1@bbc42c108b2f', 'reply_to': 'b64af5c4-8366-3299-bcd7-6eecb9eaf952', 'correlation_id': '59ad1f38-dfab-4f09-8dfe-772789b3b630', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}}, b'[[1, 2], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
celery-worker_1  | [2018-03-06 15:25:08,157: DEBUG/MainProcess] Task accepted: tasks.add[59ad1f38-dfab-4f09-8dfe-772789b3b630] pid:8
celery-worker_1  | [2018-03-06 15:25:08,159: INFO/ForkPoolWorker-1] Task tasks.add[59ad1f38-dfab-4f09-8dfe-772789b3b630] succeeded in 0.002012488999753259s: 3
celery-worker_1  | [2018-03-06 15:25:10,159: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-worker_1  | [2018-03-06 15:25:15,160: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
celery-flower_1  | Exception in thread Thread-2:
celery-flower_1  | Traceback (most recent call last):
celery-flower_1  |   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
celery-flower_1  |     self.run()
celery-flower_1  |   File "/usr/lib/python3.6/threading.py", line 864, in run
celery-flower_1  |     self._target(*self._args, **self._kwargs)
celery-flower_1  |   File "/usr/lib/python3.6/site-packages/flower/api/tasks.py", line 154, in wait_results
celery-flower_1  |     self.finish(response)
celery-flower_1  |   File "/usr/lib/python3.6/site-packages/tornado/web.py", line 1018, in finish
celery-flower_1  |     self.flush(include_footers=True)
celery-flower_1  |   File "/usr/lib/python3.6/site-packages/tornado/web.py", line 973, in flush
celery-flower_1  |     start_line, self._headers, chunk, callback=callback)
celery-flower_1  |   File "/usr/lib/python3.6/site-packages/tornado/http1connection.py", line 400, in write_headers
celery-flower_1  |     future = self._write_future = Future()
celery-flower_1  |   File "/usr/lib/python3.6/asyncio/events.py", line 678, in get_event_loop
celery-flower_1  |     return get_event_loop_policy().get_event_loop()
celery-flower_1  |   File "/usr/lib/python3.6/asyncio/events.py", line 584, in get_event_loop
celery-flower_1  |     % threading.current_thread().name)
celery-flower_1  | RuntimeError: There is no current event loop in thread 'Thread-2'.

Dockerfile

FROM alpine:3.6

# System setup
ENV USER app
RUN addgroup -g 1000 app && adduser -D -G app -u 1000 app
ENV dir /home/app
ENV LC_ALL=en_US.utf-8

# System dependencies
RUN apk add --no-cache python3 python3-dev
RUN python3 -m ensurepip
RUN pip3 install --upgrade pip setuptools
RUN apk add --no-cache openssl-dev \
  gcc \
  musl-dev \
  git

WORKDIR ${dir}

# Update pip
RUN pip3 install --upgrade pip setuptools

# App dependencies
RUN pip3 install celery[redis]==4.1.0 \
  flower==0.9.2

# File upload
COPY tasks.py ${dir}/tasks.py
RUN chown -R ${USER}: ${dir}

# App setup
USER ${USER}

# Flower port
EXPOSE 5000

CMD true

docker-compose.yml

version: '3'
services:
  celery-redis:
    image: redis:4-alpine
  celery-worker:
    build: .
    image: example/worker
    command: celery -A tasks worker --loglevel=DEBUG --concurrency 1
    depends_on:
      - celery-redis
    restart: always
  celery-flower:
    build: .
    image: example/flower
    command: celery flower -A tasks --port=5000 --debug
    depends_on:
      - celery-redis
    ports:
      - '5000:5000'

Answer 1

我遇到了同样的问题，我能够解决它。

在 DockerFile 中添加以下内容:

运行 pip install tornado==4.5.2

4.5.2 是一个旧版本，可以很好地调用 apply 调用，最新版本的 tornado 导致了这个错误。

希望对您有所帮助，如果您遇到更多问题，请告诉我，我现在正在运行环境。