python - Celery worker 收到任务但没有执行

Question

我尝试在 上使用 celery=4.2.0、Python=3.6 和 Rabbitmq=3.7.9 执行简单任务>windows 10，任务被工作程序接收，但没有执行并保持未确认状态。

任务减速:

@app.task()
def simpleTask(*args):
    logging.info('{}'.format(args[0]))

任务执行命令:

simpleTask.s('1').apply_async()

我尝试使用 -P gevent(version=1.4.0) 和 eventlet(version=0.24.1) 运行应用程序，但没有成功。虽然使用solo选项时任务可以正确执行，但对于我的最终目标，工作人员在单线程环境中运行任务不符合我的要求。

celery worker 命令:

argv = [
        'worker',
        '-l=DEBUG',
        '-P=eventlet',
        '--without-gossip',
        '--without-mingle',
        '--without-heartbeat',
        '-Ofair',
    ]
app.worker_main(argv)

输出:

[2019-03-10 23:08:18,612: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2019-03-10 23:08:18,618: DEBUG/MainProcess] | Worker: Building graph...
[2019-03-10 23:08:18,619: DEBUG/MainProcess] | Worker: New boot order: {StateDB, Timer, Hub, Pool, Autoscaler, Beat, Consumer}
[2019-03-10 23:08:18,938: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2019-03-10 23:08:18,939: DEBUG/MainProcess] | Consumer: Building graph...
[2019-03-10 23:08:18,951: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Agent, Events, Heart, Mingle, Gossip, Tasks, Control, event loop}

[2019-03-10 23:08:18,962: DEBUG/MainProcess] | Worker: Starting Pool
 -------------- celery@DESKTOP-IEMN7E0 v4.2.1 (windowlicker)
[2019-03-10 23:08:18,962: DEBUG/MainProcess] ^-- substep ok
---- **** ----- 
[2019-03-10 23:08:18,962: DEBUG/MainProcess] | Worker: Starting Consumer
--- * ***  * -- Windows-10-10.0.17134-SP0 2019-03-10 23:08:18
[2019-03-10 23:08:18,962: DEBUG/MainProcess] | Consumer: Starting Connection
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         workflowEngine.Main.Server.tasks:0x2771cf629e8
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     mongodb://
- *** --- * --- .> concurrency: 4 (eventlet)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery


[tasks]
  . PanelTask
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . workflowEngine.Main.Server.tasks.finalChordTask
  . workflowEngine.Main.Server.tasks.simpleTask

[2019-03-10 23:08:18,978: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@DESKTOP-IEMN7E0', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.1', 'product': 'RabbitMQ', 'version': '3.7.9'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-03-10 23:08:18,980: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2019-03-10 23:08:18,981: DEBUG/MainProcess] ^-- substep ok
[2019-03-10 23:08:18,981: DEBUG/MainProcess] | Consumer: Starting Events
[2019-03-10 23:08:18,998: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@DESKTOP-IEMN7E0', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.1', 'product': 'RabbitMQ', 'version': '3.7.9'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-03-10 23:08:19,001: DEBUG/MainProcess] ^-- substep ok
[2019-03-10 23:08:19,001: DEBUG/MainProcess] | Consumer: Starting Tasks
[2019-03-10 23:08:19,010: DEBUG/MainProcess] using channel_id: 1
[2019-03-10 23:08:19,028: DEBUG/MainProcess] Channel open
[2019-03-10 23:08:19,037: DEBUG/MainProcess] ^-- substep ok
[2019-03-10 23:08:19,037: DEBUG/MainProcess] | Consumer: Starting Control
[2019-03-10 23:08:19,052: DEBUG/MainProcess] ^-- substep ok
[2019-03-10 23:08:19,052: DEBUG/MainProcess] | Consumer: Starting event loop
[2019-03-10 23:08:19,054: INFO/MainProcess] celery@DESKTOP-IEMN7E0 ready.
[2019-03-10 23:08:19,055: DEBUG/MainProcess] basic.qos: prefetch_count->16
[2019-03-10 23:09:48,701: INFO/MainProcess] Received task: workflowEngine.Main.Server.tasks.simpleTask[13644976-55e4-4df4-a48a-a9274ff3daff]  
[2019-03-10 23:09:48,703: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x000002771E1B9C80> (args:('workflowEngine.Main.Server.tasks.simpleTask', '13644976-55e4-4df4-a48a-a9274ff3daff', {'lang': 'py', 'task': 'workflowEngine.Main.Server.tasks.simpleTask', 'id': '13644976-55e4-4df4-a48a-a9274ff3daff', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '13644976-55e4-4df4-a48a-a9274ff3daff', 'parent_id': None, 'argsrepr': "('1',)", 'kwargsrepr': '{}', 'origin': 'gen16440@DESKTOP-IEMN7E0', 'reply_to': 'e1566cdc-a4b7-3017-8167-36ea0a95af14', 'correlation_id': '13644976-55e4-4df4-a48a-a9274ff3daff', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[["1"], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})

Answer 1

经过一些调试，我发现我的问题是错误地传递了 -P 参数，令人惊讶的是，通过 celery 输出，应用程序确实认识到我想使用 eventlet包名称。

argv = [
    'worker',
    '-l=DEBUG',
    '-P',
    'eventlet',
    '-c=100',
    '--without-gossip',
    '--without-mingle',
    '--without-heartbeat',
    '-Ofair',
]
app.worker_main(argv)

这个命令对我来说非常适合 eventlet 和 gevent