python - 找出差异，Celery 任务随机失败且没有错误

Question

我的一些 Remote Celery任务似乎永远不会到达我的经纪人 (RabbitMQ)。这似乎是随机发生的。我的日志中没有NO 错误，它们永远不会到达工作人员或失败。 Flower/Rabbit 从不报告任务失败。

我使用 tcpflow -p -c -i eth0 port 5672 来监控 API 上发送任务 (client) 的流量。

当API成功发送任务时，外出流量记录如下:

(已删除敏感数据)

192.018.000.002.42738-052.048.150.171.05672: AMQP
052.048.150.171.05672-192.018.000.002.42738:

capabilitiesFpublisher_confirmstexchange_exchange_bindingst
basic.nacktconsumer_cancel_notifytconnection.blockedtconsumer_prioritiestauthentication_failure_closetper_consumer_qostcluster_nameSrabbit@d8b85eb5ab91copyrightS.Copyright (C) 2007-2015 Pivotal Software, Inc.informationS5Licensed under the MPL.  See http://www.rabbitmq.com/platformS
Erlang/OTPproductSRabbitMQversionS3.6.0PLAIN AMQPLAINen_US
192.018.000.002.42738-052.048.150.171.05672:
nproductSpy-amqpproduct_versionS1.4.9capabilitiesF.connection.blockedtconsumer_cancel_notifytAMQPLAIN1LOGINSusernamePASSWORDSxxxxxxen_US
052.048.150.171.05672-192.018.000.002.42738:
<
192.018.000.002.42738-052.048.150.171.05672:

192.018.000.002.42738-052.048.150.171.05672:
(/
052.048.150.171.05672-192.018.000.002.42738:
)
192.018.000.002.42738-052.048.150.171.05672:

052.048.150.171.05672-192.018.000.002.42738:
192.018.000.002.42738-052.048.150.171.05672: $(
estimate_geometrydirect
052.048.150.171.05672-192.018.000.002.42738: (
192.018.000.002.42738-052.048.150.171.05672: 2
estimate_geometry
052.048.150.171.05672-192.018.000.002.42738: 2estimate_geometry
192.018.000.002.42738-052.048.150.171.05672: G2estimate_geometryestimate_geometrytasks.estimate_geometry
052.048.150.171.05672-192.018.000.002.42738: 2
192.018.000.002.42738-052.048.150.171.05672: 1<(estimate_geometrytasks.estimate_geometry
192.018.000.002.42738-052.048.150.171.05672: <application/x-python-serializebinary$021e5308-e6ac-43eb-9a06-8473ba386802$bedeb08f-9614-38b1-9b60-9eded43c3c71
192.018.000.002.42738-052.048.150.171.05672: }q(UexpiresqNUutcqUargsq]qCaUchordqNUcallbacksqNUerrbacksqNUtasksetqNUidq
Utasks.estimate_geometryqUtimelimitqNNUetaqNUkwargsq}qu.
192.018.000.002.42738-052.048.150.171.05672:  (
segment_imagedirect
052.048.150.171.05672-192.018.000.002.42738: (
192.018.000.002.42738-052.048.150.171.05672: 2
segment_image
segment_image71.05672-192.018.000.002.42738: 2
segment_imagetasks.segment_image0.171.05672: ;2
052.048.150.171.05672-192.018.000.002.42738: 2
segment_imagetasks.segment_image0.171.05672: )<(
192.018.000.002.42738-052.048.150.171.05672: <application/x-python-serializebinary$45280975-9611-41e1-bf99-388cdf1b7064$bedeb08f-9614-38b1-9b60-9eded43c3c71
192.018.000.002.42738-052.048.150.171.05672: }q(UexpiresqNUutcqUargsq]qCaUchordqNUcallbacksqNUerrbacksqNUtasksetqNUidq
Utasks.segment_imageqUtimelimitqNNUetaqNUkwargsq}qu.skq

这是一个永远不会交给经纪人的任务，谁能找出其中的区别并告诉我哪里出了问题？

192.018.000.002.35908-052.017.119.221.05672: 1<(estimate_geometrytasks.estimate_geometry
192.018.000.002.35908-052.017.119.221.05672: <application/x-python-serializebinary$206c1ae0-43d0-4031-bac6-92d2df92b13c$f4c7420c-b9c2-3525-bd5a-d5955f884f43
192.018.000.002.35908-052.017.119.221.05672: }q(UexpiresqNUutcqUargsq]qCaUchordqNUcallbacksqNUerrbacksqNUtasksetqNUidq
Utasks.estimate_geometryqUtimelimitqNNUetaqNUkwargsq}qu.
segment_imagetasks.segment_image9.221.05672: )<(
192.018.000.002.35908-052.017.119.221.05672: <application/x-python-serializebinary$ce0da18a-6534-42d0-9919-cd2e85c8d5e9$f4c7420c-b9c2-3525-bd5a-d5955f884f43
192.018.000.002.35908-052.017.119.221.05672: }q(UexpiresqNUutcqUargsq]qCaUchordqNUcallbacksqNUerrbacksqNUtasksetqNUidq
Utasks.segment_imageqUtimelimitqNNUetaqNUkwargsq}qu.skq

附加信息:

celery_app = Celery('tasks')
celery_app.config_from_object('django.conf:settings')
celery_app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

celery_app.send_task('tasks.estimate_geometry', args=[instance.id], kwargs={})

设置:

import os
from kombu import Exchange, Queue

BROKER_URL = 'amqp://xxxx:xxxx@broker-domain.com:5672//'

CELERY_RESULT_BACKEND = "cache"
CELERY_CACHE_BACKEND = 'memcached://xxxxxx:11211'

CELERY_DEFAULT_QUEUE = 'default'
CELERY_DEFAULT_EXCHANGE_TYPE = 'topic'
CELERY_DEFAULT_ROUTING_KEY = 'default'
CELERY_QUEUES = (
    Queue('default', Exchange('default'), routing_key='default'),
    Queue('estimate_geometry', Exchange('estimate_geometry'), routing_key='tasks.estimate_geometry'),
    Queue('segment_image', Exchange('segment_image'), routing_key='tasks.segment_image'),
    Queue('geometry_feature', Exchange('geometry_feature'), routing_key='tasks.geometry_feature'),
)
CELERY_ROUTES = {
    'tasks.estimate_geometry': {
        'queue': 'estimate_geometry',
        'routing_key': 'tasks.estimate_geometry',
    },
    'tasks.segment_image': {
        'queue': 'segment_image',
        'routing_key': 'tasks.segment_image',
    },
    'tasks.geometry_feature': {
        'queue': 'geometry_feature',
        'routing_key': 'tasks.geometry_feature',
    },
}


BROKER_HEARTBEAT = 10

Answer 1

看来您在调试问题时走错了路。我看到线:

segment_imagetasks.segment_image0.171.05672: )<(" 

在两个日志中。

如果它是您代码的打印输出，那么它可以正常工作，但不正确。所以问题不在传输层。

在第一种情况下，我可以看到线条:

192.018.000.002.42738-052.048.150.171.05672:  (
segment_imagedirect
052.048.150.171.05672-192.018.000.002.42738: (
192.018.000.002.42738-052.048.150.171.05672: 2
segment_image
segment_image71.05672-192.018.000.002.42738: 2
segment_imagetasks.segment_image0.171.05672: ;2
052.048.150.171.05672-192.018.000.002.42738: 2

所以，有时它运作良好。

您应该使用遇到问题的实例 ID 手动运行您的代码(tasks.estimate_geometry)。

如果是服务器日志:可能是某些 C 库未在服务器上编译，您的代码在本地运行但在服务器上不起作用。因此，检查图像格式。