rabbitmq - Celery RabbitMQ 代理故障转移连接问题-6ren

rabbitmq - Celery RabbitMQ 代理故障转移连接问题

转载作者：行者123 更新时间：2023-12-02 03:05:36

我在 HA 模式的集群中有 3 个 RabbitMQ 节点。每个节点都在单独的 Docker 容器上。

我正在使用 Celery 版本 4 和 kombu 版本 4。

我已使用此命令来设置 HA 策略:

rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

Celery 配置如下所示:

CELERY = dict(
    broker_url=[
        'amqp://guest@rabbitmq1:5672',
        'amqp://guest@rabbitmq2:5672',
        'amqp://guest@rabbitmq3:5672',
    ],
   celery_queue_ha_policy='all',
   ...
)

一切正常，直到我停止主 RabbitMQ 应用程序以使用以下命令测试 Celery 故障转移功能:

rabbitmqctl stop_app

在 RabbitMQ 应用程序停止后，我立即开始在下面的日志中看到错误。日志消息的频率非常高，并且不会随着尝试次数的增加而减慢。

根据日志，Celery 尝试使用下一次故障转移重新连接，但它被另一次重新连接到已停止的主节点的尝试中断。同样的事情在无限循环中一遍又一遍地发生。

[2017-03-17 15:10:28,084: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq1:5672//: [Errno 111] Connection refused.
Will retry using next failover.

[2017-03-17 15:10:28,300: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2016 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@rabbitmq1', 'platform': 'Erlang/OTP', 'version': '3.6.6'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2017-03-17 15:10:28,302: DEBUG/MainProcess] ^-- substep ok
[2017-03-17 15:10:28,303: DEBUG/MainProcess] | Consumer: Starting Mingle
[2017-03-17 15:10:28,303: INFO/MainProcess] mingle: searching for neighbors
[2017-03-17 15:10:28,303: DEBUG/MainProcess] using channel_id: 1
[2017-03-17 15:10:28,318: DEBUG/MainProcess] Channel open
[2017-03-17 15:10:28,470: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 38, in start
    self.sync(c)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 42, in sync
    replies = self.send_hello(c)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 55, in send_hello
    replies = inspect.hello(c.hostname, our_revoked._data) or {}
  File "/usr/local/lib/python2.7/site-packages/celery/app/control.py", line 129, in hello
    return self._request('hello', from_node=from_node, revoked=revoked)
  File "/usr/local/lib/python2.7/site-packages/celery/app/control.py", line 81, in _request
    timeout=self.timeout, reply=True,
  File "/usr/local/lib/python2.7/site-packages/celery/app/control.py", line 436, in broadcast
    limit, callback, channel=channel,
  File "/usr/local/lib/python2.7/site-packages/kombu/pidbox.py", line 315, in _broadcast
    serializer=serializer)
  File "/usr/local/lib/python2.7/site-packages/kombu/pidbox.py", line 290, in _publish
    serializer=serializer,
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 187, in _publish
    channel = self.channel
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 209, in _get_channel
    channel = self._channel = channel()
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/functional.py", line 38, in __call__
    value = self.__value__ = self.__contract__()
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 224, in <lambda>
    channel = ChannelPromise(lambda: connection.default_channel)
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 819, in default_channel
    self.connection
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 802, in connection
    self._connection = self._establish_connection()
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 757, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 294, in connect
    self.transport.connect()
  File "/usr/local/lib/python2.7/site-packages/amqp/transport.py", line 120, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/usr/local/lib/python2.7/site-packages/amqp/transport.py", line 161, in _connect
    self.sock.connect(sa)
  File "/usr/local/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
[2017-03-17 15:10:28,508: DEBUG/MainProcess] Closed channel #1
[2017-03-17 15:10:28,570: DEBUG/MainProcess] | Consumer: Restarting event loop...
[2017-03-17 15:10:28,572: DEBUG/MainProcess] | Consumer: Restarting Gossip...
[2017-03-17 15:10:28,575: DEBUG/MainProcess] | Consumer: Restarting Heart...
[2017-03-17 15:10:28,648: DEBUG/MainProcess] | Consumer: Restarting Control...
[2017-03-17 15:10:28,655: DEBUG/MainProcess] | Consumer: Restarting Tasks...
[2017-03-17 15:10:28,655: DEBUG/MainProcess] Canceling task consumer...
[2017-03-17 15:10:28,655: DEBUG/MainProcess] | Consumer: Restarting Mingle...
[2017-03-17 15:10:28,655: DEBUG/MainProcess] | Consumer: Restarting Events...
[2017-03-17 15:10:28,672: DEBUG/MainProcess] | Consumer: Restarting Connection...
[2017-03-17 15:10:28,673: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-03-17 15:10:28,947: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq1:5672//: [Errno 111] Connection refused.
Will retry using next failover.

[2017-03-17 15:10:29,345: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2016 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@rabbitmq1', 'platform': 'Erlang/OTP', 'version': '3.6.6'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2017-03-17 15:10:29,506: INFO/MainProcess] Connected to amqp://guest:**@rabbitmq2:5672//
[2017-03-17 15:10:29,535: DEBUG/MainProcess] ^-- substep ok
[2017-03-17 15:10:29,569: DEBUG/MainProcess] | Consumer: Starting Events
[2017-03-17 15:10:29,682: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq1:5672//: [Errno 111] Connection refused.
Will retry using next failover.

[2017-03-17 15:10:29,740: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'product': 'RabbitMQ', 'copyright': 'Copyright (C) 2007-2016 Pivotal Software, Inc.', 'capabilities': {'exchange_exchange_bindings': True, 'connection.blocked': True, 'authentication_failure_close': True, 'direct_reply_to': True, 'basic.nack': True, 'per_consumer_qos': True, 'consumer_priorities': True, 'consumer_cancel_notify': True, 'publisher_confirms': True}, 'cluster_name': 'rabbit@rabbitmq1', 'platform': 'Erlang/OTP', 'version': '3.6.6'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2017-03-17 15:10:29,768: DEBUG/MainProcess] ^-- substep ok
[2017-03-17 15:10:29,770: DEBUG/MainProcess] | Consumer: Starting Mingle
[2017-03-17 15:10:29,770: INFO/MainProcess] mingle: searching for neighbors
[2017-03-17 15:10:29,771: DEBUG/MainProcess] using channel_id: 1
[2017-03-17 15:10:29,795: DEBUG/MainProcess] Channel open
[2017-03-17 15:10:29,874: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 38, in start
    self.sync(c)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 42, in sync
    replies = self.send_hello(c)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/mingle.py", line 55, in send_hello
    replies = inspect.hello(c.hostname, our_revoked._data) or {}
  File "/usr/local/lib/python2.7/site-packages/celery/app/control.py", line 129, in hello
    return self._request('hello', from_node=from_node, revoked=revoked)
  File "/usr/local/lib/python2.7/site-packages/celery/app/control.py", line 81, in _request
    timeout=self.timeout, reply=True,
  File "/usr/local/lib/python2.7/site-packages/celery/app/control.py", line 436, in broadcast
    limit, callback, channel=channel,
  File "/usr/local/lib/python2.7/site-packages/kombu/pidbox.py", line 315, in _broadcast
    serializer=serializer)
  File "/usr/local/lib/python2.7/site-packages/kombu/pidbox.py", line 290, in _publish
    serializer=serializer,
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 187, in _publish
    channel = self.channel
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 209, in _get_channel
    channel = self._channel = channel()
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/functional.py", line 38, in __call__
    value = self.__value__ = self.__contract__()
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 224, in <lambda>
    channel = ChannelPromise(lambda: connection.default_channel)
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 819, in default_channel
    self.connection
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 802, in connection
    self._connection = self._establish_connection()
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 757, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 294, in connect
    self.transport.connect()
  File "/usr/local/lib/python2.7/site-packages/amqp/transport.py", line 120, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/usr/local/lib/python2.7/site-packages/amqp/transport.py", line 161, in _connect
    self.sock.connect(sa)
  File "/usr/local/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused
[2017-03-17 15:10:29,887: DEBUG/MainProcess] Closed channel #1
[2017-03-17 15:10:29,907: DEBUG/MainProcess] | Consumer: Restarting event loop...
[2017-03-17 15:10:29,908: DEBUG/MainProcess] | Consumer: Restarting Gossip...
[2017-03-17 15:10:29,908: DEBUG/MainProcess] | Consumer: Restarting Heart...
[2017-03-17 15:10:29,908: DEBUG/MainProcess] | Consumer: Restarting Control...
[2017-03-17 15:10:29,909: DEBUG/MainProcess] | Consumer: Restarting Tasks...
[2017-03-17 15:10:29,910: DEBUG/MainProcess] Canceling task consumer...
[2017-03-17 15:10:29,911: DEBUG/MainProcess] | Consumer: Restarting Mingle...
[2017-03-17 15:10:29,912: DEBUG/MainProcess] | Consumer: Restarting Events...
[2017-03-17 15:10:29,953: DEBUG/MainProcess] | Consumer: Restarting Connection...
[2017-03-17 15:10:29,954: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-03-17 15:10:30,036: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq1:5672//: [Errno 111] Connection refused.
Will retry using next failover.

不幸的是，Celery 文档并没有对故障转移主题进行太多说明。

最佳答案

它绝对是错误，我在 GitHub 上创建了问题:https://github.com/celery/celery/issues/3921

感谢 George Psarakis，我设法使用 --without-mingle 避免了错误。 celery worker 的标志，例如:

celery worker -A app.tasks -l debug --without-mingle

关于rabbitmq - Celery RabbitMQ 代理故障转移连接问题，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/42996655/

文章推荐： amazon-web-services - 使用月度许可在云 (AWS) 上许可产品

文章推荐： ubuntu-16.04 - Upstart 到 Systemd

文章推荐： amazon-s3 - 在 AWS EMR 上创建 Hive 外部表

android - 如何在 Windows 上使用最新的 Android 模拟器配置 HTTP 代理，如 Fiddler2 或 Charles Web 代理？
我完全不同意配置代理来检查我正在开发的应用程序的 HTTP(S) 流量。我试过运行 Fiddler2 和 Charles Web Proxy，它们都在 127.0.0.1:888 上运行，并使用以下参
amazon-web-services - 在我的 EC2 实例上安装 SSM 代理，以便在没有 SSH 或 key 对的情况下安装 Inspector 代理
我有一个 AWS 环境，其中有几个实例没有预安装 SSM 代理，也没有 key 对，有没有办法在不使用 SSH 登录我的实例的情况下安装 SSM 代理？在此先感谢您的帮助! 最佳答案没有。这是一个
JavaScript 代理
在本教程中，您将借助示例了解 JavaScript 代理。在 JavaScript 中，proxy（代理对象）用于包装对象并将各种操作重新定义到对象中，例如读取、插入、验证等。代理允许您向对
去大文件下载并传递给客户端(代理)
我有一个基于 Martini 的小型应用程序，但遇到了一个我无法解决的问题。我想添加一个应用程序功能，允许用户从第三个服务器获取文件，并在 HTTP header 中进行一些更改。某种代理。这些文件
设计模式：代理、装饰和适配器模式的区别
结构对比讲实话，博主当初学习完整设计模式时，这三种设计模式单独摘哪一种都是十分清晰和明确的，但是随着模式种类的增加，在实际使用的时候竟然会出现恍惚，例如读开源代码时，遇到不以模式命名规范的代码时，
nginx 代理 ELK
我正在尝试代理运行 ELK 的后端服务器。这是我的环境信息: root@proxy:~# root@proxy:~# cat /etc/*release DISTRIB_ID=Ubuntu DISTR
Java 代理、检测和数组创建
我需要为我的 java 应用程序编写一个代理，它在每个数组创建时执行一些特定的操作。到目前为止，我无法找到在此事件上运行我的代码的任何方法。 java.lang.instrument.ClassFil
PHP 代理 - 基本说明
PHP 代理如何工作？我希望制作一个类似于其他 php 代理的小脚本但是它实际上是如何工作的呢？最佳答案我正在考虑一个 PHP 代理，用于绕过 AJAX Sane Origin 策略。如果您需
proxy - Electron 代理
我有一个 Electron 应用程序，试图通过该应用程序从同一网络调用url，但是出于安全考虑，我考虑了使用代理的想法。 function createWindow () { const mai
c# - 内部路由/代理
我有 1 台计算机，安装了 1 个网卡。网卡有 10 个 IP 地址分配给它。我在那里运行了一个 Windows 桌面应用程序。该应用程序基本上是一个调用 1 个特定网站的网络浏览器。我想要实现的是
Java 代理 Burp
我想将 Burp 配置为我的 java 代码的代理，以查看请求和响应。Burp 作为 Web 浏览器之间的代理可以很好地工作，但它不适用于 Java 应用程序。我已经在代码中添加了这样的行: Web
java - 在拦截方法上配置拦截器(代理)
据我所知，在Spring AOP中，当我们想要拦截某些方法调用时，我们会配置一个具有与所需方法调用相匹配的切入点配置的Aspect。也就是说，我们在Aspect端配置拦截。有没有一种方法可以完全从相
Java 代理 - 获取传出请求
这可能是一个常见问题，但是:我有一个正在向发出请求的应用程序elldmess.cz/api/... 但是这个api已经没有了。现在我想要“东西”，即 catch 对 elldmess.cz/api
使用套接字的 Android 代理
我正在尝试在 Android 中创建代理，但我必须使用套接字。我已经阅读了很多教程并提出了以下代码。不幸的是，浏览器似乎没有获得任何数据，一段时间后它显示标准网页，说网页不可用。可能是什么原因？感谢您
使用套接字的 Java 代理
我在使用此代码时遇到了一些问题，具体取决于我使用的浏览器，有些 URL 在 IE 中显示正确，但在 Firefox 中显示为纯文本(例如 www.microsoft.es 在 IE 上看起来不错，但在
Python urllib 代理
我正在尝试通过 urllib 获取一些 url 并通过我的代理进行 Mechanize 。使用 mechanize 我尝试以下操作: from mechanize import Browser im
Python https 代理
我安装了一个嵌入式设备(光伏转换器)，它提供了一个正常的 http Web 界面(信息和设置)。该转换器具有用户身份验证，但只能通过 http 进行。出于安全考虑，我不想将服务器直接发布到互联网上。在
ruby MITM 代理
我正在搜索有关如何使用支持 HTTPS 的 Ruby 编写代理的一些示例。我有一个使用 Webricks HTTPProxyServer 实现的简单代理，但我注意到，HTTPS 流量只是隧道传输(它应
SSL 代理/解密？
我的一位客户刚收到他选择的开发商订购的软件，让我看一下并准备托管程序。这是一个 Java (jar) 应用程序，到目前为止一切顺利......但我看到了一些可疑的东西，软件每隔 60 分钟左右连接到
c#- HTTPS 代理
我试图在 C# 中创建一个 HTTPS 代理服务器。这里有人发布了解决方案: string host = "encrypted.google.com"; string

行者123

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

rabbitmq - Celery RabbitMQ 代理故障转移连接问题