python - 滑子/RabbitMQ : OSError: Server unexpectedly closed connection

Question

我有两个通过 RabbitMQ 使用 RPC 进行通信的 nameko 服务。在本地使用 docker-compose 一切正常。然后我将所有内容部署到 DigitalOcean 上的 Kubernetes/Istio 集群并开始出现以下错误。它在 10/20/60 分钟内连续重复 1 次。服务之间的通信工作正常(我想是在 recconect 之前和之后)，但日志因那些不应该发生的意外重新连接而变得困惑。

Helm RabbitMQ configuration file

我尝试增加 RAM 和 CPU 配置(增加到上面配置文件中的值:512Mb 和 400m)但仍然有相同的行为。

注意:我在部署后没有接触服务，没有发送消息或发出任何请求，这是我在大约 60 分钟内第一次遇到此错误。当我发出请求时，它们成功了，但是最终我们在日志中仍然有这个错误。

Nameko 服务日志:

"Connection to broker lost, trying to re-establish connection...",
"exc_info": "Traceback (most recent call last):
File \"/usr/local/lib/python3.6/site-packages/kombu/mixins.py\", line 175, in run for _ in self.consume(limit=None, **kwargs):
File \"/usr/local/lib/python3.6/site-packages/kombu/mixins.py\", line 197, in consume   conn.drain_events(timeout=safety_interval)
File \"/usr/local/lib/python3.6/site-packages/kombu/connection.py\", line 323, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File \"/usr/local/lib/python3.6/site-packages/kombu/transport/pyamqp.py\", line 103, in drain_events
return connection.drain_events(**kwargs)
File \"/usr/local/lib/python3.6/site-packages/amqp/connection.py\", line 505, in drain_events
while not self.blocking_read(timeout):
File \"/usr/local/lib/python3.6/site-packages/amqp/connection.py\", line 510, in blocking_read\n    frame = self.transport.read_frame()
File \"/usr/local/lib/python3.6/site-packages/amqp/transport.py\", line 252, in read_frame
frame_header = read(7, True)
File \"/usr/local/lib/python3.6/site-packages/amqp/transport.py\", line 446, in _read
raise IOError('Server unexpectedly closed connection')
OSError: Server unexpectedly closed connection"}
{"name": "kombu.mixins", "asctime": "29/12/2019 20:22:54", "levelname": "INFO", "message": "Connected to amqp://user:**@rabbit-rabbitmq:5672//"}

RabbitMQ 日志

2019-12-29 20:22:54.563 [warning] <0.718.0> closing AMQP connection <0.718.0> (127.0.0.1:46504 -> 127.0.0.1:5672, vhost: '/', user: 'user'):
client unexpectedly closed TCP connection
2019-12-29 20:22:54.563 [warning] <0.705.0> closing AMQP connection <0.705.0> (127.0.0.1:46502 -> 127.0.0.1:5672, vhost: '/', user: 'user'):
client unexpectedly closed TCP connection
2019-12-29 20:22:54.681 [info] <0.3424.0> accepting AMQP connection <0.3424.0> (127.0.0.1:43466 -> 127.0.0.1:5672)
2019-12-29 20:22:54.689 [info] <0.3424.0> connection <0.3424.0> (127.0.0.1:43466 -> 127.0.0.1:5672): user 'user' authenticated and granted access to vhost '/'
2019-12-29 20:22:54.690 [info] <0.3431.0> accepting AMQP connection <0.3431.0> (127.0.0.1:43468 -> 127.0.0.1:5672)
2019-12-29 20:22:54.696 [info] <0.3431.0> connection <0.3431.0> (127.0.0.1:43468 -> 127.0.0.1:5672): user 'user' authenticated and granted access to vhost '/'

更新:

Rabbit pod yaml

Answer 1

问题是 istio 代理被作为 sidecar 容器注入(inject)到 rabbitmq pod 中。您需要从 rabbitmq 中排除 istio 代理，然后它应该可以工作。