python - ZMQ DEALER ROUTER 高频丢消息？

Question

我正在使用 pyzmq 从 DEALER 向 ROUTER 发送 20000 条消息>.

当我在每条消息之间暂停 0.0001 秒时，它们都会到达，但如果我通过每条消息暂停 0.00001 来将它们发送速度提高 10 倍，则只有大约一半的消息到达。

是什么导致了这个问题？

Answer 1

是什么导致了这个问题？

ZMQ IO 线程的默认设置 - 负责操作模式。

我敢将其称为问题，如果您投入时间并深入研究出色的 ZMQ 概念和架构，问题会更多。

自 ZMQ 库的早期版本以来，有一些重要的参数可以帮助核心杰作(IO 线程)保持稳定 和可扩展 并且从而为您提供这个强大的框架。

零共享/零复制/(几乎)零延迟是不会以零成本实现的格言。

ZMQ.Context 实例具有相当丰富的内部参数化，可以通过 API 方法进行修改。

让我引用一个奇妙而珍贵的来源——Pieter HINTJENS 的书，Code Connected，第 1 卷。

( 花时间逐步浏览 PDF 副本绝对值得。C 语言代码片段不会伤害任何人的 Python 思维状态，因为关键信息包含在 Pieter 精心制作的 300 多页令人兴奋的页面中的文本和故事中).

High-Water Marks

When you can send messages rapidly from process to process, you soon discover that memory is a precious resource, and one that can be trivially filled up. A few seconds of delay somewhere in a process can turn into a backlog that blows up a server unless you understand the problem and take precautions.

...

ØMQ uses the concept of HWM (high-water mark) to define the capacity of its internal pipes. Each connection out of a socket or into a socket has its own pipe, and HWM for sending, and/or receiving, depending on the socket type. Some sockets (PUB, PUSH) only have send buffers. Some (SUB, PULL, REQ, REP) only have receive buffers. Some (DEALER, ROUTER, PAIR) have both send and receive buffers.

In ØMQ v2.x, the HWM was infinite by default. This was easy but also typically fatal for high-volume publishers. In ØMQ v3.x, it’s set to 1,000 by default, which is more sensible. If you’re still using ØMQ v2.x, you should always set a HWM on your sockets, be it 1,000 to match ØMQ v3.x or another figure that takes into account your message sizes and expected subscriber performance.

When your socket reaches its HWM, it will either block or drop data depending on the socket type. PUB and ROUTER sockets will drop data if they reach their HWM, while other socket types will block. Over the inproc transport, the sender and receiver share the same buffers, so the real HWM is the sum of the HWM set by both sides.

Lastly, the HWM-s are not exact; while you may get up to 1,000 messages by default, the real buffer size may be much lower (as little as half), due to the way libzmq implements its queues.