apache-kafka - Kafka 缓冲区大小和时间间隔

Question

Kafka 按照 buffer.memory(在我的例子中是 32 MB)将数据保存在 Buffer 中。 kafka 是否在达到 32 MB 限制后将记录写入主题，还是有任何与之相关的时间？

Answer 1

来自 Kafka 文档 buffer.memory 只是指定生产者使用的缓冲区限制的属性。但是由于设置这个属性生产者不会让生产者等到缓冲区变满才将记录发送到服务器。

buffer.memory

The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block for max.block.ms after which it will throw an exception.

This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests.

如果你想让producer等到buffer中的一些记录batch获取，你可以使用linger.ms属性让producer等待。但据我所知，没有严格的方法让生产者等待并仅在缓冲区已满时才发送记录

KafkaProducer

By default a buffer is available to send immediately even if there is additional unused space in the buffer. However, if you want to reduce the number of requests you can set linger.ms to something greater than 0. This will instruct the producer to wait up to that number of milliseconds before sending a request in hope that more records will arrive to fill up the same batch.