java - KafkaProducer 未向主题生成 > 1 MB 的消息

Question

我正在使用 kafka_2.11-2.2.1。我在 server.properties 中有以下设置。

message.max.bytes=20971520
replica.fetch.max.bytes=23068672

代码级别的生产者配置。

@Bean("serviceKafkaTemplate")
public KafkaTemplate kafkaTemplate(@Autowired final ProducerFactory producerFactory) {
    return new KafkaTemplate(producerFactory);
}

 @Bean
public ProducerFactory<String, Object> producerFactory() {
    return new DefaultKafkaProducerFactory(props());
}

 private Properties props() {
    Properties properties = new Properties();     
    properties.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip");
    properties.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 10000);
    properties.put(ProducerConfig.BATCH_SIZE_CONFIG, 1);
    properties.put(ProducerConfig.LINGER_MS_CONFIG, 1);
    properties.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 15728640);
    properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    return properties;
}

@Autowired
private KafkaTemplate serviceKafkaTemplate;

serviceKafkaTemplate.send("topic_name", data);

我只是在谈论生产者而不是消费者。当我通过 cmd 检查偏移量时，它没有增加，这意味着生产者没有发送数据。我的数据大小小于 1 MB 然后它正在工作，当前大小为 1.75 MB。 kafka server.log 文件中也没有与此相关的内容。如果我遗漏任何配置，请告诉我。

Answer 1

请看房

max.request.size=1048576

当前文档的描述:

The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size which may be different from this.

认为这应该对您有所帮助。另请查看文档。

Apache Kafka documentation - producer configs