go - Kafka 在生产时将 Offset 返回为 0

Question

使用 confluent-kafka-go 在 golang 中编写了一个基本的基于 channel 的生产者以及通过 confluent 设置的单分区单代理 kafka。在通过 acks 迭代期间生成时，成功生成的消息的偏移量有时被接收为 0。

示例日志:

{"level":"info","msg":"Published successfully to topic test, offset 0, Partition 0 and Length 1316","time":"2017-10-03T17:03:43+05:30"}

{"level":"info","msg":"Published successfully to topic test, offset 0, Partition 0 and Length 1316","time":"2017-10-03T17:03:43+05:30"}

这是遍历 kafka acks 的函数:

func (kc *KafkaClient) HandleAcknowledgements() {
  for event := range kc.AckChannel {
    message := event.(*kafka.Message)

    if message.TopicPartition.Error != nil {
      log.Errorf("Delivery Failed: Partition: %d Reason: %v", message.TopicPartition.Partition, message.TopicPartition.Error)
    } else {
      if message.TopicPartition.Offset == 0 {
        bodyLen := len(message.Value)
        log.Infof("Published successfully to topic %s, offset %s, Partition %d and Length %d", *message.TopicPartition.Topic, message.TopicPartition.Offset, message.TopicPartition.Partition, bodyLen)
      }
    }
  }
}

我的保留政策如下:

   log.retention.hours=168

任何形式的帮助都会很好，至于为什么会发生这种情况!

谢谢。

Answer 1

尝试在配置中将 produce.offset.report 设置为 true。

Try setting produce.offset.report to true for the producer, otherwise it will only report the offset of the last message in each batch/messageSet.

The go client should really do this by default.

https://github.com/confluentinc/confluent-kafka-go/issues/95#issuecomment-330837647

如果这不能解决您的问题，您可以发布您用于生产的代码和配置吗？