hadoop - 现有 CDH 5.5.2 集群上的 Kafka 配置

Question

我正在我现有的 CDH 5.5.2 集群上安装 Kafka-2.0，这是我遵循的过程

1. 从 CM 添加服务
2. 选择Kafka(在此之前我在所有节点上下载并分发并激活了kafka parcel)
3. 为 KafkaBroker 选择了 1 个节点，为 Kafka MirrorMaker 选择了 4 个节点
4. 然后我使用 Mirror Maker 节点之一更新了我的目标代理列表 (bootstrap.servers) 属性以及具有相同节点的源代理列表 (source.bootstrap.servers)

5. 我得到以下错误(日志文件)

   Fatal error during KafkaServerStartable startup. Prepare to shutdown 
   java.lang.OutOfMemoryError: Java heap space
       at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
       at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
       at kafka.log.SkimpyOffsetMap.<init>(OffsetMap.scala:43)
       at kafka.log.LogCleaner$CleanerThread.<init>(LogCleaner.scala:186)
       at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
       at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
       at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
       at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
       at scala.collection.immutable.Range.foreach(Range.scala:166)
       at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
       at scala.collection.AbstractTraversable.map(Traversable.scala:104)
       at kafka.log.LogCleaner.<init>(LogCleaner.scala:83)
       at kafka.log.LogManager.<init>(LogManager.scala:64)
       at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:601)
       at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
       at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
       at kafka.Kafka$.main(Kafka.scala:67)
       at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76)
       at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)
   

Answer 1

您需要将 broker_max_heap_size 值增加到至少 1GB 并从 Cloudera Manager 重新启动 kafka 服务。如果您仍然遇到同样的问题，请尝试根据您的集群配置增加