hive - 如何修复 org.apache.kafka.common.config.ConfigException : Missing required configuration "group.id" which has no default value

Question

我有一个 Kafka 主题设置，我正在尝试在 Hive 中创建一个外部表来查询 Kafka 流。

但是，当查询外部表时，我收到错误消息

Error: java.io.IOException: org.apache.kafka.common.config.ConfigException: Missing required configuration "group.id" which has no default value. (state=,code=0)

尝试在启动 Kafka 服务器时将 group.id 放入 server.properties。

尝试将 group.id 放在外部表属性中。

CREATE EXTERNAL TABLE kafka_table2
  (`timestamp` timestamp , `page` string,  `newPage` boolean,
  added int, deleted bigint, delta double)
  STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
  TBLPROPERTIES
  ("kafka.topic" = "connect-test", "kafka.bootstrap.servers"="mykafka:9092","kafka.group.id"="1")


INFO  : Completed compiling command(queryId=hive_20190426082255_729f8adb-bb23-4317-8f3f-2f9049b62bd7); Time taken: 0.6 seconds
INFO  : Executing command(queryId=hive_20190426082255_729f8adb-bb23-4317-8f3f-2f9049b62bd7): select * from kafka_table2
INFO  : Completed executing command(queryId=hive_20190426082255_729f8adb-bb23-4317-8f3f-2f9049b62bd7); Time taken: 0.018 seconds
INFO  : OK
Error: java.io.IOException: org.apache.kafka.common.config.ConfigException: Missing required configuration "group.id" which has no default value. (state=,code=0)

Answer 1

您应该在 TBLPROPERTIES 中输入 "kafka.consumer.group.id"="1"而不是 "kafka.group.id"="1"。

参见:https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/integrating-hive/content/hive_set_consumer_producer.html