apache-kafka - 使用 SSL 的 Kafka SASL/PLAIN 设置

Question

我正在尝试在我们的 Kafka 环境中使用 SSL 配置 SASL/PLAIN。 SSL 部分已完成，但我在启动 Zookeeper 时遇到以下错误。
是否有人为 Zookeeper 和代理配置了带 SSL 的 SASL/PLAIN？

服务器代理配置

broker.id=19
port=19001

#host.name=localhost
host.name=jcia9081

#advertised.host.name=11.72.29.153

num.network.threads=3
num.io.threads=8

socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

log.dirs=/appsdata/osp/kafka/datastore1/AIN_TEST_SSL/kafka,/appsdata/osp/kafka/datastore2/AIN_TEST_SSL/kafka,/appsdata/osp/kafka/datastore3/AIN_TEST_SSL/kafka

num.partitions=1
num.recovery.threads.per.data.dir=1

log.retention.hours=96
#log.retention.bytes=1073741824

log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

log.cleaner.enable=false

replica.fetch.max.bytes=52428800

#The maximum size of a message that the server can receive.
message.max.bytes=52428800

zookeeper.connect=jcia9081:19101,jcia9131:19101,jcia9434:19101
zookeeper.connection.timeout.ms=6000
#################################
auto.create.topics.enable=false
auto.leader.rebalance.enable=true
delete.topic.enable=true
controlled.shutdown.enable=true

### SSL properties
security.inter.broker.protocol=SASL_SSL
listeners=SSL://jcia9081:19001,SASL_SSL://jcia9081:19002
ssl.keystore.location=/home/kafka/certs/jcia9081.federated.fds.jks
ssl.keystore.password=xxxx
ssl.key.password=xxxx
ssl.truststore.location=/home/kafka/certs/test.truststore.jks
ssl.truststore.password=xxx
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
#listeners=SSL://jcia9081:19001
allow.everyone.if.no.acl.found=true

动物园管理员配置

dataDir=/appsdata/osp/kafka/datastore1/AIN_TEST_SSL/zookeeper

clientPort=19101

maxClientCnxns=0

tickTime=2000

initLimit=60
syncLimit=10

server.19=jcia9081:19121:19131
server.20=jcia9131:19121:19131
server.21=jcia9434:19121:19131

JAAS Zookeeper 配置文件

Server {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin-secret"
   user_admin="admin-secret"
};

JAAS 代理配置文件

    KafkaServer {
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="admin"
        password="admin-secret"
        user_admin="admin-secret";
        };
Client {
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="admin"
        password="admin-secret";
};

启动错误

[2017-04-26 10:05:24,775] INFO Client environment:user.dir=/appsdata/osp/kafka/AIN_TEST_SSL/startup (org.apache.zookeeper.ZooKeeper)
[2017-04-26 10:05:24,776] INFO Initiating client connection, connectString=jcia9081:19101 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@43195e57 (org.apache.zookeeper.ZooKeeper)
[2017-04-26 10:05:24,776] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2017-04-26 10:05:24,778] DEBUG zookeeper.disableAutoWatchReset is false (org.apache.zookeeper.ClientCnxn)
[2017-04-26 10:05:24,789] DEBUG JAAS loginContext is: Client (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-04-26 10:05:24,795] DEBUG Awaiting connection to Zookeeper server (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:24,795] INFO Waiting for keeper state SaslAuthenticated (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:24,881] INFO successfully logged in. (org.apache.zookeeper.Login)
[2017-04-26 10:05:24,881] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-04-26 10:05:24,895] INFO Opening socket connection to server jcia9081.federated.fds/11.168.24.82:19101. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2017-04-26 10:05:24,968] INFO Socket connection established to jcia9081.federated.fds/11.168.24.82:19101, initiating session (org.apache.zookeeper.ClientCnxn)
[2017-04-26 10:05:24,970] DEBUG Session establishment request sent on jcia9081.federated.fds/11.168.24.82:19101 (org.apache.zookeeper.ClientCnxn)
[2017-04-26 10:05:25,001] INFO Session establishment complete on server jcia9081.federated.fds/11.168.24.82:19101, sessionid = 0x15baa8300580003, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2017-04-26 10:05:25,002] DEBUG ClientCnxn:sendSaslPacket:length=0 (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-04-26 10:05:25,004] DEBUG Received event: WatchedEvent state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,004] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,004] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,006] ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-04-26 10:05:25,006] DEBUG Received event: WatchedEvent state:AuthFailed type:None path:null (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,006] INFO zookeeper state changed (AuthFailed) (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,006] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,006] DEBUG Closing ZkClient... (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,006] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2017-04-26 10:05:25,006] DEBUG Closing ZooKeeper connected to jcia9081:19101 (org.I0Itec.zkclient.ZkConnection)
[2017-04-26 10:05:25,006] DEBUG Close called on already closed client (org.apache.zookeeper.ZooKeeper)
[2017-04-26 10:05:25,006] DEBUG Closing ZkClient...done (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,007] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure
        at org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.java:946)
        at org.I0Itec.zkclient.ZkClient.waitUntilConnected(ZkClient.java:923)
        at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1230)
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
        at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:76)
        at kafka.utils.ZkUtils$.apply(ZkUtils.scala:58)
        at kafka.server.KafkaServer.initZk(KafkaServer.scala:327)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:200)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39)
        at kafka.Kafka$.main(Kafka.scala:67)
        at kafka.Kafka.main(Kafka.scala)
[2017-04-26 10:05:25,009] INFO shutting down (kafka.server.KafkaServer)

代理启动命令

    . /appsdata/osp/kafka/AIN_TEST_SSL/startup/setenv.sh
export JMX_PORT=19011
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_CONFIG/server_AIN_TEST_SSL.properties > $LOG_DIR/kafka-broker-startup.log 2>&1 &

设置环境

    export APP_HOME=/apps/osp

export JAVA_HOME=$APP_HOME/jdk1.8.0_101
export JAVA=$JAVA_HOME/bin/java

export KAFKA_HOME=$APP_HOME/kafka/v2.11-0.10.1.1
export KAFKA_CONFIG=/appsdata/osp/kafka/AIN_TEST_SSL/config

export ZK_HOME=$KAFKA_HOME
export ZK_CONFIG=$KAFKA_CONFIG

export LOG_DIR=/applogs/osp/kafka/AIN_TEST_SSL
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$KAFKA_CONFIG/log4j.properties"

在 1 号航站楼

echo $KAFKA_OPTS
-Djava.security.auth.login.config=/appsdata/osp/kafka/AIN_TEST_SSL/config/jaas_AIN_TEST_SSL_zookeeper.conf

2 号航站楼

echo $KAFKA_OPTS
-Djava.security.auth.login.config=/appsdata/osp/kafka/AIN_TEST_SSL/config/jaas_AIN_TEST_SSL_server.conf

Answer 1

您应该通过导出 KAFKA_OPTS 环境变量来提供您的 jaas 文件的路径。将您的 jaas 文件复制到 /appsdata/osp/kafka/AIN_TEST_SSL/config/

下

我想，这是您文件的内容:

kafka_server_jaas.conf

KafkaServer {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin-secret"
   user_admin="admin-secret";
};
Client {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin-secret";
};

zookeeper.properties

dataDir=/appsdata/osp/kafka/datastore1/AIN_TEST_SSL/zookeeper
clientPort=19101
maxClientCnxns=0
tickTime=2000
initLimit=60
syncLimit=10
server.19=jcia9081:19121:19131
server.20=jcia9131:19121:19131
server.21=jcia9434:19121:19131

#New configuration
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000

[开始更新]

在启动 zookeeper ensemble 之前添加 zookeeper 服务器 jaas 文件并导入它。

Server {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin-secret"
   user_admin="admin-secret"
};

之后，在zookeeper终端上导出那个jaas并启动zookeeper

$ export KAFKA_OPTS="-Djava.security.auth.login.config=/appsdata/osp/kafka/AIN_TEST_SSL/config/zookeeper_jaas.conf"
$ bin/zookeeper-server-start.sh config/zookeeper.properties

打开一个新的终端并点击以下命令启动你的kafka服务器:

$ export KAFKA_OPTS="-Djava.security.auth.login.config=/appsdata/osp/kafka/AIN_TEST_SSL/config/kafka_server_jaas.conf"
$ bin/kafka-server-start.sh config/server.properties

[结束更新]

我认为它会解决您的问题。

如果出现任何问题，请查看我在这篇文章中的配置:

Kafka SASL zookeeper authentication