ssl - 使用 TLS/SSL 实现后控制台生产者错误

Question

我想将 TLS/SSL 添加到我的 kafka 设置中。首先，我浏览了主网站上的 kafka SSL 文档。我做了以下事情:

1) 将签名的证书导入 keystore

2) 导入根CA

3) 使用keytool验证keystore和trust store密码是否正确。

4) 启动zookeeper和kafka。

5) 从 server.log 文件确认以下内容:

 Registered broker 0 at path /brokers/ids/0 with addresses: 
EndPoint(localhost,9092,ListenerName(PLAINTEXT),PLAINTEXT),EndPoint(localhost,9093,ListenerName(SSL),SSL) (kafka.utils.ZkUtils)

我的 server.properties 文件中的 listeners 和 advertised.listeners 设置如下:

PLAINTEXT://localhost:9092,SSL://localhost:9093

我还启用了自动创建主题。当我这样做时:

kafka-console-producer.bat --broker-list localhost:9093 --topic test_ssl --producer.config ....\config\producer.properties

我收到以下错误:

[2017-08-04 16:28:15,265] WARN Error while fetching metadata with correlation id 0 : {test_ssl=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-08-04 16:28:15,372] WARN Error while fetching metadata with correlation id 1 : {test_ssl=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-08-04 16:28:15,474] WARN Error while fetching metadata with correlation id 2 : {test_ssl=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-08-04 16:28:20,302] WARN Error while fetching metadata with correlation id 3 : {test_ssl=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-08-04 16:28:20,406] WARN Error while fetching metadata with correlation id 4 : {test_ssl=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-08-04 16:28:20,512] WARN Error while fetching metadata with correlation id 5 : {test_ssl=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

我尝试使用 openssl 打印出 SSL 通信数据

openssl s_client -connect localhost:9093 -debug -tls1 // default kafka broker configs have tlsv1 included

我得到以下信息:

Certificate chain
 0 s:/C=GB/ST=Unknown/L=London/O=SOAPYSUDS/OU=SOAPYSUDS/CN=M. Manna
   i:/C=GB/ST=Some-State/L=London/O=SOAPYSUDS/OU=SOAPYSUDS/CN=localhost/emailAddress=xyz@xyz.com
 1 s:/C=GB/ST=Some-State/L=London/O=SOAPYSUDS/OU=SOAPYSUDS/CN=localhost/emailAddress=xyz@xyz.com
   i:/C=GB/ST=Some-State/L=London/O=SOAPYSUDS/OU=SOAPYSUDS/CN=localhost/emailAddress=xyz@xyz.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDfDCCAmQCCQDJKjhOy8KBWjANBgkqhkiG9w0BAQsFADCBlTELMAkGA1UEBhMC
R0IxEzARBgNVBAgMClNvbWUtU3RhdGUxDzANBgNVBAcMBkxvbmRvbjEMMAoGA1UE
CgwDU0FQMRcwFQYDVQQLDA5TQVAgRmllbGRnbGFzczESMBAGA1UEAwwJbG9jYWxo
b3N0MSUwIwYJKoZIhvcNAQkBFhZtb2hhbW1lZC5tYW5uYUBzYXAuY29tMB4XDTE3
MDgwNzA5MDU0MFoXDTE5MDgwNzA5MDU0MFowajELMAkGA1UEBhMCR0IxEDAOBgNV
BAgTB1Vua25vd24xDzANBgNVBAcTBkxvbmRvbjEMMAoGA1UEChMDU0FQMRcwFQYD
VQQLEw5TQVAgRmllbGRnbGFzczERMA8GA1UEAxMITS4gTWFubmEwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3+sGslb3XocPzigNOmwi9DRE7U6Gos+k1
ADWCS1UGDXPRuS9JocCc2oZ2GJxwipDpF+xRGmeAXIo77S/4GXFjs+DG1R8kojXn
bth3QRTsI6XAMuWOX3AHd4ob0/WSWsPLkGRggCyqU7qwvhuD+MzXB2Q0isRgab/c
lQl6Bj6ETaYTSY8Q2YjCR0ggL4qpYwO62fUexTB5aJx8JMpEbi0V3NZX/2xpic2m
27NW95j7oxi55Q36sYQNpFxfp9pIft6n3+RYTWIQ0bDV5O4W0TGAGPMDkH+t1K9x
ZZEoLik0aOfHw9sp96LkU5oZGGeWEGt6CmWHZQDiBSAyuTe6IjTpAgMBAAEwDQYJ
KoZIhvcNAQELBQADggEBAC4m+Lpl6pH45VGGvpaolL6PnL5TGcd7FL39C6DxL7Cl
MftcLuG6Egka+eJr/tg+6pdOFfpQjrkeDpeiRIABcLu/VrK99Te7c946QWc8QupC
G/VSQRIHOW6ReBsFf2/wrwT8MWDq2I2GAZSuZ9eTGWggQEwyem6y7C4ujtrcpixq
XwkeE2AU/91O2LlRrP/gW3hwEgD3VruMFhnm/3tTc7ZeLhJPSIE8EMhCk3+j2tgI
ta9J6NZClMxRLgBe1cOH2ru8eGsJtDnT2J+h0m/Ra3t9pD70Amoo84ytE3Iyui8i
kL2gg278c7PF5qosjPkp5sjjvWFypk8iYHTgTidbRCg=
-----END CERTIFICATE-----
subject=/C=GB/ST=Unknown/L=London/O=SOAPYSUDS/OU=SOAPYSUDS/CN=M. Manna
issuer=/C=GB/ST=Some-State/L=London/O=SOAPYSUDS/OU=SOAPYSUDS/CN=localhost/emailAddress=xyz@xyz.com
---
Acceptable client certificate CA names
/C=GB/ST=Some-State/L=London/O=SOAPYSUDS/OU=SOAPYSUDS/CN=localhost/emailAddress=xyz@xyz.com
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5048 bytes and written 285 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.0, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : ECDHE-RSA-AES256-SHA
    Session-ID: 59884152B1D0B4716F30AC8E43BAC10EBBE92E6BD771AAAD31046035564F2B30
    Session-ID-ctx:
    Master-Key: 124F0A4796CCE67A696105F4F88798CFC31E76885DEDF3EB1F702EA565543462AB1CCC9B4E6D726BD7489C17ED77C744
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1502101842
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
---

即使上面的“自签名证书验证”有错误，我认为如果 CA 证书是自签名的，这很常见。可能是因为 SSL 握手已写入/读取数据，它才起作用。

我可以从 kafka-topics 命令(还有 server.log)确认主题“test_ssl”创建成功。我希望这不是因为这个下划线“_”。

如果存在握手问题，它会被记录在日志中(我认为，除非记录器已关闭)，但看起来我的 SSL 配置已被正确接受。只是想知道我是否遗漏了一些我在这里无法完全发现的东西。

注意 - 我的 Zookeeper 没有使用任何 SSL/TLS。此外，因为我是在本地开始 TLS 测试，所以我现在使用的是通用信任库(jre/lib/security 中的 cacerts)。

-- 我的客户端 SSL 配置

advertised.listeners=SSL://localhost:9093
listeners=SSL://localhost:9093
security.protocol=SSL
ssl.truststore.location=$java_path/jre/lib/security/cacerts
ssl.truststore.password=changeit
ssl.keystore.location=/kafka_2.10-0.10.2.1/config/kafka_client.jks
ssl.keystore.password=test1234
ssl.key.password=test1234

-- 我的服务器 SSL 相关属性

security.inter.broker.protocol=SSL
ssl.keystore.location=/kafka_2.10-0.10.2.1/config/kafka_server.jks
ssl.keystore.password=test1234
ssl.key.password=test1234
ssl.truststore.location=$java_path/jre/lib/security/cacerts
ssl.truststore.password=changeit
ssl.endpoint.identification.algorithm=HTTPS
ssl.secure.random.implementation=SHA1PRNG
ssl.client.auth=required

启动后我的服务器日志的一部分(启用 SSL 调试):

Using SSLEngineImpl.
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
Using SSLEngineImpl.
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1
kafka-network-thread-0-ListenerName(SSL)-SSL-0, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1
kafka-network-thread-0-ListenerName(SSL)-SSL-0, called closeOutbound()
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for TLSv1
kafka-network-thread-0-ListenerName(SSL)-SSL-0, closeOutboundInternal()
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
[Raw write]: length = 7
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
0000Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
: 15Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for TLSv1.1
 03Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for TLSv1
 Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1
03 00Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
 Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1
02 02Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for TLSv1.1
%% No cached client session
 50*** ClientHello, TLSv1.2

我不确定我缺少什么配置才能让它正常工作。我认为我的证书导入顺序没有任何问题，因为我已经通过匹配说明确认了我的方法 here .

问候，

Answer 1

这只是一个配置 - 但我希望文档对此有稍微更长的解释 - 但仍然是我的错。

documentation

ssl.endpoint.identification.algorithm

我将其设置为 HTTPS - 这意味着我的客户将根据以下其中一项验证我的完全限定域名 FQDN:

1) 通用名称 (CN)

2) 主题备用名称 (SAN)

当我创建我的证书时，我出于礼貌添加了我的名字和姓氏，心想“这是我的名字和姓氏”。由于我的原始证书没有以下任何一项:

1) localhost 作为 CN

2) localhost 作为 DNS

客户端无法根据提供的证书的 SAN/CN 值验证代理的 FQDN。我相信这就是我在颁发新的自签名 SAN 证书(并将它们导入客户端信任库)后让它工作的原因。