ssl - Mount dir 在使用 SSL/TLS 加密的 GlusterFS 存储集群中出错

Question

我使用 SSL/TLS 加密的 GlusterFS 存储集群有问题。

我已经在很多方面尝试了好几天。我希望有人能帮助我。

首先，我启用了 GlusterFS 的 SSL 并在每个对等节点中重新启动了 glusterd。

然后，GlusterFS 本地客户端将启用 GlusterFS 卷 afr_vol_1 的 SSL 并将其挂载到/mnt/gluster。

通常情况下，如果我的挂载目录中只有子文件，我可以重复读取或写入。

但是，如果我的挂载目录中的子目录被重复读取或写入一段时间，我的挂载目录就会出错。

在这里，我使用了命令“find”。重现它。

错误信息如下:

find: ‘./gluster’: Transport endpoint is not connected

最后，如果我禁用 SSL 并重新启动 glusterd，一切都会好的。

那么，我的 SSL 配置有什么问题吗？

---

我的操作系统是 CentOS 7.2。

GlusterFS版本是glusterfs-3.13.1(也试过glusterfs-4.0.0rc1)

OpenSSL版本为OpenSSL 1.0.1e-fips(也试过OpenSSL 1.1.0g)。

我创建了 glusterfs.pem 和 gluster.ca 并将它们复制到其他对等节点:

# cd /etc/ssl/
# sudo openssl genrsa -out glusterfs.key 2048
# sudo openssl req -new -x509 -key glusterfs.key –days 3650 -subj "/CN=glusterfs" -out glusterfs.pem
# cat /etc/ssl/glusterfs.pem > glusterfs.ca

我启用了 GlusterFS 的 SSL 并在每个对等节点中重新启动了 glusterd:

# sudo touch /var/lib/glusterd/secure-access
# systemctl restart glusterd.service

我启用了 GlusterFS 卷 afr_vol_1 的 SSL:

# gluster volume set afr_vol_1 server.ssl on
# gluster volume set afr_vol_1 client.ssl on
# gluster volume set afr_vol_1 auth.ssl-allow '*'
# gluster volume set afr_vol_1 ssl.cipher-list 'ALL'
# gluster volume stop afr_vol_1
# gluster volume start afr_vol_1

GlusterFS 卷 afr_vol_1 由 GlusterFS native 客户端安装在/mnt/gluster 上:

# mount -t glusterfs localhost:/afr_vol_1 /mnt/gluster/

mnt-gluster.log如下:

[2018-03-05 06:18:13.440850] W [rpc-clnt.c:1739:rpc_clnt_submit] 0-afr_vol_1-client-2: error returned while attempting to connect to host:(null), port:0 
[2018-03-05 06:18:13.441577] E [socket.c:2167:__socket_read_frag] 0-rpc: wrong MSG-TYPE (20538) received from 192.168.8.99:24007 
[2018-03-05 06:18:13.442245] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x12a)[0x7f1584f4e43a] (--> /usr/local/lib/libgfrpc.so.0(+0xc452)[0x7f1584d1b452] (--> /usr/local/lib/libgfrpc.so.0(+0xc55e)[0x7f1584d1b55e] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f1584d1caed] (--> /usr/local/lib/libgfrpc.so.0(+0xe638)[0x7f1584d1d638] ))))) 0-afr_vol_1-client-2: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2018-03-05 06:18:13.441030 (xid=0x505d) 
[2018-03-05 06:18:14.454846] W [rpc-clnt.c:1739:rpc_clnt_submit] 0-afr_vol_1-client-1: error returned while attempting to connect to host:(null), port:0 
[2018-03-05 06:18:14.455487] E [socket.c:2167:__socket_read_frag] 0-rpc: wrong MSG-TYPE (13199) received from 192.168.8.98:24007 
[2018-03-05 06:18:14.456464] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x12a)[0x7f1584f4e43a] (--> /usr/local/lib/libgfrpc.so.0(+0xc452)[0x7f1584d1b452] (--> /usr/local/lib/libgfrpc.so.0(+0xc55e)[0x7f1584d1b55e] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f1584d1caed] (--> /usr/local/lib/libgfrpc.so.0(+0xe638)[0x7f1584d1d638] ))))) 0-afr_vol_1-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2018-03-05 06:18:14.455147 (xid=0x33ac) 
[2018-03-05 06:18:15.043577] W [socket.c:592:__socket_rwv] 0-afr_vol_1-client-1: writev on 192.168.8.98:49152 failed (No data available) 
[2018-03-05 06:18:15.043944] E [socket.c:2777:socket_poller] 0-afr_vol_1-client-1: poll error on socket 
[2018-03-05 06:18:15.044065] I [MSGID: 114018] [client.c:2227:client_rpc_notify] 0-afr_vol_1-client-1: disconnected from afr_vol_1-client-1. Client process will keep trying to connect to glusterd until brick's port is available 
[2018-03-05 06:18:15.044103] W [MSGID: 108001] [afr-common.c:5383:afr_notify] 0-afr_vol_1-replicate-0: Client-quorum is not met 
[2018-03-05 06:18:15.044856] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x12a)[0x7f17d9c7644a] (--> /usr/local/lib/libgfrpc.so.0(+0xc452)[0x7f17d9a43452] (--> /usr/local/lib/libgfrpc.so.0(+0xc55e)[0x7f17d9a4355e] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f17d9a44aed] (--> /usr/local/lib/libgfrpc.so.0(+0xe638)[0x7f17d9a45638] ))))) 0-afr_vol_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(RELEASEDIR(42)) called at 2018-03-05 06:18:15.043227 (xid=0x2ddf) 
[2018-03-05 06:18:15.045142] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x12a)[0x7f17d9c7644a] (--> /usr/local/lib/libgfrpc.so.0(+0xc452)[0x7f17d9a43452] (--> /usr/local/lib/libgfrpc.so.0(+0xc55e)[0x7f17d9a4355e] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f17d9a44aed] (--> /usr/local/lib/libgfrpc.so.0(+0xe638)[0x7f17d9a45638] ))))) 0-afr_vol_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(RELEASEDIR(42)) called at 2018-03-05 06:18:15.043690 (xid=0x2de0) 
[2018-03-05 06:18:15.105207] W [fuse-bridge.c:2922:fuse_readdirp_cbk] 0-glusterfs-fuse: 28837: READDIRP => -1 (Transport endpoint is not connected) 
[2018-03-05 06:18:15.161590] W [MSGID: 108027] [afr-common.c:2255:afr_attempt_readsubvol_set] 0-afr_vol_1-replicate-0: no read subvols for /
[2018-03-05 06:18:15.161676] E [MSGID: 101046] [dht-common.c:1904:dht_revalidate_cbk] 0-afr_vol_1-dht: dict is null 
[2018-03-05 06:18:15.161730] W [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 28839: LOOKUP() / => -1 (Transport endpoint is not connected) 
[2018-03-05 06:18:15.162301] I [MSGID: 108031] [afr-common.c:2530:afr_local_discovery_cbk] 0-afr_vol_1-replicate-0: selecting local read_child afr_vol_1-client-0 
[2018-03-05 06:18:15.163540] W [MSGID: 108027] [afr-common.c:2255:afr_attempt_readsubvol_set] 0-afr_vol_1-replicate-0: no read subvols for / 
[2018-03-05 06:18:15.163619] E [MSGID: 101046] [dht-common.c:1904:dht_revalidate_cbk] 0-afr_vol_1-dht: dict is null 
[2018-03-05 06:18:15.163689] W [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 28840: LOOKUP() / => -1 (Transport endpoint is not connected) 
[2018-03-05 06:18:15.230220] W [MSGID: 108027] [afr-common.c:2255:afr_attempt_readsubvol_set] 0-afr_vol_1-replicate-0: no read subvols for / 
[2018-03-05 06:18:15.230331] E [MSGID: 101046] [dht-common.c:1904:dht_revalidate_cbk] 0-afr_vol_1-dht: dict is null
[2018-03-05 06:18:15.230454] W [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 28841: LOOKUP() / => -1 (Transport endpoint is not connected) 
[2018-03-05 06:18:15.231788] W [MSGID: 108027] [afr-common.c:2255:afr_attempt_readsubvol_set] 0-afr_vol_1-replicate-0: no read subvols for / 
[2018-03-05 06:18:15.231870] E [MSGID: 101046] [dht-common.c:1904:dht_revalidate_cbk] 0-afr_vol_1-dht: dict is null
[2018-03-05 06:18:15.231942] W [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 28842: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-03-05 06:18:15.295447] W [MSGID: 108027] [afr-common.c:2255:afr_attempt_readsubvol_set] 0-afr_vol_1-replicate-0: no read subvols for / 
[2018-03-05 06:18:15.295537] E [MSGID: 101046] [dht-common.c:1904:dht_revalidate_cbk] 0-afr_vol_1-dht: dict is null 
[2018-03-05 06:18:15.295590] W [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 28843: LOOKUP() / => -1 (Transport endpoint is not connected) 
[2018-03-05 06:18:15.296719] W [MSGID: 108027] [afr-common.c:2255:afr_attempt_readsubvol_set] 0-afr_vol_1-replicate-0: no read subvols for / 
[2018-03-05 06:18:15.296777] E [MSGID: 101046] [dht-common.c:1904:dht_revalidate_cbk] 0-afr_vol_1-dht: dict is null 
[2018-03-05 06:18:15.296820] W [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 28844: LOOKUP() / => -1 (Transport endpoint is not connected) 
[2018-03-05 06:18:15.358443] W [MSGID: 108027] [afr-common.c:2255:afr_attempt_readsubvol_set] 0-afr_vol_1-replicate-0: no read subvols for / 
[2018-03-05 06:18:15.358538] E [MSGID: 101046] [dht-common.c:1904:dht_revalidate_cbk] 0-afr_vol_1-dht: dict is null 
[2018-03-05 06:18:15.358590] W [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 28845: LOOKUP() / => -1 (Transport endpoint is not connected)

另一个peer节点的glusterd.log如下:

[2018-03-05 06:18:14.931907] E [socket.c:2777:socket_poller] 0-socket.management: poll error on socket
[2018-03-05 06:18:34.555460] E [socket.c:2785:socket_poller] 0-socket.management: socket_poller 192.168.8.97:49141 failed (Input/output error)

Answer 1

嗯，问题自己解决了...-_-b

如果您通过源代码安装GlusterFS，除非正确修改make configure，否则您会得到与上述相同的错误。

不过，如果你通过rpm安装GlusterFS就可以了!

谢谢。