gpt4 book ai didi

kubernetes - iptables 模式下的 kube-proxy 无法正常工作

转载 作者:行者123 更新时间:2023-12-04 19:02:40 27 4
gpt4 key购买 nike

我有

  • Kubernetes:v.1.1.1
  • iptables v1.4.21
  • 内核:4.2.0-18-generic,Ubuntu 自带
  • 网络通过在交换机
  • 上终止的 L2 VLAN 完成
  • 没有云提供商

  • 我在做什么

    我正在尝试使用 kube-proxy 的 iptables 模式。我已使用 --proxy_mode=iptables 启用它争论。似乎缺少一些规则:
    iptables -t nat -nvL

    Chain PREROUTING (policy ACCEPT 8 packets, 459 bytes)
    pkts bytes target prot opt in out source destination
    2116 120K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */

    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target prot opt in out source destination

    Chain OUTPUT (policy ACCEPT 2 packets, 120 bytes)
    pkts bytes target prot opt in out source destination
    718 45203 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */

    Chain POSTROUTING (policy ACCEPT 5 packets, 339 bytes)
    pkts bytes target prot opt in out source destination
    0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ mark match 0x4d415351

    Chain KUBE-NODEPORTS (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/docker-registry-fe:tcp */ tcp dpt:31195 MARK set 0x4d415351
    0 0 KUBE-SVC-XZFGDLM7GMJHZHOY tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/docker-registry-fe:tcp */ tcp dpt:31195
    0 0 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* mngbox/jumpbox:ssh */ tcp dpt:30873 MARK set 0x4d415351
    0 0 KUBE-SVC-GLKZVFIDXOFHLJLC tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* mngbox/jumpbox:ssh */ tcp dpt:30873

    Chain KUBE-SEP-5IXMK7UWPGVTWOJ7 (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK all -- * * 10.116.160.8 0.0.0.0/0 /* mngbox/jumpbox:ssh */ MARK set 0x4d415351
    0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* mngbox/jumpbox:ssh */ tcp to:10.116.160.8:22

    Chain KUBE-SEP-BNPLX5HQYOZINWEQ (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK all -- * * 10.116.161.6 0.0.0.0/0 /* kube-system/monitoring-influxdb:api */ MARK set 0x4d415351
    0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-influxdb:api */ tcp to:10.116.161.6:8086

    Chain KUBE-SEP-CJMHKLXPTJLTE3OP (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK all -- * * 10.116.254.2 0.0.0.0/0 /* default/kubernetes: */ MARK set 0x4d415351
    0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes: */ tcp to:10.116.254.2:6443


    Chain KUBE-SEP-GSM3BZTEXEBWDXPN (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK all -- * * 10.116.160.7 0.0.0.0/0 /* kube-system/kube-dns:dns */ MARK set 0x4d415351
    0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.116.160.7:53


    Chain KUBE-SEP-OAYOAJINXRPUQDA3 (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK all -- * * 10.116.160.7 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ MARK set 0x4d415351
    0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:10.116.160.7:53

    Chain KUBE-SEP-PJJZDQNXDGWM7MU6 (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK all -- * * 10.116.160.5 0.0.0.0/0 /* default/docker-registry-fe:tcp */ MARK set 0x4d415351
    0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/docker-registry-fe:tcp */ tcp to:10.116.160.5:443

    Chain KUBE-SEP-RWODGLKOVWXGOHUR (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK all -- * * 10.116.161.6 0.0.0.0/0 /* kube-system/monitoring-influxdb:http */ MARK set 0x4d415351
    0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-influxdb:http */ tcp to:10.116.161.6:8083

    Chain KUBE-SEP-WE3Z7KMHA6KPJWKK (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK all -- * * 10.116.161.6 0.0.0.0/0 /* kube-system/monitoring-grafana: */ MARK set 0x4d415351
    0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-grafana: */ tcp to:10.116.161.6:8080

    Chain KUBE-SEP-YBQVM4LA4YMMZIWH (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK all -- * * 10.116.161.3 0.0.0.0/0 /* kube-system/monitoring-heapster: */ MARK set 0x4d415351
    0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-heapster: */ tcp to:10.116.161.3:8082

    Chain KUBE-SEP-YMZS7BLP4Y6MWTX5 (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK all -- * * 10.116.160.9 0.0.0.0/0 /* infra/docker-registry-backend:docker-registry-backend */ MARK set 0x4d415351
    0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* infra/docker-registry-backend:docker-registry-backend */ tcp to:10.116.160.9:5000

    Chain KUBE-SEP-ZDOOYAKDERKR43R3 (1 references)
    pkts bytes target prot opt in out source destination
    0 0 MARK all -- * * 10.116.160.10 0.0.0.0/0 /* default/kibana-logging: */ MARK set 0x4d415351
    0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kibana-logging: */ tcp to:10.116.160.10:5601

    Chain KUBE-SERVICES (2 references)
    pkts bytes target prot opt in out source destination
    0 0 KUBE-SVC-JRXTEHDDTAFMSEAS tcp -- * * 0.0.0.0/0 10.116.0.48 /* kube-system/monitoring-grafana: cluster IP */ tcp dpt:80
    0 0 KUBE-SVC-CK6HVV5A27TDFNIA tcp -- * * 0.0.0.0/0 10.116.0.188 /* kube-system/monitoring-influxdb:api cluster IP */ tcp dpt:8086
    0 0 KUBE-SVC-DKEW3YDJFV3YJLS2 tcp -- * * 0.0.0.0/0 10.116.0.6 /* infra/docker-registry-backend:docker-registry-backend cluster IP */ tcp dpt:5000
    0 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.116.0.2 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
    0 0 KUBE-SVC-WEHLQ23XZWSA5ZX3 tcp -- * * 0.0.0.0/0 10.116.0.188 /* kube-system/monitoring-influxdb:http cluster IP */ tcp dpt:8083
    0 0 KUBE-SVC-XZFGDLM7GMJHZHOY tcp -- * * 0.0.0.0/0 10.116.1.142 /* default/docker-registry-fe:tcp cluster IP */ tcp dpt:443
    0 0 MARK tcp -- * * 0.0.0.0/0 10.116.254.3 /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 MARK set 0x4d415351
    0 0 KUBE-SVC-XZFGDLM7GMJHZHOY tcp -- * * 0.0.0.0/0 10.116.254.3 /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
    0 0 KUBE-SVC-XZFGDLM7GMJHZHOY tcp -- * * 0.0.0.0/0 10.116.254.3 /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 ADDRTYPE match dst-type LOCAL
    0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.116.0.2 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
    0 0 KUBE-SVC-7IHGTXJ4CF2KVXJZ tcp -- * * 0.0.0.0/0 10.116.1.126 /* kube-system/monitoring-heapster: cluster IP */ tcp dpt:80
    0 0 KUBE-SVC-GLKZVFIDXOFHLJLC tcp -- * * 0.0.0.0/0 10.116.1.175 /* mngbox/jumpbox:ssh cluster IP */ tcp dpt:2345
    0 0 MARK tcp -- * * 0.0.0.0/0 10.116.254.3 /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 MARK set 0x4d415351
    0 0 KUBE-SVC-GLKZVFIDXOFHLJLC tcp -- * * 0.0.0.0/0 10.116.254.3 /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
    0 0 KUBE-SVC-GLKZVFIDXOFHLJLC tcp -- * * 0.0.0.0/0 10.116.254.3 /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 ADDRTYPE match dst-type LOCAL
    0 0 KUBE-SVC-6N4SJQIF3IX3FORG tcp -- * * 0.0.0.0/0 10.116.0.1 /* default/kubernetes: cluster IP */ tcp dpt:443
    0 0 KUBE-SVC-B6ZEWWY2BII6JG2L tcp -- * * 0.0.0.0/0 10.116.0.233 /* default/kibana-logging: cluster IP */ tcp dpt:8888
    0 0 MARK tcp -- * * 0.0.0.0/0 10.116.254.3 /* default/kibana-logging: external IP */ tcp dpt:8888 MARK set 0x4d415351
    0 0 KUBE-SVC-B6ZEWWY2BII6JG2L tcp -- * * 0.0.0.0/0 10.116.254.3 /* default/kibana-logging: external IP */ tcp dpt:8888 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
    0 0 KUBE-SVC-B6ZEWWY2BII6JG2L tcp -- * * 0.0.0.0/0 10.116.254.3 /* default/kibana-logging: external IP */ tcp dpt:8888 ADDRTYPE match dst-type LOCAL
    0 0 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

    Chain KUBE-SVC-6N4SJQIF3IX3FORG (1 references)
    pkts bytes target prot opt in out source destination
    0 0 KUBE-SEP-CJMHKLXPTJLTE3OP all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes: */

    Chain KUBE-SVC-7IHGTXJ4CF2KVXJZ (1 references)
    pkts bytes target prot opt in out source destination
    0 0 KUBE-SEP-YBQVM4LA4YMMZIWH all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-heapster: */

    Chain KUBE-SVC-B6ZEWWY2BII6JG2L (3 references)
    pkts bytes target prot opt in out source destination
    0 0 KUBE-SEP-ZDOOYAKDERKR43R3 all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kibana-logging: */

    Chain KUBE-SVC-CK6HVV5A27TDFNIA (1 references)
    pkts bytes target prot opt in out source destination
    0 0 KUBE-SEP-BNPLX5HQYOZINWEQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-influxdb:api */

    Chain KUBE-SVC-DKEW3YDJFV3YJLS2 (1 references)
    pkts bytes target prot opt in out source destination
    0 0 KUBE-SEP-YMZS7BLP4Y6MWTX5 all -- * * 0.0.0.0/0 0.0.0.0/0 /* infra/docker-registry-backend:docker-registry-backend */

    Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
    pkts bytes target prot opt in out source destination
    0 0 KUBE-SEP-OAYOAJINXRPUQDA3 all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */

    Chain KUBE-SVC-GLKZVFIDXOFHLJLC (4 references)
    pkts bytes target prot opt in out source destination
    0 0 KUBE-SEP-5IXMK7UWPGVTWOJ7 all -- * * 0.0.0.0/0 0.0.0.0/0 /* mngbox/jumpbox:ssh */

    Chain KUBE-SVC-JRXTEHDDTAFMSEAS (1 references)
    pkts bytes target prot opt in out source destination
    0 0 KUBE-SEP-WE3Z7KMHA6KPJWKK all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-grafana: */

    Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
    pkts bytes target prot opt in out source destination
    0 0 KUBE-SEP-GSM3BZTEXEBWDXPN all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */

    Chain KUBE-SVC-WEHLQ23XZWSA5ZX3 (1 references)
    pkts bytes target prot opt in out source destination
    0 0 KUBE-SEP-RWODGLKOVWXGOHUR all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-influxdb:http */

    Chain KUBE-SVC-XZFGDLM7GMJHZHOY (4 references)
    pkts bytes target prot opt in out source destination
    0 0 KUBE-SEP-PJJZDQNXDGWM7MU6 all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/docker-registry-fe:tcp */

    当我请求服务 ip 时,在我的情况下它是 10.116.0.2 我收到一个错误
    ;; connection timed out; no servers could be reached

    而当我向 10.116.160.7 服务器发出请求时,它工作正常。
    我可以看到流量根本没有指向 kube-proxy 规则,所以可能缺少一些东西。

    我将非常感谢有关缺少规则的任何提示

    编辑
    我用 thokin 请求的缺失信息更新了我的初始请求,他指出了调试 kube-proxy 的 iptables 规则的非常好的方法,我可以通过以下方式确定我的问题:
    for c in PREROUTING OUTPUT POSTROUTING; do iptables -t nat -I $c -d 10.116.160.7 -j LOG --log-prefix "DBG@$c: "; done
    for c in PREROUTING OUTPUT POSTROUTING; do iptables -t nat -I $c -d 10.116.0.2 -j LOG --log-prefix "DBG@$c: "; done

    然后我执行了以下命令:
    # nslookup kubernetes.default.svc.psc01.cluster 10.116.160.7
    服务器:10.116.160.7
    地址:10.116.160.7#53
    Name:   kubernetes.default.svc.psc01.cluster
    Address: 10.116.0.1

    # nslookup kubernetes.default.svc.psc01.cluster 10.116.0.2
    ;; connection timed out; no servers could be reached

    结果,我得到了不同的“源”地址和传出接口(interface):
    [701768.263847] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=12436 PROTO=UDP SPT=54501 DPT=53 LEN=62 
    [702620.454211] DBG@OUTPUT: IN= OUT=docker0 SRC=10.116.176.1 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=22733 PROTO=UDP SPT=28704 DPT=53 LEN=62
    [702620.454224] DBG@POSTROUTING: IN= OUT=docker0 SRC=10.116.176.1 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=22733 PROTO=UDP SPT=28704 DPT=53 LEN=62
    [702626.318258] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62
    [702626.318263] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62
    [702626.318266] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62
    [702626.318270] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62
    [702626.318284] DBG@POSTROUTING: IN= OUT=docker0 SRC=10.116.250.252 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62

    所以,通过添加路线
    ip route add 10.116.0.0/23 dev docker0

    现在它工作正常!

    最佳答案

    对于 future ,iptables-save 的结果更容易阅读(无论如何对我来说)。

    我看不出这里有什么遗漏。
    KUBE-SERVICES捕获 10.116.0.2 端口 53/UDP 并将其传递给 KUBE-SVC-TCOU7JCQXEZGVUNUKUBE-SVC-TCOU7JCQXEZGVUNU只有一个端点,所以跳转到 KUBE-SEP-GSM3BZTEXEBWDXPNKUBE-SEP-GSM3BZTEXEBWDXPN DNAT 到 10.116.160.7 端口 53/UDP

    如果您断言 10.116.160.7 有效而 10.116.0.2 无效,那确实很奇怪。这表明 iptables 规则根本没有触发。您是从节点本身还是从容器进行测试?

    你用的是什么网络? L3(垫层?)法兰绒? OVS?还有什么?

    什么云提供商(如果有)?

    调试的第一步:运行:for c in PREROUTING OUTPUT; do iptables -t nat -I $c -d 10.116.0.2 -j LOG --log-prefix "DBG@$c: "; done
    这会将 iptables 看到的任何数据包记录到您的服务 IP。现在看看 dmesg .

    关于kubernetes - iptables 模式下的 kube-proxy 无法正常工作,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/34001758/

    27 4 0
    Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
    广告合作:1813099741@qq.com 6ren.com