ssl - 带有 HTTPS Nginx 入口 Controller 的 HAProxy LoadBalancer

Question

我正在使用 HAproxy 作为我的 Kubernetes 集群的本地负载均衡器。这是 cfg 文件:

global
  chroot      /var/lib/haproxy
  pidfile     /var/run/haproxy.pid
  maxconn     40000
  user        haproxy
  group       haproxy
  daemon
  tune.ssl.default-dh-param 2048
  log stdout local0  info
defaults
mode tcp
  log global
  option                  httplog
  retries                 3
  timeout http-request    50s
  timeout queue           1m
  timeout connect         1m
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 50s
  timeout check           10s
  maxconn                 1000
frontend https_front
  mode http
  bind *:443 ssl crt /etc/haproxy/haproxy.pem ca-file /etc/haproxy/haproxy.crt verify optional
  redirect scheme https if !{ ssl_fc }
  acl sadmin path_beg /sadmin
  use_backend sadmin_server if sadmin
  default_backend sadmin_server
backend sadmin_server
  balance roundrobin
  mode http
  server node1 staging-node1:30000 check-ssl verify required ca-file /etc/haproxy/backend-ca.crt
  server node2 staging-node2:30000 check-ssl verify required ca-file /etc/haproxy/backend-ca.crt
  server node3 staging-node3:30000 check-ssl verify required ca-file /etc/haproxy/backend-ca.crt
  server node4 staging-node4:30000 check-ssl verify required ca-file /etc/haproxy/backend-ca.crt

我使用了用于向 Kubernetes 中的入口对象颁发证书的相同 ca.crt。我在 cert-manager 中使用这个 ca 创建了一个发行者。

但是，现在我得到了错误:

none of the servers are available to take requests.

<134>Oct 28 21:18:59 haproxy[6]: 10.119.49.97:64484 [28/Oct/2019:21:18:56.891] https_front~ sadmin_server/node1 1/0/-1/-1/3046 503 237 - - SC-- 1/1/0/0/3 0/0 "GET /sadmin/ HTTP/1.1"

使用选项 ssl verify none ，流程有效。

谁能告诉我在这种情况下使用哪个证书来加密 haproxy 和 nginx 入口 Controller 之间的连接？

PS:我不使用 ssl 传递，因为我必须将不能发生的 acls 放在 tcp 模式下。

更新:

kubectl describe svc nginx-ingress -n ingress
Name:                     nginx-ingress
Namespace:                ingress
Labels:                   <none>
Annotations:              <none>
Selector:                 app=nginx-ingress-lb
Type:                     NodePort
IP:                       10.xxx.xx.xxx
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  32170/TCP
Endpoints:                10.xxx.xx.xxx:80
Port:                     http-mgmt  18080/TCP
TargetPort:               18080/TCP
NodePort:                 http-mgmt  32000/TCP
Endpoints:                10.xxx.xx.xxx:18080
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  30000/TCP
Endpoints:                10.xxx.xx.xxx:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:  

kubectl describe deployment ngins-ingress-controller -n ingress
Name:                   nginx-ingress-controller
Namespace:              ingress
CreationTimestamp:      Mon, 09 Sep 2019 19:00:45 +0000
Labels:                 app=nginx-ingress-lb
Annotations:            deployment.kubernetes.io/revision: 2
Selector:               app=nginx-ingress-lb
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           app=nginx-ingress-lb
  Service Account:  nginx
  Containers:
   nginx-ingress-controller:
    Image:       nginx-ingress-controller:0.9.0
    Ports:       80/TCP, 18080/TCP
    Host Ports:  0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=ingress/default-backend
      --configmap=ingress/nginx-ingress-controller-conf
      --v=2
    Liveness:   http-get http://:10254/healthz delay=10s timeout=5s period=10s #success=1 #failure=3
    Readiness:  http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
    Mounts:           <none>
  Volumes:            <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-ingress-controller-5cdf7fff4c (1/1 replicas created)
Events:          <none>

在该命名空间中没有定义入口

Answer 1

我解决这个问题的方法是在 nginx-ingress-controller default-ssl 证书参数中使用 CA 签名证书。现在所有不需要使用 cert-manager 证书的入口都可以使用这个 CA 签名证书进行 tls 通信。

在入口配置中要注意的一件事是不要提及 secretName。这样它将采用 nginx ingress 的默认证书。

  tls:
  - hosts:
    - myworld.com.com

您现在可以在 HA 代理中提供根证书，效果很好。