ssl - Kubernetes Haproxy Ingress + Nginx 作为后端 Https

Question

我用这个软件

Kubernetes 1.11.5
Haproxy: last
Nginx: 1.15.7

我从我购买的由 comodo CA 制造的证书中制作 default/tls-secret

并得到这个错误:

  Error code: SSL_ERROR_RX_RECORD_TOO_LONG

这是我的配置Haproxy 入口

---
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-controller
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-controller
  namespace: ingress-controller
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: ingress-controller
  namespace: ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
      - create
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-controller
subjects:
  - kind: ServiceAccount
    name: ingress-controller
    namespace: ingress-controller
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: ingress-controller
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: ingress-controller
  namespace: ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-controller
subjects:
  - kind: ServiceAccount
    name: ingress-controller
    namespace: ingress-controller
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: ingress-controller
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: ingress-default-backend
  name: ingress-default-backend
  namespace: ingress-controller
spec:
  selector:
    matchLabels:
      run: ingress-default-backend
  template:
    metadata:
      labels:
        run: ingress-default-backend
    spec:
      containers:
      - name: ingress-default-backend
        image: gcr.io/google_containers/defaultbackend:1.0
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-default-backend
  namespace: ingress-controller
spec:
  ports:
  - port: 8080
  selector:
    run: ingress-default-backend
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: haproxy-ingress
  namespace: ingress-controller
data:
  ssl-options: force-tlsv12

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    run: haproxy-ingress
  name: haproxy-ingress
  namespace: ingress-controller
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      run: haproxy-ingress
  template:
    metadata:
      labels:
        run: haproxy-ingress
    spec:
      hostNetwork: true
      nodeSelector:
        role: edge-router
      serviceAccountName: ingress-controller
      containers:
      - name: haproxy-ingress
        image: quay.io/jcmoraisjr/haproxy-ingress
        args:
        - --default-backend-service=$(POD_NAMESPACE)/ingress-default-backend
        - --default-ssl-certificate=default/tls-secret
        - --configmap=$(POD_NAMESPACE)/haproxy-ingress
        - --sort-backends
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: stat
          containerPort: 1936
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace

这是我的应用程序和它的入口配置

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: meteo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: meteo
  template:
    metadata:
      labels:
        app: meteo
    spec:
      containers:
      - name: meteo
        image: devprofi/meteo:v39

        ports:
          - containerPort: 443
      imagePullSecrets:
        - name: meteo-secret

---
apiVersion: v1
kind: Service
metadata:
  name: meteo-svc
  namespace: default
spec:
  type: NodePort
  ports:
#    - port: 80
#      targetPort: 80
#      protocol: TCP
#      name: http
    - port: 443
      targetPort: 443
      protocol: TCP
      name: https
  selector:
    app: meteo

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
    kubernetes.io/ingress.class: "haproxy"
    ingress.kubernetes.io/secure-backends: "true"   
    ingress.kubernetes.io/backend-protocol: "HTTPS"   
  name: meteo-ingress
  namespace: default
spec:
  tls:
  - hosts:
    - meteotravel.ru
    secretName: cafe-secret  # this is another copy of secret made from my buyed cert and key
  rules:
  - host: meteotravel.ru
    http:
      paths:
      - path: /
        backend:
          serviceName: meteo-svc
          servicePort: 443

我尝试这个命令并得到错误

openssl s_client -connect meteotravel.ru:443

-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=PositiveSSL/CN=meteotravel.ru
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 6055 bytes and written 312 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: BDD996AF8814404E3E385A6FBE49F56CA2668C54FD157FD3FB28F38DB64F771E
    Session-ID-ctx: 
    Master-Key: F8EB4A4DA674F286E44C71605DF1D7DE4A6FE58D249162B086CE17E899FAC88CFA213018F89B8A9939CB842639D2B68A
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 5e 9a 46 20 a8 60 30 88-fa 2e c5 37 b7 29 0b 4e   ^.F .`0....7.).N
    0010 - 41 67 2b b6 e7 8e 2e 12-8b 55 0c ad 59 80 f7 d5   Ag+......U..Y...
    0020 - d1 07 8e fc 92 a1 2e 01-59 cf 00 2d d5 39 11 10   ........Y..-.9..
    0030 - bf f3 89 af 2d 7a 02 59-49 54 3a bf e4 8b 97 f3   ....-z.YIT:.....
    0040 - 55 da 4b 6f 9b 86 c4 85-eb e4 f9 a1 e3 74 76 be   U.Ko.........tv.
    0050 - 65 57 76 ec e3 76 c9 c8-5a 47 c6 c2 ee eb bd ec   eWv..v..ZG......
    0060 - 61 88 7c 35 8c a6 c0 b3-25 b5 79 06 99 df 66 75   a.|5....%.y...fu
    0070 - 8e 9d 3a 17 61 40 7c 1c-09 e3 07 aa 49 b9 c3 cf   ..:.a@|.....I...
    0080 - d7 ff 7d 1b cc 3f b9 3f-c7 bd ad 4d f9 4f eb 6c   ..}..?.?...M.O.l
    0090 - 6f 42 2e c8 30 75 a9 07-d4 9e f0 12 6b 9c ca ac   oB..0u......k...

    Start Time: 1544706461
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
---
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
closed

我也试过这个命令

     curl -vL https://meteotravel.ru >/dev/null

* Rebuilt URL to: https://meteotravel.ru/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 212.26.248.233...
* TCP_NODELAY set
* Connected to meteotravel.ru (212.26.248.233) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [222 bytes data]
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* stopped the pause stream!
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

我的 nginx 可以独立工作有配置

server {
access_log /var/log/nginx/default.access.log;
error_log /var/log/nginx/default.error.log warn;


    listen 443 ssl default;
    #listen 443 ssl http2 default reuseport;


   # Redirect HTTP to HTTPS
    if ($scheme = http) {
        return 301 https://$host$request_uri;
    }


    ssl_certificate /etc/nginx/ssl/meteotravel.ru/mt.crt;
    ssl_certificate_key /etc/nginx/ssl/meteotravel.ru/pk;
    server_name meteotravel.ru;



    root /var/www/html;

    # Add index.php to the list if you are using PHP
    index index.html index.htm index.nginx-debian.html;

#   server_name _;

    location /fop2{
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.


    # Add index.php to the list if you are using PHP

        try_files $uri $uri/ =404;
    }
location /{
        try_files $uri $uri/ =404;
    }

}

哪里出错了？？

这是我在服务器上的 haproxy 配置

我还尝试为 nginx 设置 proxy_protocol:listen 443 ssl proxy_protocol;和 haproxy-ingress 的选项:代理协议(protocol):[v1|v2|v2-ssl|v2-ssl-cn]所有这一切都是分开的并在 nginx 后端的日志中获取错误

����kjih9876������2��.��*��&����=5"while reading PROXY protocol, client: 10.244.5.0, server: 0.0.0.0:4432018 年 12 月 13 日 19:55:24 [错误] 7#7:2271 损坏的 header :“�6�v+\��w �?����3�Alm9i��L$&��h ��0��,��(��$�� ����kjih9876������2��..��&����=5"while reading PROXY protocol, client: 10.244.6.0, server: 0.0.0.0:443

Answer 1

这个问题是因为注释 ingress.kubernetes.io/secure-backends: "true" 。它不需要，因为我们已经从 haproxy 中获取了 TCP 流中的安全数据。有了这个注解我们做了两次加密，nginx 不能正确解密