ssl - EKS - Envoy 动态转发代理 CA 验证错误

Question

我试图在 eks 中设置一个转发代理，使用一个 envoy sidecar-containers，它应该进行 TLS 发起；
IE。 envoy 从主容器获取 http 请求，执行 ssl 加密并使用文件 ca-certificates.crt 验证 ca。
envoy 配置基于官方文档 (https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/dynamic_forward_proxy_filter#config-http-filters-dynamic-forward-proxy)。
特使将 ca 文件 ca-certificates.crt 从 aws secrets manager 挂载到/etc/cert-volume/ca-certificates.crt，这似乎有效，因为该文件存在于容器中。
ca-certificates.crt 文件似乎也有效，因为以下 cmd，从我的本地机器运行 ，给出一个 HTTP 302 Found(远程 wget 不支持 --ca-certificate 参数):

wget --ca-certificate=ca-certificates.crt  https://git.tech.rz.blub.de
--2021-04-26 11:03:16--  https://git.tech.rz.blub.de/
Loaded CA certificate 'ca-certificates.crt'
Resolving git.tech.rz.blub.de (git.tech.rz.blub.de)... 10.104.85.137, 10.104.84.192, 10.104.86.175
Connecting to git.tech.rz.blub.de (git.tech.rz.blub.de)|10.104.85.137|:443... connected.
HTTP request sent, awaiting response... 302 Found

当我在 envoy-Container 上打开 shell 并尝试通过代理发送 http-Request 时，我收到 503 Service Unavailable:

export http_proxy="localhost:10000"
wget git.tech.rz.blub.de
Connecting to localhost:10000 (127.0.0.1:10000)
wget: server returned error: HTTP/1.1 503 Service Unavailable

正如特使日志显示的那样，请求已处理并且 DNS 解析成功，但是 ca-validation 失败并出现错误 TLS 错误:268435581:SSL 例程:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
日志可以在这里找到: https://pastebin.com/pUZZqhSA
完整的特使配置如下所示:

  admin:
  access_log_path: /dev/stdout
  address: 
    socket_address: {​ address: 127.0.0.1, port_value: 9901, protocol: TCP }​
static_resources:
  secrets:
  - name: server_cert
    tls_certificate:
      certificate_chain:
        filename: /etc/cert-volume/server.crt
      private_key:
        filename: /etc/cert-volume/server.key
  - name: validation_context
    validation_context:
      trusted_ca:
        filename: /etc/cert-volume/ca-certificates.crt
  listeners:
    - name: listener_incoming
      address:
        socket_address: {​ address: 0.0.0.0, port_value: 8443 }​
      filter_chains:
        filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_http
              route_config:
                name: local_route
                virtual_hosts:
                  - name: local_service
                    domains: ["*"]
                    routes:
                      - match: {​ prefix: "/config/" }​
                        route: {​ cluster: local_config }​
                      - match: {​ prefix: "/basic/" }​
                        route: {​ cluster: local }​
                      - match: {​ prefix: "/" }​
                        route: {​ cluster: local }​
              access_log:
              - name: envoy.access_loggers.file
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                  path: /dev/stdout
                  log_format:
                    json_format:
                      protocol: "%PROTOCOL%"
                      response_code: "%RESPONSE_CODE%"
                      response_code_details: "%RESPONSE_CODE_DETAILS%"
                      connection_termination_details: "%CONNECTION_TERMINATION_DETAILS%"
                      message: "%LOCAL_REPLY_BODY%"
                      response_flags: "%RESPONSE_FLAGS%"
                      local_reply_body: "%LOCAL_REPLY_BODY%"
              http_filters:
                  - name: envoy.filters.http.router
        transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
            common_tls_context:
              tls_certificate_sds_secret_configs:
              - name: server_cert
    - name: listener_outgoing
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 10000
      filter_chains:
      - filters:
        - name: envoy.filters.network.http_connection_manager
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            codec_type: AUTO
            stat_prefix: http_to_https
            route_config:
              name: local_route
              virtual_hosts:
              - name: app
                domains:
                - "*"
                routes:
                - match:
                    prefix: "/"
                  route:
                    cluster: dynamic_forward_proxy_cluster
            http_filters:
            - name: envoy.filters.http.dynamic_forward_proxy
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
                dns_cache_config:
                  name: dynamic_forward_proxy_cache_config
                  dns_lookup_family: V4_ONLY
            - name: envoy.filters.http.router
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: local
    connect_timeout: 0.25s
    type: STATIC
    load_assignment:
      cluster_name: local
      endpoints:
        - lb_endpoints:
          - endpoint:
              address:
                socket_address:
                  address: 127.0.0.1
                  port_value: 8080
  - name: local_config
    connect_timeout: 0.25s
    type: STATIC
    load_assignment:
      cluster_name: local_config
      endpoints:
        - lb_endpoints:
          - endpoint:
              address:
                socket_address:
                  address: 127.0.0.1
                  port_value: 8081
  - name: dynamic_forward_proxy_cluster
    connect_timeout: 1s
    lb_policy: CLUSTER_PROVIDED
    cluster_type:
      name: envoy.clusters.dynamic_forward_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
        dns_cache_config:
          name: dynamic_forward_proxy_cache_config
          dns_lookup_family: V4_ONLY
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:
          validation_context_sds_secret_config:
            name: validation_context

envoy-admin 端点显示，使用了正确的证书:

wget localhost:9901/certs
Connecting to localhost:9901 (127.0.0.1:9901)
saving to 'certs'
certs                100% |*********************************************|   710  0:00:00 ETA
'certs' saved
/ # cat certs
{​
 "certificates": [
  {​
   "ca_cert": [
    {​
     "path": "/etc/cert-volume/ca-certificates.crt",
     "serial_number": "612a55dd000000000002",
     "subject_alt_names": [],
     "days_until_expiration": "1305",
     "valid_from": "2013-11-21T15:40:58Z",
     "expiration_time": "2024-11-21T15:50:58Z"
    }​
   ],
   "cert_chain": []
  }​,
  {​
   "ca_cert": [],
   "cert_chain": [
    {​
     "path": "/etc/cert-volume/server.crt",
     "serial_number": "74af53b8463a1714",
     "subject_alt_names": [
      {​
       "dns": "*.test.comp.blub.de"
      }​
     ],
     "days_until_expiration": "173",
     "valid_from": "2019-10-18T07:10:38Z",
     "expiration_time": "2021-10-17T07:10:38Z"
    }​
   ]
  }​
 ]
}​

有谁知道，这里出了什么问题？

Answer 1

问题解决了。
ca-certificates.crt 缺少必需的 CA。我不知道为什么 wget 没有抛出异常，但是 openssl 确实......