nginx - Kubernetes - 使用 gRPC 进入

Question

我有一个非常简单的 grpc-server。当我直接通过 NodePort 公开它们时，一切正常。我对此没有任何问题。

grpc-client --> NodePort --> grpc-server

我将 NodePort 服务更改为 ClusterIP 服务，并尝试使用 ingress controller 将流量路由到 grpc-server。设置是这样的。 ingress 和 grpc-server 是 k8s 集群的一部分。 grpc-client 在外面——这是我的本地机器。

grpc-client --> ingress --> clusterip --> grpc-server

我的入口是这样的

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: web-ingress
  namespace: test
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;    
spec:
  rules:
  - http:
      paths:
      - backend:
         serviceName: grpc-server
         servicePort: 6565

每当我发送请求时，我都会在客户端收到这样的异常。

io.grpc.StatusRuntimeException: INTERNAL: HTTP/2 error code: FRAME_SIZE_ERROR
Received Rst Stream

Ingress日志是这样的

127.0.0.1 - - [28/Jul/2020:03:52:38 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - c9d46af55aa8a2b450f9f72e9d550023
127.0.0.1 - - [28/Jul/2020:03:52:38 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.000 [] [] - - - - a9e0f02851473467fc553f2a4645377d
127.0.0.1 - - [28/Jul/2020:03:52:40 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - fcc0668f300bd5e7e12325ae7c26fa9d
127.0.0.1 - - [28/Jul/2020:03:52:41 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - dea2b8eb8f6884ac512654af3c2e6b47
127.0.0.1 - - [28/Jul/2020:03:52:43 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - 0357acafce928ee05aaef1334eed1b69
127.0.0.1 - - [28/Jul/2020:03:52:44 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - df9b75d79d9177e39a901dfa020912f2
127.0.0.1 - - [28/Jul/2020:03:52:44 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.000 [] [] - - - - bb27acd5b86fd057af1ca8fe93f7d857
127.0.0.1 - - [28/Jul/2020:03:52:44 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - 2ced9207652fc564bb94e0d4a14ae6fc
127.0.0.1 - - [28/Jul/2020:03:52:45 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - b99fab13df2f64a6a0a901237dadfa27
127.0.0.1 - - [28/Jul/2020:03:52:45 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.002 [] [] - - - - 40cb75f4e4b3df28be66044a0825f1c6
127.0.0.1 - - [28/Jul/2020:03:52:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - 89cac6709278b4e54b0349075500bf74
127.0.0.1 - - [28/Jul/2020:03:52:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.000 [] [] - - - - 0b032199a053069df185c620de497c90
127.0.0.1 - - [28/Jul/2020:03:52:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - 5a07b1e24ea3e8e2066a7ed8950f7040
127.0.0.1 - - [28/Jul/2020:03:53:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.000 [] [] - - - - 0f63eec0d0ccb9cc87c2fbb1fdfae48f

看起来 grpc-server 从未收到任何请求，因为我在那里没有看到任何日志。

这是怎么回事？

Answer 1

您遇到的错误来自 nginx 入口 Controller 的默认配置(这取自 ingress github config ):

// Enables or disables the HTTP/2 support in secure connections
// http://nginx.org/en/docs/http/ngx_http_v2_module.html
// Default: true
UseHTTP2 bool `json:"use-http2,omitempty"`

这也可以通过检查您的入口 Controller 设置来验证:

kubectl exec -it -n <name_space> <nginx_ingress_controller_pod> -- cat /etc/nginx/nginx.conf 

## start server _
   server {
          server_name _ ;

          listen 80 default_server reuseport backlog=511 ;
          listen 443 default_server reuseport backlog=511 ssl http2 ;

您正在尝试通过默认配置中不支持 http2 的端口 80 进行连接。 GRCP 仅适用于 http2，因此您应该尝试使用端口 443 发出请求。

让我知道它是否有效。