kubernetes - 如何让 hasura websocket 在我的本地 Kubernetes 集群上工作？

Question

我已经在 Windows 下建立了一个本地 K8s 集群，如下所示:

为桌面安装 docker

在桌面版 docker 中，启用 kubernetes

安装 nginx 入口 Controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.29.0/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.29.0/deploy/static/provider/cloud-generic.yaml

将以下域添加到主机 ( C:\Windows\System32\drivers\etc\hosts )

127.0.0.1  localhost api.shopozor

我在这里没有做任何特别的事情，我将所有内容都保留为默认设置。
然后，我使用以下 yamls 将 hasura 部署到我的集群(为简洁起见，我没有显示 postgres 部署):

---
# Source: api/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: api
  labels:
    app.kubernetes.io/name: api
    helm.sh/chart: api-0.0.0
    app.kubernetes.io/instance: api
    app.kubernetes.io/version: "0.0"
    app.kubernetes.io/managed-by: Helm
type: Opaque
data:
  admin-secret: "c2VjcmV0"
---
# Source: api/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: api
  labels:
    app.kubernetes.io/name: api
    helm.sh/chart: api-0.0.0
    app.kubernetes.io/instance: api
    app.kubernetes.io/version: "0.0"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
    - port: 8080
      targetPort: 8080
      # TODO: we cannot use string port because devspace doesn't support it in its UI
      # targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: api
    app.kubernetes.io/instance: api
---
# Source: api/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app.kubernetes.io/name: api
    helm.sh/chart: api-0.0.0
    app.kubernetes.io/instance: api
    app.kubernetes.io/version: "0.0"
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: api
      app.kubernetes.io/instance: api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: api
        app.kubernetes.io/instance: api
    spec:
      serviceAccountName: api
      securityContext:
        {}
      initContainers:
      # App has to wait for the database to be online "depends_on" workaround
      - name: wait-for-db
        image: darthcabs/tiny-tools:1
        args:
        - /bin/bash
        - -c
        - >
          set -x;
          while [[ "$(nc -zv 'postgres' 5432 &> /dev/null; echo $?)" != 0 ]]; do
            echo '.'
            sleep 15;
          done
      containers:
      - name: api
        securityContext:
            {}
        image: shopozor/graphql-engine:EM5Aya
        imagePullPolicy: 
        env:
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: shared-postgresql
              key: postgresql-username
        - name: POSTGRES_DATABASE
          value: shopozor
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: shared-postgresql
              key: postgresql-password
        - name: POSTGRES_HOST
          value: postgres
        - name: POSTGRES_PORT
          value: "5432"
        - name: HASURA_GRAPHQL_SERVER_PORT
          value: "8080"
        - name: HASURA_GRAPHQL_ENABLE_CONSOLE
          value: "true"
        - name: HASURA_GRAPHQL_ENABLED_LOG_TYPES
          value: startup, http-log, webhook-log, websocket-log, query-log
        - name: HASURA_GRAPHQL_ENABLE_TELEMETRY
          value: "false"
        - name: HASURA_GRAPHQL_CORS_DOMAIN
          value: "*"
        - name: HASURA_GRAPHQL_DISABLE_CORS
          value: "false"
        - name: HASURA_GRAPHQL_UNAUTHORIZED_ROLE
          value: incognito
        - name: HASURA_GRAPHQL_ADMIN_SECRET
          valueFrom:
            secretKeyRef:
              name: api
              key: admin-secret
        - name: HASURA_GRAPHQL_JWT_SECRET
          value: "{\"type\": \"HS256\", \"key\": \"my-access-token-signing-key-secret\", \"audience\": [\"58640fbe-9a6c-11ea-bb37-0242ac130002\", \"6e707590-9a6c-11ea-bb37-0242ac130002\"], \"claims_namespace\": \"https://hasura.io/jwt/claims\", \"claims_format\": \"json\", \"issuer\": \"shopozor.com\" }"
        - name: HASURA_GRAPHQL_DATABASE_URL
          value: postgres://$(POSTGRES_USER):$(POSTGRES_PASSWORD)@$(POSTGRES_HOST):$(POSTGRES_PORT)/$(POSTGRES_DATABASE)
        - name: FUNCTION_NAMESPACE
          value: dev
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: http
        readinessProbe:
          httpGet:
            path: /healthz
            port: http
        resources:
            {}
---
# Source: api/templates/ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: api
  labels:
    app.kubernetes.io/name: api
    helm.sh/chart: api-0.0.0
    app.kubernetes.io/instance: api
    app.kubernetes.io/version: "0.0"
    app.kubernetes.io/managed-by: Helm
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: "api.shopozor"
      http:
        paths:
          - path: /
            backend:
              serviceName: api
              servicePort: 8080

现在，我有一个尝试使用 hasura 的 websocket 的 nuxt 前端。我以标准方式配置了 apollo

//---
// nuxt.config.js
[...]
  // Give apollo module options
  apollo: {
    cookieAttributes: {
      expires: 7
    },
    includeNodeModules: true, 
    authenticationType: 'Basic', 
    clientConfigs: {
      default: '~/apollo/clientConfig.js'
    }
  },
[...]
//---
// apollo/clientConfig.js
import { InMemoryCache } from 'apollo-cache-inmemory'
export default function (context) {
  return {
    httpLinkOptions: {
      uri: 'http://api.shopozor/v1/graphql',
      credentials: 'same-origin'
    },
    cache: new InMemoryCache(),
    wsEndpoint: 'ws://localhost:8080/v1/graphql'
  }
}

请注意，我目前不需要特定的标题。我应该能够在没有授权 token 的情况下访问 websocket。
现在，当我启动我的应用程序时，websocket 连接会尝试初始化。如果我端口转发我的 hasura 服务，那么上面的配置就可以了。 websocket 连接似乎工作正常。至少，哈苏拉日志显示

2020-07-16T06:49:59.937386882Z {"type":"websocket-log","timestamp":"2020-07-16T06:49:59.937+0000","level":"info","detail":{"event":{"type":"accepted"},"connection_info":{"websocket_id":"8437b784-1fce-4430-9ca9-a9e7517307f0","token_expiry":null,"msg":null},"user_vars":null}}

但是，如果我更改 wsEndpoint在上述配置中使用入口进入我的 hasura 实例，

wsEndpoint: 'ws://api.shopozor/v1/graphql'

然后它不再起作用了。相反，我不断得到 404 Not Found .但是，我可以通过 http://api.shopozor 访问 hasura 控制台。 . hasura 日志显示

2020-07-16T10:37:53.564657244Z {"type":"websocket-log","timestamp":"2020-07-16T10:37:53.564+0000","level":"error","detail":{"event":{"type":"rejected","detail":{"path":"$","error":"only '/v1/graphql', '/v1alpha1/graphql' are supported on websockets","code":"not-found"}},"connection_info":{"websocket_id":"5e031467-fb5c-460d-b2a5-11f1e21f22e7","token_expiry":null,"msg":null},"user_vars":null}}

所以我搜索了很多，找到了一些关于我应该在我的入口中使用的注释的信息，但没有任何效果。我在这里想念什么？我需要特定的 nginx 入口 Controller 配置吗？我是否需要将一些特殊注释传递给我的 hasura 入口？我需要做什么才能使其工作？
编辑
要回答这篇文章的问题，这是我在集群上应用的 hasura 入口:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: api
    meta.helm.sh/release-namespace: dev
  labels:
    app.kubernetes.io/instance: api
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: api
    app.kubernetes.io/version: '0.0'
    helm.sh/chart: api-0.0.0
  name: api
spec:
  rules:
    - host: api.shopozor
      http:
        paths:
          - backend:
              serviceName: api
              servicePort: 8080
            path: /

编辑 2
对于我的 hasura 实例，使用以下入口

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: api
    meta.helm.sh/release-namespace: dev
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_http_version 1.1;
      proxy_set_header Upgrade "websocket";
      proxy_set_header Connection "Upgrade";
  labels:
    app.kubernetes.io/instance: api
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: api
    app.kubernetes.io/version: '0.0'
    helm.sh/chart: api-0.0.0
  name: api
spec:
  rules:
    - host: api.shopozor
      http:
        paths:
          - backend:
              serviceName: api
              servicePort: 8080
            path: /

前端应用程序仍然存在与 websockets 相同的问题。此外，它已经无法真正连接到hasura了。相反，我收到以下错误:

 ERROR  Network error: Unexpected token < in JSON at position 0                                                                                                                                         23:17:06

  at new ApolloError (D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:92:26)
  at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:1588:34
  at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:2008:15
  at Set.forEach (<anonymous>)
  at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:2006:26
  at Map.forEach (<anonymous>)
  at QueryManager.broadcastQueries (D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:2004:20)
  at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:1483:29
  at runMicrotasks (<anonymous>)
  at processTicksAndRejections (internal/process/task_queues.js:97:5)

Global error handler                                                                                                                                                                                    23:17:06

 ERROR  Network error: Unexpected token < in JSON at position 0                                                                                                                                         23:17:06

  at new ApolloError (D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:92:26)
  at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:1486:27
  at runMicrotasks (<anonymous>)
  at processTicksAndRejections (internal/process/task_queues.js:97:5)

也

client.js?06a0:49 ApolloError: Network error: Unexpected token < in JSON at position 0
    at new ApolloError (D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:92:26)
    at D:\workspace\shopozor\services\node_modules\apollo-client\bundle.umd.js:1486:27
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  graphQLErrors: [],
  networkError: SyntaxError [ServerParseError]: Unexpected token < in JSON at position 0
      at JSON.parse (<anonymous>)
      at D:\workspace\shopozor\services\node_modules\apollo-link-http-common\lib\index.js:35:25
      at runMicrotasks (<anonymous>)
      at processTicksAndRejections (internal/process/task_queues.js:97:5) {
    name: 'ServerParseError',
    response: Body {
      url: 'http://api.shopozor/v1/graphql/',
      status: 404,
      statusText: 'Not Found',
      headers: [Headers],
      ok: false,
      body: [PassThrough],
      bodyUsed: true,
      size: 0,
      timeout: 0,
      _raw: [Array],
      _abort: false,
      _bytes: 153
    },
    statusCode: 404,
    bodyText: '<html>\r\n' +
      '<head><title>404 Not Found</title></head>\r\n' +
      '<body>\r\n' +
      '<center><h1>404 Not Found</h1></center>\r\n' +
      '<hr><center>nginx/1.17.8</center>\r\n' +
      '</body>\r\n' +
      '</html>\r\n'
  },
  message: 'Network error: Unexpected token < in JSON at position 0',
  extraInfo: undefined
}

和

index.js?a6d6:111 OPTIONS http://api.shopozor/v1/graphql/ net::ERR_ABORTED 404 (Not Found)

和

Access to fetch at 'http://api.shopozor/v1/graphql/' from origin 'http://localhost:3000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

没有新的注释

nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_http_version 1.1;
      proxy_set_header Upgrade "websocket";
      proxy_set_header Connection "Upgrade";

在我的 hasura 入口上，我的前端和 hasura 之间的连接工作正常，除了 websockets。
编辑 3
我尝试了以下两个入口，但没有成功:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: api
    meta.helm.sh/release-namespace: dev
    nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_http_version 1.1;
      proxy_set_header Upgrade "websocket";
      proxy_set_header Connection "Upgrade";
  labels:
    app.kubernetes.io/instance: api
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: api
    app.kubernetes.io/version: '0.0'
    helm.sh/chart: api-0.0.0
  name: api
spec:
  rules:
    - host: api.shopozor
      http:
        paths:
          - backend:
              serviceName: api
              servicePort: 8080
            path: /

和

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: api
    meta.helm.sh/release-namespace: dev
    nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
  labels:
    app.kubernetes.io/instance: api
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: api
    app.kubernetes.io/version: '0.0'
    helm.sh/chart: api-0.0.0
  name: api
spec:
  rules:
    - host: api.shopozor
      http:
        paths:
          - backend:
              serviceName: api
              servicePort: 8080
            path: /

在后一种情况下，我只是得到错误

WebSocket connection to 'ws://api.shopozor/v1/graphql/' failed: Error during WebSocket handshake: Unexpected response code: 404

而 graphql 端点功能正常。在前一种情况下，我无法访问 hasura 实例上的任何内容，并且我得到了上面的 websocket 握手问题(所以没有 graphql 也没有 websocket 工作)。
编辑 4
使用我的 api 入口配置(没有任何额外的 nginx 注释，如上面: nginx.ingress.kubernetes.io/proxy-read-timeout ， nginx.ingress.kubernetes.io/proxy-send-timeout ， nginx.ingress.kubernetes.io/configuration-snippet )，如果我这样做:

curl -i -N -H "Connection: Upgrade" \
 -H "Upgrade: websocket" \
 -H "Origin: http://localhost:3000" \
 -H "Host: api.shopozor" \ 
 -H "Sec-Websocket-Version: 13" \
 -H "Sec-WebSocket-Key: B8KgbaRLCMNCREjE5Kvg1w==" \ 
 -H "Sec-WebSocket-Protocol: graphql-ws" \
 -H "Accept-Encoding: gzip, deflate" \
 -H "Accept-Language: en-US,en;q=0.9" \
 -H "Cache-Control: no-cache" \
 -H "Pragma: no-cache" \
 -H "Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits" \
 -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.49 Safari/537.36" \
 http://api.shopozor/v1/graphql

然后哈苏拉很高兴:

2020-07-28T07:42:28.903877263Z {"type":"websocket-log","timestamp":"2020-07-28T07:42:28.894+0000","level":"info","detail":{"event":{"type":"accepted"},"connection_info":{"websocket_id":"94243bde-41c4-42c8-8d8f-355c47a3492e","token_expiry":null,"msg":null},"user_vars":null}}

我上面 curl 中的 header 与我的前端应用程序发送的 header 完全相同。关于我做错了什么的任何线索？前端调用和此 curl 之间的区别在于，在前端我将 websocket 端点定义为 ws://api.shopozor/v1/graphql当我 curl http://api.shopozor/v1/graphql .我不能在 apollo vue 中设置 wsEndpoint至 http://api.shopozor/v1/graphql .我得到一个错误。

Answer 1

听起来您正在使用 nginx 入口 Controller 和 WebSockets are supported out of the box .
你可以试试这个注解💡:

nginx.ingress.kubernetes.io/configuration-snippet: |
   proxy_http_version 1.1;
   proxy_set_header Upgrade "websocket";
   proxy_set_header Connection "Upgrade";

或/和其他注释，因为它们 docs 📄 建议将其用于 WebSockets:

nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
nginx.ingress.kubernetes.io/proxy-send-timeout: 3600

注意:我回答了 similar question前段时间⌛⌚。