kubernetes - 我正在尝试将 externalname 与 Nginx-Controller 的 nodeport 服务一起使用，但我得到 502 Bad Gateway

Question

环境:

我有:

1- NGINX 入口 Controller 版本: 1.15.9，图片:0.23.0

2- Kubernetes 版本:

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

云提供商或硬件配置: KVM 上的虚拟机

操作系统(例如来自/etc/os-release):

NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

内核(例如 uname -a):

Linux node01 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

安装工具: kubeadm

更多详情:

CNI : 编织

设置:

2 个弹性 HA 代理、3 个主节点、2 个基础设施和工作节点。

我将所有服务公开为节点端口，HA-Proxy 将它们重新分配给公共(public)虚拟 IP。

承载监控和日志工具(Grafana、Prometheus、EFK等)的infra节点上托管的专用项目

后端 NFS 存储作为持久存储

发生了什么:
我希望能够使用外部名称而不是节点端口，因此我不想通过 vip + 3000 访问 grafana，而是想通过 http://grafana.wild-card-dns-zone 访问它

部署

我创建了一个名为 ingress

的新命名空间

我将其部署如下:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: **2**
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
      name: nginx-ingress
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        node-role.kubernetes.io/infra: infra
      terminationGracePeriodSeconds: 60
      containers:
      - name: nginx-ingress-controller
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 10
        args:
          - /nginx-ingress-controller
          - --default-backend-service=ingress/ingress-controller-nginx-ingress-default-backend
          - --configmap=$(POD_NAMESPACE)/nginx-configuration
          - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
          - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
          - --publish-service=$(POD_NAMESPACE)/ingress-nginx
          - --annotations-prefix=nginx.ingress.kubernetes.io
          - --v3
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            drop:
              - ALL
            add:
              - NET_BIND_SERVICE
          # www-data -> 33
          runAsUser: 33
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  generation: 1
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.3.1
    component: default-backend
  name: ingress-controller-nginx-ingress-default-backend
  namespace: ingress
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx-ingress
      component: default-backend
      release: ingress-controller
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx-ingress
        component: default-backend
        release: ingress-controller
    spec:
      nodeSelector:
        node-role.kubernetes.io/infra: infra
      containers:
      - image: k8s.gcr.io/defaultbackend:1.4
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: nginx-ingress-default-backend
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 60
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    name: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.3.1
    component: default-backend
  name: ingress-controller-nginx-ingress-default-backend
  namespace: ingress
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: nginx-ingress
    component: default-backend
    release: ingress-controller
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrolebinding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-rolebinding
  namespace: ingress
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress

入口设置:

服务

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2019-03-25T16:03:01Z"
  labels:
    app: jaeger
    app.kubernetes.io/component: query
    app.kubernetes.io/instance: jeager
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jeager-query
    app.kubernetes.io/part-of: jaeger
  name: jeager-query
  namespace: monitoring-logging
  resourceVersion: "3055947"
  selfLink: /api/v1/namespaces/monitoring-logging/services/jeager-query
  uid: 778550f0-4f17-11e9-9078-001a4a16021e
spec:
  externalName: jaeger.example.com
  ports:
  - port: 16686
    protocol: TCP
    targetPort: 16686
  selector:
    app: jaeger
    app.kubernetes.io/component: query
    app.kubernetes.io/instance: jeager
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jeager-query
    app.kubernetes.io/part-of: jaeger
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2019-03-25T15:40:30Z"
  labels:
    app: grafana
    chart: grafana-2.2.4
    heritage: Tiller
    release: grafana
  name: grafana
  namespace: monitoring-logging
  resourceVersion: "3053698"
  selfLink: /api/v1/namespaces/monitoring-logging/services/grafana
  uid: 51b9d878-4f14-11e9-9078-001a4a16021e
spec:
  externalName: grafana.example.com
  ports:
  - name: http
    port: 3000
    protocol: TCP
    targetPort: 3000
  selector:
    app: grafana
    release: grafana
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}

入口

入口 1

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/service-upstream: "true"
  creationTimestamp: "2019-03-25T21:13:56Z"
  generation: 1
  labels:
    app: jaeger
    app.kubernetes.io/component: query-ingress
    app.kubernetes.io/instance: jeager
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jeager-query
    app.kubernetes.io/part-of: jaeger
  name: jaeger-query
  namespace: monitoring-logging
  resourceVersion: "3111683"
  selfLink: /apis/extensions/v1beta1/namespaces/monitoring-logging/ingresses/jaeger-query
  uid: e6347f6b-4f42-11e9-9e8e-001a4a16021c
spec:
  rules:
  - host: jaeger.example.com
    http:
      paths:
      - backend:
          serviceName: jeager-query
          servicePort: 16686
status:
  loadBalancer: {}

入口 2

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"labels":{"app":"grafana"},"name":"grafana","namespace":"monitoring-logging"},"spec":{"rules":[{"host":"grafana.example.com","http":{"paths":[{"backend":{"serviceName":"grafana","servicePort":3000}}]}}]}}
  creationTimestamp: "2019-03-25T17:52:40Z"
  generation: 1
  labels:
    app: grafana
  name: grafana
  namespace: monitoring-logging
  resourceVersion: "3071719"
  selfLink: /apis/extensions/v1beta1/namespaces/monitoring-logging/ingresses/grafana
  uid: c89d7f34-4f26-11e9-8c10-001a4a16021d
spec:
  rules:
  - host: grafana.example.com
    http:
      paths:
      - backend:
          serviceName: grafana
          servicePort: 3000
status:
  loadBalancer: {}

端点

端点 1

apiVersion: v1
kind: Endpoints
metadata:
  creationTimestamp: "2019-03-25T15:40:30Z"
  labels:
    app: grafana
    chart: grafana-2.2.4
    heritage: Tiller
    release: grafana
  name: grafana
  namespace: monitoring-logging
  resourceVersion: "3050562"
  selfLink: /api/v1/namespaces/monitoring-logging/endpoints/grafana
  uid: 51bb1f9c-4f14-11e9-9e8e-001a4a16021c
subsets:
- addresses:
  - ip: 10.42.0.15
    nodeName: kuinfra01.example.com
    targetRef:
      kind: Pod
      name: grafana-b44b4f867-bcq2x
      namespace: monitoring-logging
      resourceVersion: "1386975"
      uid: 433e3d21-4827-11e9-9e8e-001a4a16021c
  ports:
  - name: http
    port: 3000
    protocol: TCP

端点 2

apiVersion: v1
kind: Endpoints
metadata:
  creationTimestamp: "2019-03-25T16:03:01Z"
  labels:
    app: jaeger
    app.kubernetes.io/component: service-query
    app.kubernetes.io/instance: jeager
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: jeager-query
    app.kubernetes.io/part-of: jaeger
  name: jeager-query
  namespace: monitoring-logging
  resourceVersion: "3114702"
  selfLink: /api/v1/namespaces/monitoring-logging/endpoints/jeager-query
  uid: 7786d833-4f17-11e9-9e8e-001a4a16021c
subsets:
- addresses:
  - ip: 10.35.0.3
    nodeName: kunode02.example.com
    targetRef:
      kind: Pod
      name: jeager-query-7d9775d8f7-2hwdn
      namespace: monitoring-logging
      resourceVersion: "3114693"
      uid: fdac9771-4f49-11e9-9e8e-001a4a16021c
  ports:
  - name: query
    port: 16686
    protocol: TCP

我能够从 ingress-controller pod 内部 curl 端点:

# kubectl exec -it nginx-ingress-controller-5dd67f88cc-z2g8s  -n ingress -- /bin/bash
www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ curl -k https://localhost
<a href="/login">Found</a>.

www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ curl http://localhost
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.15.9</center>
</body>
</html>
www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ exit

但是，当我尝试访问 jaeger.example.com 或 grafana.example.com 时，我收到 502 bad gatway 和以下错误日志:

10.39.0.0 - [10.39.0.0] - - [25/Mar/2019:16:40:32 +0000] "GET /search HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" 514 0.001 [monitoring-logging-jeager-query-16686] vip:16686, vip:16686, vip:16686 0, 0, 0 0.001, 0.000, 0.000 502, 502, 502 b7c813286fccf27fffa03eb6564edfd1
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
10.39.0.0 - [10.39.0.0] - - [25/Mar/2019:16:40:32 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://jeager.example.com/search" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" 494 0.001 [monitoring-logging-jeager-query-16686] vip:16686, vip:16686, vip:16686 0, 0, 0 0.000, 0.001, 0.000 502, 502, 502 9e582912614e67dfee6be1f679de5933
I0325 16:40:32.497868       8 socket.go:225] skiping metric for host jeager.example.com that is not being served
I0325 16:40:32.497886       8 socket.go:225] skiping metric for host jeager.example.com that is not being served

Answer 1

首先感谢 cookiedough 提供有关服务问题的线索，但后来我遇到了使用外部名称创建服务的问题，但我发现我的错误感谢“Long”用户在 slack 中，错误是我正在使用服务输入 ExternalName 并且应该输入集群 IP 这里是解决问题的步骤(备注 https 问题是一个单独的问题):
1-创建指向公共(public) IP 的通配符 DNS 区域
1- 对于新服务，只需创建 ClusterIP 类型
2- 在服务的命名空间中使用以下示例 (yaml) 创建一个入口:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
labels:
app: grafana
name: grafana
namespace: grafana-namespace
spec:
rules:

host: grafana.example.com
http:
paths:
backend:
serviceName: grafana
servicePort: 3000

3- kubectl -f 应用 -f grafana-ingress.yaml
现在您可以通过 http://grafana.example,com 联系您的 grafana