nginx - K8s Helm - 带有 Nginx Ingress 的 Jenkins

Question

我正在尝试通过 Helm 部署以 nginx-ingress 为前端的 Jenkins。目标是在 nginx 上使用 SSL 终止保护 HTTPs 背后的 Jenkins。我目前使用的是自签名证书，但最终会使用证书管理器和 LetsEncrypt。 Jenkins 和 Nginx-Ingress 部署在默认命名空间中。

下面是我的部署脚本:

gcloud config set compute/zone us-central1-f

gcloud container clusters create jenkins-cd \
  --machine-type n1-standard-2 --num-nodes 2 \
  --scopes "https://www.googleapis.com/auth/projecthosting,storage-rw,cloud-platform"

wget https://storage.googleapis.com/kubernetes-helm/helm-v2.9.1-linux-amd64.tar.gz
tar zxfv helm-v2.9.1-linux-amd64.tar.gz
cp linux-amd64/helm .

kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=xxxx@xxxx.com
kubectl create clusterrolebinding tiller-admin-binding --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=xx.xx.xxxx.com"
kubectl create secret tls jenkins-ingress-ssl --key /tmp/tls.key --cert /tmp/tls.crt
kubectl describe secret jenkins-ingress-ssl

./helm init --service-account=tiller --wait
./helm update
./helm install --name nginx-ingress stable/nginx-ingress --set rbac.create=true
./helm install --name jenkins stable/jenkins --values values.yaml --version 0.19.0 --wait

ADMIN_PWD=$(kubectl get secret --namespace default cd-jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode)

下面是我的 values.yaml 文件:

Master:
  InstallPlugins:
    - kubernetes:1.12.6
    - workflow-job:2.24
    - workflow-aggregator:2.5
    - credentials-binding:1.16
    - git:3.9.1
    - google-oauth-plugin:0.6
    - google-source-plugin:0.3
  Cpu: "1"
  Memory: "3500Mi"
  JavaOpts: "-Xms3500m -Xmx3500m"
  ServiceType: ClusterIP
  HostName: "xx.xx.xxxx.com"
  Ingress:
    Annotations:
      kubernetes.io/ingress.class: "nginx"
      kubernetes.io/ingress.allow-http: "false"    
    TLS:
      - secretName: jenkins-ingress-ssl
        hosts:
          - xx.xx.xxxx.com
Agent:
  Enabled: true
Persistence:
  Size: 100Gi
NetworkPolicy:
  ApiVersion: networking.k8s.io/v1
rbac:
  install: true
  serviceAccountName: cd-jenkins

部署(默认命名空间)

xxx@cloudshell:~/stub-jenkins2.0 (automation-stub)$ kubectl get deployments
NAME                            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
jenkins                         1         1         1            1           6m
nginx-ingress-controller        1         1         1            1           6m
nginx-ingress-default-backend   1         1         1            1           6m

服务(默认命名空间)

xxx@cloudshell:~/stub-jenkins2.0 (automation-stub)$ kubectl get services
NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                      AGE
jenkins                         ClusterIP      10.11.240.123   <none>            8080/TCP                     7m
jenkins-agent                   ClusterIP      10.11.250.174   <none>            50000/TCP                    7m
kubernetes                      ClusterIP      10.11.240.1     <none>            443/TCP                      8m
nginx-ingress-controller        LoadBalancer   10.11.253.104   104.198.179.176   80:31453/TCP,443:32194/TCP   7m
nginx-ingress-default-backend   ClusterIP      10.11.245.149   <none>            80/TCP                       7m

入口(默认命名空间)

xxx@cloudshell:~/stub-jenkins2.0 (automation-stub)$ kubectl get ingress
NAME      HOSTS                      ADDRESS         PORTS     AGE
jenkins   xx.xx.xxxx.com             35.193.17.244   80, 443   7m

Ingress .yaml(由 helm 生成)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.class: nginx
  creationTimestamp: 2018-10-19T17:35:16Z
  generation: 1
  name: jenkins
  namespace: default
  resourceVersion: "845"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/jenkins
  uid: 57b76091-d3c5-11e8-b9e9-42010a8001de
spec:
  rules:
  - host: xx.xx.xxxx.com
    http:
      paths:
      - backend:
          serviceName: jenkins
          servicePort: 8080
  tls:
  - hosts:
    - xx.xx.xxxx.com
    secretName: jenkins-ingress-ssl
status:
  loadBalancer:
    ingress:
    - ip: 35.193.17.244

当访问入口 IP ( https://104.198.179.176 ) 时，我访问了默认的 nginx 后端服务并出现“默认后端 - 404”错误。我怀疑入口配置可能有问题。根据入口设置，有不正确的入口 IP (35.193.17.244) 而 nginx-ingress-controller IP 是 104.198.179.176。

Answer 1

如果您使用 https://104.198.179.176 访问 nginx ingress，您将始终访问默认后端。您需要使用 https://xx.xx.xxxx.com 或类似这样的内容来点击它:

$ curl -H 'Host: xx.xx.xxxx.com' https://104.198.179.176

关于入口 IP 地址不正确，我会检查您的后端服务是否有端点，并且每个端点都在端口 8080 上监听。

$ kubectl describe svc jenkins

或/和

$ kubectl describe ep

我还会检查 Ingress 中的事件:

$ kubectl describe ingress jenkins

最后，我会检查入口 Controller 中的日志:

$ kubectl logs nginx-ingress-controller