存储假 SSL 证书 : could not create PEM certificate file 的 Kubernetes nginx 入口意外错误

Question

我已经在 CentOS 8 Kubernetes 1.17 上使用 containerd 安装了 nignx ingress helm chat，ingress pod 失败并显示以下错误消息。相同的 helm chat 在 CentOS 7 和 Docker 上工作。

I0116 04:17:06.624547       8 flags.go:205] Watching for Ingress class: nginx
W0116 04:17:06.624803       8 flags.go:250] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0116 04:17:06.624844       8 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       0.27.1
  Build:         git-1257ded99
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.17.7

-------------------------------------------------------------------------------

I0116 04:17:06.624968       8 main.go:194] Creating API client for https://10.224.0.1:443
I0116 04:17:06.630907       8 main.go:238] Running in Kubernetes cluster version v1.17 (v1.17.0) - git (clean) commit 70132b0f130acc0bed193d9ba59dd186f0e634cf - platform linux/amd64
I0116 04:17:06.633567       8 main.go:91] Validated nginx-ingress/nginx-ingress-default-backend as the default backend.
F0116 04:17:06.843785       8 ssl.go:389] unexpected error storing fake SSL Cert: could not create PEM certificate file /etc/ingress-controller/ssl/default-fake-certificate.pem: open /etc/ingress-controller/ssl/default-fake-certificate.pem: permission denied

如果我从部署中删除它，入口 pod 正在启动。

 capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL

我想了解为什么同一个 helm chart 在 containerd 上失败

containerd --version
containerd github.com/containerd/containerd 1.2.0

添加部署。

containers:     
      - args:
        - /nginx-ingress-controller
        - --default-backend-service=nginx-ingress/nginx-ingress-default-backend
        - --election-id=ingress-controller-leader
        - --ingress-class=nginx
        - --configmap=nginx-ingress/nginx-ingress-controller
        - --default-ssl-certificate=nginx-ingress/ingress-tls
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.27.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: nginx-ingress-controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - NET_BIND_SERVICE
             drop:
            - ALL
          runAsUser: 101
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: nginx-ingress

错误信息

-------------------------------------------------------------------------------
W0116 16:02:30.074390       8 queue.go:130] requeuing nginx-ingress/nginx-ingress-controller, err
-------------------------------------------------------------------------------
Error: exit status 1
nginx: the configuration file /tmp/nginx-cfg613392629 syntax is ok
2020/01/16 16:02:30 [emerg] 103#103: bind() to 0.0.0.0:80 failed (13: Permission denied)
nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)
nginx: configuration file /tmp/nginx-cfg613392629 test failed

Answer 1

我也有同样的经历。解决方案不是删除功能部分，而是更改 runAsuser

如果你下载 Nginx 入口 Controller 的新版本(0.27.1)部署，你可以看到:

       securityContext:
         allowPrivilegeEscalation: true
         capabilities:
           drop:
             - ALL
           add:
             - NET_BIND_SERVICE
         # www-data -> 101
         runAsUser: 101

“runAsUser”行具有不同的用户 ID。我的旧部署中的用户 ID 不同，所以我收到了这个错误。由于我将 runAsUser 更改为 ID 101，kubernetes 定义中的 id 与新 Nginx 镜像中使用的 ID 相同，并且它再次工作:)