azure - AKS 中的证书管理器 ClusterIssuer 存在问题

Question

我在 clusterissuer(cert-manager 版本 1.7.1)中收到此错误:

“获取 CA 颁发者 key 对时出错:解码证书 PEM block 时出错”

我将 ca.crt、tls.crt 和 tls.key 存储在 Azure 的 Key Vault 中。

kubectl 描述 clusterissuer ca-issuer

  Ca:
    Secret Name:  cert-manager-secret
Status:
  Conditions:
    Last Transition Time:  2022-02-25T11:40:49Z
    Message:               Error getting keypair for CA issuer: error decoding certificate PEM block
    Observed Generation:   1
    Reason:                ErrGetKeyPair
    Status:                False
    Type:                  Ready
Events:
  Type     Reason         Age                  From          Message
  ----     ------         ----                 ----          -------
  Warning  ErrGetKeyPair  3m1s (x17 over 58m)  cert-manager  Error getting keypair for CA issuer: error decoding certificate PEM block
  Warning  ErrInitIssuer  3m1s (x17 over 58m)  cert-manager  Error initializing issuer: error decoding certificate PEM block

kubectl 获取 clusterissuer

NAME        READY   AGE
ca-issuer   False   69m 

• 这是 clusterissuer yaml 文件:

ca-issuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer
  namespace: cert-manager
spec:
  ca:
    secretName: cert-manager-secret

这是用于检索 ca.crt、tls.crt 和 tls.key 的 KeyVault yaml 文件

keyvauls.yaml

apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: secret-akscacrt
  namespace: cert-manager
spec:
  vault:
    name: kv-xx # name of key vault
    object:
      name: akscacrt # name of the akv object
      type: secret # akv object type
  output: 
    secret: 
      name: cert-manager-secret # kubernetes secret name
      dataKey: ca.crt # key to store object value in kubernetes secret
---
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: secret-akstlscrt
  namespace: cert-manager
spec:
  vault:
    name: kv-xx # name of key vault
    object:
      name: akstlscrt # name of the akv object
      type: secret # akv object type
  output: 
    secret: 
      name: cert-manager-secret # kubernetes secret name
      dataKey: tls.crt # key to store object value in kubernetes secret
---
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: secret-akstlskey
  namespace: cert-manager
spec:
  vault:
    name: kv-xx # name of key vault
    object:
      name: akstlskey # name of the akv object
      type: secret # akv object type
  output: 
    secret: 
      name: cert-manager-secret # kubernetes secret name
      dataKey: tls.key # key to store object value in kubernetes secret
---

这些是使用的证书:

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: argocd-xx
  namespace: argocd
spec:
  secretName: argocd-xx
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
  commonName: "argocd.xx"
  dnsNames:
    - "argocd.xx"
  privateKey:
    size: 4096
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: sonarqube-xx
  namespace: sonarqube
spec:
  secretName: "sonarqube-xx"
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
  commonName: "sonarqube.xx"
  dnsNames:
    - "sonarqube.xx"
  privateKey:
    size: 4096

我可以看到我可以从 key 保管库检索证书的 secret :

kubectl 获取 secret -n cert-manager cert-manager-secret -o yaml

apiVersion: v1
data:
  ca.crt: XXX
  tls.crt: XXX
  tls.key: XXX

另外，另一个奇怪的事情是，我在 sonarqube/argocd 命名空间中获取了其他 secret ，这些 secret 是我之前部署的，但不再出现在我的部署文件中。我无法删除它们，当我尝试删除它们时，它们会自动重新创建。看起来它们存储在某种缓存中。我还尝试删除 namespace akv2k8s/cert-manager 并删除 cert-manager/akv2k8s Controller 并重新安装它们，但在重新安装和应用部署后出现同样的问题...

kubectl get secret -n sonarqube

NAME                                      TYPE                                  DATA   AGE
cert-manager-secret                       Opaque                                3      155m
default-token-c8b86                       kubernetes.io/service-account-token   3      2d1h
sonarqube-xx-7v7dh   Opaque                                1      107m
sql-db-secret                             Opaque                                2      170m

kubectl get secret -n argocd   
NAME                                   TYPE                                  DATA   AGE
argocd-xx-7b5kb   Opaque                                1      107m
cert-manager-secret-argo               Opaque                                3      157m
default-token-pjb4z                    kubernetes.io/service-account-token   3      3d15h

kubectl 描述证书 sonarqube-xxx -n sonarqube

Status:
  Conditions:
    Last Transition Time:        2022-02-25T11:04:08Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2022-02-25T11:04:08Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  sonarqube-xxx-7v7dh
Events:                          <none>

有什么想法吗？

谢谢。

Answer 1

我发现只需上传证书信息ca.crt。 Azure Key Vault secret 中的 tls.crt 和 tls.key 纯文本形式，没有 BASE64 编码。

当 AKV2K8S 从 Key Vault 检索 secret 并将其存储在 Kubernetes 中时，它会自动以 BASE64 进行编码。

问候，