azure - AKS 的 key 保管库问题

Question

几周前，我发布了有关使用 Key Vault(使用用户分配的托管身份方法)的 Kubernetes 部署的类似问题。问题已解决，但当尝试从头开始实现所有内容时，有些事情对我来说毫无意义。

基本上我收到了有关安装卷的错误:

Volumes:
  sonar-data-new:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sonar-data-new
    ReadOnly:   false
  sonar-extensions-new2:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sonar-extensions-new2
    ReadOnly:   false
  secrets-store-inline:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=azure-kv-provider
  default-token-zwxzg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-zwxzg
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    12s               default-scheduler  Successfully assigned default/sonarqube-d44d498f8-46mpz to aks-agentpool-35716862-vmss000000
  Warning  FailedMount  3s (x5 over 11s)  kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/sonarqube-d44d498f8-46mpz, err: rpc error: code = Unknown desc = failed to mountobjects, error: failed to get objectType:secret, objectName:username, objectVersion:: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://SonarQubeHelm.vault.azure.net/secrets/username/?api-version=2016-10-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"}

这是我的 secret-class.yml 文件(keyvault 的名称是正确的)。另外，xxx-xxxx-xxx-xxx-xxxxx4b5ec83 是 AKS 托管身份 (SonarQubeHelm-agentpool) 的 objectID

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kv-provider
spec:
  provider: azure
  secretObjects:
   - data:
      - key: username
        objectName: username
      - key: password
        objectName: password
     secretName: test-secret
     type: Opaque
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "xxx-xxxx-xxx-xxx-xxxxx4b5ec83"
    keyvaultName: "SonarQubeHelm"
    cloudName: ""
    objects:  |
      array:
        - |
          objectName: username
          objectType: secret
          objectAlias: username
          objectVersion: ""
        - |
          objectName: password
          objectType: secret
          objectAlias: password
          objectVersion: ""
    resourceGroup: "rg-LD-sandbox"
    subscriptionId: "xxxx"
    tenantId: "yyyy"

这是我的deployment.yml 文件

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: sonarqube
  name: sonarqube
spec:
  selector:
    matchLabels:
      app: sonarqube
  replicas: 1
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      containers:
        - name: sonarqube
          image: sonarqube:8.9-developer
          resources:
            requests:
              cpu: 500m
              memory: 1024Mi
            limits:
              cpu: 2000m
              memory: 4096Mi
          volumeMounts:
          - mountPath: "/mnt/"
            name: secrets-store-inline
          - mountPath: "/opt/sonarqube/data/"
            name: sonar-data-new
          - mountPath: "/opt/sonarqube/extensions/plugins/"
            name: sonar-extensions-new2
          env:
          - name: "SONARQUBE_JDBC_USERNAME"
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: username
          - name: "SONARQUBE_JDBC_PASSWORD"
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: password
          - name: "SONARQUBE_JDBC_URL"
            valueFrom:
              configMapKeyRef:
                name: sonar-config
                key: url
          ports:
          - containerPort: 9000
            protocol: TCP
      volumes:
      - name: sonar-data-new
        persistentVolumeClaim:
          claimName: sonar-data-new
      - name: sonar-extensions-new2
        persistentVolumeClaim:
          claimName: sonar-extensions-new2
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
           secretProviderClass: "azure-kv-provider"

我为 AKS 托管身份分配了适当的权限，以访问 keyvault(xxx-xxxx-xxx-xxx-xxxxx4b5ec83 是 AKS 托管身份的 objectID - SonarQubeHelm-agentpool)

xxxx@Azure:~/clouddrive/kubernetes/sonarqubekeyvault$ az role assignment list --assignee xxx-xxxx-xxx-xxx-xxxxx4b5ec83 --all
[
  {
    "canDelegate": null,
    "condition": null,
    "conditionVersion": null,
    "description": null,
    "id": "/subscriptions/xxxx-xxx-xxx-xxx-xxxe22e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm/providers/Microsoft.Authorization/roleAssignments/xxxx-x-xx-xx-xx86584218f",
    "name": "xxxx-xx-x-x-xx86584218f",
    "principalId": "xxx-xxx-x-xxx-xx3a4b5ec83",
    "principalName": "xxxx-xxxx-xxx-xxx-xx79a3906b8",
    "principalType": "ServicePrincipal",
    "resourceGroup": "rg-LD-sandbox",
    "roleDefinitionId": "/subscriptions/xxxx-xxxx-xxxx-xxx-0e1e22e8804e/providers/Microsoft.Authorization/roleDefinitions/xxx-xxxx-xxx-xxxx-xxxfe8e74483",
    "roleDefinitionName": "Key Vault Administrator",
    "scope": "/subscriptions/xxxx-xxx-xxx-xxxx-xxx2e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm",
    "type": "Microsoft.Authorization/roleAssignments"
  },
  {
    "canDelegate": null,
    "condition": null,
    "conditionVersion": null,
    "description": null,
    "id": "/subscriptions/xxxx-xxxx-xxxx-xxxx-xxxe22e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm/providers/Microsoft.Authorization/roleAssignments/xxxx-xxxx-xxxx-xxxx-xxx5137f480",
    "name": "xxxx-xxxx-xxxx-xxxx-xx5137f480",
    "principalId": "xxxx-xxxx-xxxx-xxxx-xx3a4b5ec83",
    "principalName": "xxxx-xxxx-xxxx-xxxx-xx79a3906b8",
    "principalType": "ServicePrincipal",
    "resourceGroup": "rg-LD-sandbox",
    "roleDefinitionId": "/subscriptions/xxxx-xxxx-xxxx-xxxx-0e1e22e8804e/providers/Microsoft.Authorization/roleDefinitions/xxxx-xxxx-xxxx-xxxx-xx2c155cd7",
    "roleDefinitionName": "Key Vault Secrets Officer",
    "scope": "/subscriptions/xxxx/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm",
    "type": "Microsoft.Authorization/roleAssignments"
  }
]

这是有关我的 Key Vault 的信息。

az keyvault show --name SonarQubeHelm

{
  "id": "/subscriptions/xxxx-xxxxx-xxxx-xxxx-xxxxxxxx804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm",
  "location": "xxxxx",
  "name": "SonarQubeHelm",
  "properties": {
    "accessPolicies": [
      {
        "applicationId": null,
        "objectId": "xxxx-xxx-xxxx-xxxx-xxxxa4b5ec83",
        "permissions": {
          "certificates": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "ManageContacts",
            "ManageIssuers",
            "GetIssuers",
            "ListIssuers",
            "SetIssuers",
            "DeleteIssuers",
            "Purge"
          ],
          "keys": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "Decrypt",
            "Encrypt",
            "UnwrapKey",
            "WrapKey",
            "Verify",
            "Sign",
            "Purge"
          ],
          "secrets": [
            "Get",
            "List",
            "Set",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "Purge"
          ],
          "storage": null
        },
        "tenantId": "xxxx-xxxx-xxxx-xxxx-xxxxxdb8c610"
      },
      {
        "applicationId": null,
        "objectId": "xxxx-xxxx-xxxx-xxxx-xxxx531f67f8",
        "permissions": {
          "certificates": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "ManageContacts",
            "ManageIssuers",
            "GetIssuers",
            "ListIssuers",
            "SetIssuers",
            "DeleteIssuers",
            "Purge"
          ],
          "keys": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "Decrypt",
            "Encrypt",
            "UnwrapKey",
            "WrapKey",
            "Verify",
            "Sign",
            "Purge"
          ],
          "secrets": [
            "Get",
            "List",
            "Set",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "Purge"
          ],
          "storage": null
        },
        "tenantId": "xxxxx-xxxxxx-xxx-xxx8db8c610"
      },
      {
        "applicationId": null,
        "objectId": "xxx-xxxx-xxxx-xxx-xxxx0df6af9",
        "permissions": {
          "certificates": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "ManageContacts",
            "ManageIssuers",
            "GetIssuers",
            "ListIssuers",
            "SetIssuers",
            "DeleteIssuers"
          ],
          "keys": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore"
          ],
          "secrets": [
            "Get",
            "List",
            "Set",
            "Delete",
            "Recover",
            "Backup",
            "Restore"
          ],
          "storage": null
        },
        "tenantId": "xxx-xxx-xxx-xxx-xxx8db8c610"
      }
    ],
    "createMode": null,
    "enablePurgeProtection": null,
    "enableRbacAuthorization": false,
    "enableSoftDelete": true,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": false,
    "enabledForTemplateDeployment": false,
    "hsmPoolResourceId": null,
    "networkAcls": null,
    "privateEndpointConnections": null,
    "provisioningState": "Succeeded",
    "sku": {
      "family": "A",
      "name": "Standard"
    },
    "softDeleteRetentionInDays": 90,
    "tenantId": "xxx-xxx-xxx-xxx-xxx68db8c610",
    "vaultUri": "https://sonarqubehelm.vault.azure.net/"
  },
  "resourceGroup": "rg-LD-sandbox",
  "systemData": null,
  "tags": {},
  "type": "Microsoft.KeyVault/vaults"
}

这是当前正在运行的CSI pod:

NAME                                                READY   STATUS              RESTARTS   AGE
csi-secrets-store-provider-azure-1632148185-hggl4   1/1     Running             0          5h4m
ingress-nginx-controller-65c4f84996-99pkh           1/1     Running             0          5h49m
secrets-store-csi-driver-xsx2r                      3/3     Running             0          5h4m
sonarqube-d44d498f8-46mpz                           0/1     ContainerCreating   0          26m

我使用了这个CSI驱动程序

helm install csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --set Secrets-store-csi-driver.syncSecret.enabled=true --generate-name

要为我遵循的 AKS 托管身份分配适当的权限(以防万一我也使用了 clientID，但它不起作用)。但是，从前面的命令来看，托管身份的权限似乎是正确的。

export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

在 keystore “SonarQubeHelm”的访问策略中，这些是添加了所有权限的应用程序:SonarQubeHelm-agentpool 和 >SonarQubeHelm

Secrets(Key Vault 内)中的键值为用户名和密码。

所有内容都位于同一区域、资源组和命名空间(默认)，并且我正在使用 1 节点集群。

对这个错误有什么想法吗？

提前致谢!

Answer 1

经过一些测试，看来我遵循的过程是正确的。最有可能的是，我在 AKS 托管身份的角色分配中使用 principalId 而不是 clientId。

其他面临类似问题的人的要点:

1. 检查 AKS 自动创建的托管标识是什么。检查clientId；例如，

   az vmss identity show -g MC_rg-LX-sandbox_SonarQubeHelm_southcentralus -n 
   aks-agentpool-xxxxx62-vmss -o yaml
   

   Remember that MC_** is the resource group that AKS creates automatically to keep AKS resources. VMSS is the Virtual Machine Scale Set; you can get it under the same resource group.

2. 检查它是否具有访问您创建的 key 保管库的正确权限:例如(其中xxxx-xxxx-xxx-xxx-xx79a3906b8是托管身份clientId):

   az role assignment list --assignee xxxx-xxxx-xxx-xxx-xx79a3906b8 --all
   

   它应该有:

   "roleDefinitionName": "Key Vault Administrator"
   

3. 如果没有正确的权限，请分配它们:

   export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.clientId -o tsv)
   export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
   az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID
   

AKS 中托管标识的角色分配适用于 clientId。