azure - 无法将 Azure Key Vault 与 Azure Kubernetes 服务集成

Question

坚持让这个集成工作。我正在关注the documentation一步一步。

以下是我从头开始所做的一切，所以如果这里没有列出，我还没有尝试过(我为冗长的一系列命令提前道歉):

# create the resource group
az group create -l westus -n k8s-test

# create the azure container registery
az acr create -g k8s-test -n k8stestacr --sku Basic -l westus

# create the azure key vault and add a test value to it
az keyvault create --name k8stestakv --resource-group k8s-test -l westus
az keyvault secret set --vault-name k8stestakv --name SECRETTEST --value abc123

# create the azure kubernetes service
az aks create -n k8stestaks -g k8s-test --kubernetes-version=1.19.7 --node-count 1 -l westus --enable-managed-identity --attach-acr k8stestacr -s Standard_B2s

# switch to the aks context
az aks get-credentials -b k8stestaks -g k8s-test

# install helm charts for secrets store csi
helm repo add csi-secrets-store-provider-azure https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/charts
helm install csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --generate-name

# create role managed identity operator
az role assignment create --role "Managed Identity Operator" --assignee <k8stestaks_clientId> --scope /subscriptions/<subscriptionId>/resourcegroups/MC_k8s-test_k8stestaks_westus

# create role virtual machine contributor
az role assignment create --role "Virtual Machine Contributor" --assignee <k8stestaks_clientId> --scope /subscriptions/<subscriptionId>/resourcegroups/MC_k8s-test_k8stestaks_westus

# install more helm charts
helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
helm install pod-identity aad-pod-identity/aad-pod-identity

# create identity
az identity create -g MC_k8s-test_k8stestaks_westus -n TestIdentity  

# give the new identity a reader role for AKV
az role assignment create --role "Reader" --assignee <TestIdentity_principalId> --scope /subscriptions/<subscription_id/resourceGroups/k8s-test/providers/Microsoft.KeyVault/vaults/k8stestakv

# allow the identity to get secrets from AKV
az keyvault set-policy -n k8stestakv --secret-permissions get --spn <TestIdentity_clientId>

对于 az cli 命令来说就差不多了。到目前为止，一切都执行得很好，没有错误。我可以进入门户，查看 MC_ 组的这些角色、只读 secret 的 TestIdentity 等。

之后，文档让您构建 secretProviderClass.yaml:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kvname
spec:
  provider: azure
  parameters:
    usePodIdentity: "true"                   
    useVMManagedIdentity: "false"             
    userAssignedIdentityID: ""                       
    keyvaultName: "k8stestakv"                
    cloudName: ""                               
    objects:  |
      array:
        - |
          objectName: SECRETTEST             
          objectType: secret                 
          objectVersion: ""                 
    resourceGroup: "k8s-test"     
    subscriptionId: "<subscriptionId>"         
    tenantId: "<tenantId>"       

还有podIdentityBinding.yaml:

apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
    name: azureIdentity               
spec:
    type: 0                                 
    resourceID: /subscriptions/<subscriptionId>/resourcegroups/MC_k8s-test_k8stestaks_westus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/TestIdentity
    clientID: <TestIdentity_clientId>    
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
    name: azure-pod-identity-binding
spec:
    azureIdentity: azureIdentity     
    selector: azure-pod-identity-binding-selector

然后应用它们:

# this one executes fine
kubectl apply -f k8s/secret/secretProviderClass.yaml

# this one does not
kubectl apply -f k8s/identity/podIdentityBinding.yaml

问题#1

我得到的最后一个:

unable to recognize "k8s/identity/podIdentityBinding.yaml": no matches for kind "AzureIdentity" in version "aadpodidentity.k8s.io/v1"
unable to recognize "k8s/identity/podIdentityBinding.yaml": no matches for kind "AzureIdentityBinding" in version "aadpodidentity.k8s.io/v1"

不知道为什么，因为 helm install pod-identity aad-pod-identity/aad-pod-identity 命令成功了。然而看看我的 Pod...

问题#2

我已按照这些步骤操作三次，每次问题都是相同的 - aad-pod-identity-nmi-xxxxx 不会启动:

$ kubectl get pods
NAME                                                              READY   STATUS             RESTARTS   AGE
aad-pod-identity-mic-7b4558845f-hwv8t                             1/1     Running            0          37m
aad-pod-identity-mic-7b4558845f-w8mxt                             1/1     Running            0          37m
aad-pod-identity-nmi-4sf5q                                        0/1     CrashLoopBackOff   12         37m
csi-secrets-store-provider-azure-1613256848-cjlwc                 1/1     Running            0          41m
csi-secrets-store-provider-azure-1613256848-secrets-store-m4wth   3/3     Running            0          41m

$ kubectl describe pod aad-pod-identity-nmi-4sf5q
Name:         aad-pod-identity-nmi-4sf5q
Namespace:    default
Priority:     0
Node:         aks-nodepool1-40626841-vmss000000/10.240.0.4
Start Time:   Sat, 13 Feb 2021 14:57:54 -0800
Labels:       app.kubernetes.io/component=nmi
              app.kubernetes.io/instance=pod-identity
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=aad-pod-identity
              controller-revision-hash=669df55fd8
              helm.sh/chart=aad-pod-identity-3.0.3
              pod-template-generation=1
              tier=node
Annotations:  <none>
Status:       Running
IP:           10.240.0.4
IPs:
  IP:           10.240.0.4
Controlled By:  DaemonSet/aad-pod-identity-nmi
Containers:
  nmi:
    Container ID:  containerd://5f9e17e95ae395971dfd060c1db7657d61e03052ffc3cbb59d01c774bb4a2f6a
    Image:         mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.4
    Image ID:      mcr.microsoft.com/oss/azure/aad-pod-identity/nmi@sha256:0b4e296a7b96a288960c39dbda1a3ffa324ef33c77bb5bd81a4266b85efb3498
    Port:          <none>
    Host Port:     <none>
    Args:
      --node=$(NODE_NAME)
      --http-probe-port=8085
      --operation-mode=standard
      --kubelet-config=/etc/default/kubelet
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Sat, 13 Feb 2021 15:34:40 -0800
      Finished:     Sat, 13 Feb 2021 15:34:40 -0800
    Ready:          False
    Restart Count:  12
    Limits:
      cpu:     200m
      memory:  512Mi
    Requests:
      cpu:     100m
      memory:  256Mi
    Liveness:  http-get http://:8085/healthz delay=10s timeout=1s period=5s #success=1 #failure=3
    Environment:
      NODE_NAME:         (v1:spec.nodeName)
      FORCENAMESPACED:  false
    Mounts:
      /etc/default/kubelet from kubelet-config (ro)
      /run/xtables.lock from iptableslock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from aad-pod-identity-nmi-token-8sfh4 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  iptableslock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  kubelet-config:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/default/kubelet
    HostPathType:  
  aad-pod-identity-nmi-token-8sfh4:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  aad-pod-identity-nmi-token-8sfh4
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  38m                    default-scheduler  Successfully assigned default/aad-pod-identity-nmi-4sf5q to aks-nodepool1-40626841-vmss000000
  Normal   Pulled     38m                    kubelet            Successfully pulled image "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.4" in 14.677657725s
  Normal   Pulled     38m                    kubelet            Successfully pulled image "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.4" in 5.976721016s
  Normal   Pulled     37m                    kubelet            Successfully pulled image "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.4" in 627.112255ms
  Normal   Pulling    37m (x4 over 38m)      kubelet            Pulling image "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.4"
  Normal   Pulled     37m                    kubelet            Successfully pulled image "mcr.microsoft.com/oss/azure/aad-pod-identity/nmi:v1.7.4" in 794.669637ms
  Normal   Created    37m (x4 over 38m)      kubelet            Created container nmi
  Normal   Started    37m (x4 over 38m)      kubelet            Started container nmi
  Warning  BackOff    3m33s (x170 over 38m)  kubelet            Back-off restarting failed container

我想我不确定这两个问题是否相关，而且我无法启动失败的 Pod。

这里有什么建议吗？

Answer 1

如果您没有为网络选项指定“高级”，它看起来与 AKS 为您选择的默认网络插件有关:kubenet。

可以使用此处概述的 kubenet 来完成此集成:

https://azure.github.io/aad-pod-identity/docs/configure/aad_pod_identity_on_kubenet/

如果您要创建新集群，请启用高级网络或添加 --network-plugin azure 标志和参数。