azure - 通过 Terraform 和 Helm 添加并安装 Bitnami 外部 DNS 后，NMI pod 中出现错误 - 找不到 pod 的 AzureIdentityBinding

Question

我正在努力获取外部 DNS 绑定(bind)的 azureIdentity 并将 DNS 条目获取到我们的区域中。

Key error: I0423 19:27:52.830107 1 mic.go:610] No AzureIdentityBinding found for pod default/external-dns-84dcc5f68c-cl5h5 that matches selector: external-dns. it will be ignoredAlso, no azureAssignedIdentity is created since there is no match for the pod and selector/aadpodidbinding.

我正在使用 Terraform、Helm、Azure、Azure AKS、VSCODE 构建 IaaC，到目前为止，还使用了三个 Kubernetes 插件 - aad pod Identity、application-gateway-kubernetes-ingress 和 Bitnami external-dns。

由于身份未绑定(bind)，因此不会创建 azureAssignedIdentity，并且ExternalDNS 无法将记录放入我们的 DNS 区域。

名称和 aadpodidbindings 似乎是正确的。我尝试在 Terraform kubectl_manifest 提供程序中传递 fullnameOverride 来安装 BitnamiExternalDNS 的 Helm。我尝试抑制外部 DNS 名称和标签上的后缀。我尝试在集群本身上编辑 Helm 和 Kubernetes YAML 以尝试强制绑定(bind)。我尝试使用 AKS 用户管理的身份，该身份用于 AAD Pod 身份，位于集群的节点池资源组中。我尝试让 BitnamiExternalDNS 配置并添加 azure.json 文件，并且在添加和安装ExternalDNS 之前我也手动完成了此操作。我已尝试将托管标识分配给 AKS 群集的 VMSS。

谢谢!

JBP

PS C:\Workspace\tf\HelmOne> kubectl logs pod/external-dns-84dcc5f68c-542mv
: Refresh request failed. Status Code = '404'. Response body: getting assigned identities for pod default/external-dns-84dcc5f68c-542mv in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors\n"
time="2021-04-24T19:57:30Z" level=debug msg="Retrieving Azure DNS zones for resource group: one-hi-sso-dnsrg-tf."
time="2021-04-24T20:06:02Z" level=error msg="azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/8fb55161-REDACTED-3400b5271a8c/resourceGroups/one-hi-sso-dnsrg-tf/providers/Microsoft.Network/dnsZones?api-version=2018-05-01: StatusCode=404 -- Original Error: adal: Refresh request failed. Status Code = '404'. Response body: getting assigned identities for pod default/external-dns-84dcc5f68c-542mv in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors\n"
time="2021-04-24T20:06:02Z" level=debug msg="Retrieving Azure DNS zones for resource group: one-hi-sso-dnsrg-tf."



PS C:\Workspace\tf\HelmOne> kubectl logs pod/aad-pod-identity-nmi-vtmwm
I0424 20:07:22.400942       1 server.go:196] status (404) took 80007557875 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.0.8.7
E0424 20:08:44.427353       1 server.go:375] failed to get matching identities for pod: default/external-dns-84dcc5f68c-542mv, error: getting assigned identities for pod default/external-dns-84dcc5f68c-542mv in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0424 20:08:44.427400       1 server.go:196] status (404) took 80025612263 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=10.0.8.7



PS C:\Workspace\TF\HelmOne>  kubectl logs pod/aad-pod-identity-mic-86944f67b8-k4hds
I0422 21:05:11.298958       1 main.go:114] starting mic process. Version: v1.7.5. Build date: 2021-04-02-21:14
W0422 21:05:11.299031       1 main.go:119] --kubeconfig not passed will use InClusterConfig
I0422 21:05:11.299038       1 main.go:136] kubeconfig () cloudconfig (/etc/kubernetes/azure.json)
I0422 21:05:11.299205       1 main.go:144] running MIC in namespaced mode: false
I0422 21:05:11.299223       1 main.go:148] client QPS set to: 5. Burst to: 5
I0422 21:05:11.299243       1 mic.go:139] starting to create the pod identity client. Version: v1.7.5. Build date: 2021-04-02-21:14
I0422 21:05:11.318835       1 mic.go:145] Kubernetes server version: v1.18.14
I0422 21:05:11.319465       1 cloudprovider.go:122] MIC using user assigned identity: c380##### REDACTED #####814b for authentication.
I0422 21:05:11.392322       1 probes.go:41] initialized health probe on port 8080
I0422 21:05:11.392351       1 probes.go:44] started health probe
I0422 21:05:11.392458       1 metrics.go:341] registered views for metric
I0422 21:05:11.392544       1 prometheus_exporter.go:21] starting Prometheus exporter
I0422 21:05:11.392561       1 metrics.go:347] registered and exported metrics on port 8888
I0422 21:05:11.392568       1 mic.go:244] initiating MIC Leader election
I0422 21:05:11.393053       1 leaderelection.go:243] attempting to acquire leader lease  default/aad-pod-identity-mic...
E0423 01:47:52.730839       1 leaderelection.go:325] error retrieving resource lock default/aad-pod-identity-mic: etcdserver: request timed out



resource "helm_release" "external-dns" {
    name          = "external-dns"
    repository    = "https://charts.bitnami.com/bitnami"
    chart         = "external-dns"
    namespace     = "default"
    version       = "4.0.0"
  
  set {
      name  = "azure.cloud"
      value = "AzurePublicCloud"
  }
  
  #MyDnsResourceGroup
  set {
      name  = "azure.resourceGroup"
      value = data.azurerm_resource_group.dnsrg.name 
  }
  
  set {
      name  = "azure.tenantId"
      value = data.azurerm_subscription.currenttenantid.tenant_id
  }
  
  set {
      name  = "azure.subscriptionId"
      value = data.azurerm_subscription.currentSubscription.subscription_id
  }
  
  set {
      name    = "azure.userAssignedIdentityID"
      value   = azurerm_user_assigned_identity.external-dns-mi-tf.client_id
  }
  
  #Verbosity of the logs (options: panic, debug, info, warning, error, fatal, trace)
  set {
      name  = "logLevel"
      value = "trace" 
  }
  
  set {
      name  = "sources"
      value = "{service,ingress}"
  }
  
  set {
      name  = "domainFilters"
      value = "{${var.child_domain_prefix}.${lower(var.parent_domain)}}"
  }
  
  #DNS provider where the DNS records will be created (mandatory) (options: aws, azure, google, ...)
  set {
      name  = "provider"
      value = "azure"
  }
  
  #podLabels: {aadpodidbinding: <selector>} # selector you defined above in AzureIdentityBinding 
  set {
      name  = "podLabels.aadpodidbinding"
      value = "external-dns"
  }
  
  set {
      name = "azure.useManagedIdentityExtension"
      value = true
  }
}  



resource "helm_release" "aad-pod-identity" {
    name       = "aad-pod-identity"
    repository = "https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts"
    chart      = "aad-pod-identity"
  }


  resource "helm_release" "ingress-azure" {
    name          = "ingress-azure"
    repository    = "https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/"
    chart         = "ingress-azure"
    namespace     = "default"
    version       = "1.4.0"
  
  set {
      name        = "debug"
      value       = "true"
  }
  
  set {
      name        = "appgw.name"
      value       = data.azurerm_application_gateway.appgwpub.name
  }
  
  set {
      name        = "appgw.resourceGroup"
      value       = data.azurerm_resource_group.appgwpubrg.name
  }
  
  set {
      name        = "appgw.subscriptionId"
      value       = data.azurerm_subscription.currentSubscription.subscription_id
  }
  
  set {
      name        = "appgw.usePrivateIP"
      value       = "false"
  }
  
  set {
      name        = "armAuth.identityClientID"
      value       = azurerm_user_assigned_identity.agic-mi-tf.client_id
  }
  
  set {
      name        = "armAuth.identityResourceID"
      value       =  azurerm_user_assigned_identity.agic-mi-tf.id
  }
  
  set {
      name        = "armAuth.type"
      value       = "aadPodIdentity"
  }
  
  set {
      name        = "rbac.enabled"
      value       = "true"
  }
  
  set {
      name        = "verbosityLevel"
      value       = "5"
  }
  
  set {
      name        = "appgw.environment"
      value       = "AZUREPUBLICCLOUD"
  }
  
  set {
      name        = "metadata.name"
      value       = "ingress-azure"
  }
  
  }



PS C:\Workspace\tf\HelmOne> kubectl get azureassignedidentities
NAME                                                   AGE
ingress-azure-68c97fd496-qbptf-default-ingress-azure   23h


PS C:\Workspace\tf\HelmOne> kubectl get azureidentity
NAME                            AGE
ingress-azure                   23h
one-hi-sso-agic-mi-tf           23h
one-hi-sso-external-dns-mi-tf   23h


PS C:\Workspace\tf\HelmOne> kubectl edit azureidentity one-hi-sso-external-dns-mi-tf
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"aadpodidentity.k8s.io/v1","kind":"AzureIdentity","metadata":{"annotations":{},"name":"one-hi-sso-external-dns-mi-tf","namespace":"default"},"spec":{"clientID":"f58e7c55-REDACTED-a6e358e53912","resourceID":"/subscriptions/8fb55161-REDACTED-3400b5271a8c/resourceGroups/one-hi-sso-kuberg-tf/providers/Microsoft.ManagedIdentity/userAssignedIdentities/one-hi-sso-external-dns-mi-tf","type":0}}
  creationTimestamp: "2021-04-22T20:44:42Z"
  generation: 2
  name: one-hi-sso-external-dns-mi-tf
  namespace: default
  resourceVersion: "432055"
  selfLink: /apis/aadpodidentity.k8s.io/v1/namespaces/default/azureidentities/one-hi-sso-external-dns-mi-tf
  uid: f8e22fd9-REDACTED-6cdead0d7e22
spec:
  clientID: f58e7c55-REDACTED-a6e358e53912
  resourceID: /subscriptions/8fb55161-REDACTED-3400b5271a8c/resourceGroups/one-hi-sso-kuberg-tf/providers/Microsoft.ManagedIdentity/userAssignedIdentities/one-hi-sso-external-dns-mi-tf
  type: 0


PS C:\Workspace\tf\HelmOne> kubectl edit azureidentitybinding external-dns-mi-binding
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"aadpodidentity.k8s.io/v1","kind":"AzureIdentityBinding","metadata":{"annotations":{},"name":"external-dns-mi-binding","namespace":"default"},"spec":{"AzureIdentity":"one-hi-sso-external-dns-mi-tf","Selector":"external-dns"}}
  creationTimestamp: "2021-04-22T20:44:42Z"
  generation: 1
  name: external-dns-mi-binding
  namespace: default
  resourceVersion: "221101"
  selfLink: /apis/aadpodidentity.k8s.io/v1/namespaces/default/azureidentitybindings/external-dns-mi-binding
  uid: f39e7418-e896-4b8e-b596-035cf4b66252
spec:
  AzureIdentity: one-hi-sso-external-dns-mi-tf
  Selector: external-dns



  resource "kubectl_manifest" "one-hi-sso-external-dns-mi-tf" {
    yaml_body = <<YAML
apiVersion: "aadpodidentity.k8s.io/v1"  
kind: AzureIdentity
metadata:
  name: one-hi-sso-external-dns-mi-tf
  namespace: default
spec:
  type: 0
  resourceID: /subscriptions/8fb55161-REDACTED-3400b5271a8c/resourceGroups/one-hi-sso-kuberg-tf/providers/Microsoft.ManagedIdentity/userAssignedIdentities/one-hi-sso-external-dns-mi-tf
  clientID: f58e7c55-REDACTED-a6e358e53912
  YAML
}


resource "kubectl_manifest" "external-dns-mi-binding" {
    yaml_body = <<YAML
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: external-dns-mi-binding
spec:
  AzureIdentity: one-hi-sso-external-dns-mi-tf
  Selector: external-dns
YAML
}

Answer 1

我正在使用的托管标识未添加到虚拟机规模集 VMSS。添加后，绑定(bind)将起作用并创建 azureAssignedIdentity。

此外 - 我将 AzureIdentity YAML 中的 AzureIdentity 和选择器行从大写首字母转换为小写首字母。

正确: azure 身份:选择器: