kubernetes - GCP GKE : Service account "is attempting to grant RBAC permissions not currently held"

Question

我正在设置 CI/CD 管道，以自动部署基于 Kubernetes 的应用程序。此部署的一部分涉及创建其他服务帐户及其相关角色。

当我的管道运行时，部署失败并显示以下错误消息:

Error: roles.rbac.authorization.k8s.io "mongodb-kubernetes-operator" is forbidden: user "cicd-bot@my-project.iam.gserviceaccount.com" (groups=["system:authenticated"]) is attempting to grant RBAC permissions not currently held:
│ {APIGroups:[""], Resources:["configmaps"], Verbs:["list" "watch" "create" "update" "patch" "get" "delete"]}
│ {APIGroups:[""], Resources:["pods"], Verbs:["list" "watch" "create" "update" "patch" "get" "delete"]}
│ {APIGroups:[""], Resources:["secrets"], Verbs:["list" "watch" "create" "update" "patch" "get" "delete"]}
│ {APIGroups:[""], Resources:["services"], Verbs:["list" "watch" "create" "update" "patch" "get" "delete"]}
│ {APIGroups:["apps"], Resources:["statefulsets"], Verbs:["list" "watch" "create" "update" "patch" "get" "delete"]}
│ {APIGroups:["mongodbcommunity.mongodb.com"], Resources:["mongodbcommunity"], Verbs:["list" "watch" "update" "patch" "get"]}
│ {APIGroups:["mongodbcommunity.mongodb.com"], Resources:["mongodbcommunity/finalizers"], Verbs:["list" "watch" "update" "patch" "get"]}
│ {APIGroups:["mongodbcommunity.mongodb.com"], Resources:["mongodbcommunity/spec"], Verbs:["list" "watch" "update" "patch" "get"]}
│ {APIGroups:["mongodbcommunity.mongodb.com"], Resources:["mongodbcommunity/status"], Verbs:["list" "watch" "update" "patch" "get"]}
│ 
│   with module.db_document.kubernetes_role.operator_mongodb,
│   on modules/db_document/main.tf line 17, in resource "kubernetes_role" "operator_mongodb":
│   17: resource "kubernetes_role" "operator_mongodb" {
│ 

错误看起来很简单:我的服务帐户无法授予它没有的权限。由于错误消息提到了我的 GCP 服务帐户 cicd-bot@my-project.iam.gserviceaccount.com，我在我的角色定义中添加了我认为是匹配权限的内容。

以下是我的最终角色。它具有 configMaps、pod、secrets、services、statefulsets 和 thirdPartyObjects 的创建、删除、获取、列出和更新权限，我认为这些应该满足要求。

resource "google_project_iam_custom_role" "cicd_bot_role" {
  project = var.project
  role_id = "cicd_bot"
  title   = "CICD Bot"
  permissions = [
    "artifactregistry.repositories.downloadArtifacts",
    "artifactregistry.repositories.uploadArtifacts",
    "compute.instanceGroupManagers.get",
    "container.clusters.get",
    "container.configMaps.create",
    "container.configMaps.delete",
    "container.configMaps.get",
    "container.configMaps.list",
    "container.configMaps.update",
    "container.cronJobs.create",
    "container.cronJobs.delete",
    "container.cronJobs.get",
    "container.cronJobs.update",
    "container.customResourceDefinitions.create",
    "container.customResourceDefinitions.delete",
    "container.customResourceDefinitions.get",
    "container.customResourceDefinitions.list",
    "container.customResourceDefinitions.update",
    "container.deployments.create",
    "container.deployments.delete",
    "container.deployments.get",
    "container.deployments.update",
    "container.ingresses.create",
    "container.ingresses.delete",
    "container.ingresses.get",
    "container.ingresses.update",
    "container.jobs.create",
    "container.jobs.delete",
    "container.jobs.get",
    "container.jobs.update",
    "container.namespaces.get",
    "container.persistentVolumeClaims.create",
    "container.persistentVolumeClaims.delete",
    "container.persistentVolumeClaims.get",
    "container.persistentVolumeClaims.update",
    "container.pods.create",
    "container.pods.delete",
    "container.pods.get",
    "container.pods.list",
    "container.pods.update",
    "container.roleBindings.create",
    "container.roleBindings.delete",
    "container.roleBindings.get",
    "container.roleBindings.update",
    "container.roles.create",
    "container.roles.delete",
    "container.roles.get",
    "container.roles.update",
    "container.secrets.create",
    "container.secrets.delete",
    "container.secrets.get",
    "container.secrets.list",
    "container.secrets.update",
    "container.serviceAccounts.create",
    "container.serviceAccounts.delete",
    "container.serviceAccounts.get",
    "container.serviceAccounts.update",
    "container.services.create",
    "container.services.delete",
    "container.services.get",
    "container.services.list",
    "container.services.update",
    "container.statefulSets.create",
    "container.statefulSets.delete",
    "container.statefulSets.get",
    "container.statefulSets.list",
    "container.statefulSets.update",
    "container.thirdPartyObjects.create",
    "container.thirdPartyObjects.delete",
    "container.thirdPartyObjects.get",
    "container.thirdPartyObjects.list",
    "container.thirdPartyObjects.update",
    "dns.changes.create",
    "dns.changes.get",
    "dns.resourceRecordSets.get",
    "dns.resourceRecordSets.list",
    "dns.resourceRecordSets.update",
    "storage.buckets.get",
    "storage.objects.create",
    "storage.objects.delete",
    "storage.objects.get",
    "storage.objects.list",
  ]
}

然而，部署后，错误仍然存​​在。我想知道是否有必要在 kubernetes 端添加等效权限，所以我也创建了以下 ClusterRole 和 ClusterRoleBinding。

resource "kubernetes_cluster_role" "cicd_bot" {
  metadata {
    name = kubernetes_service_account.cicd_bot.metadata[0].name
  }
  rule {
    api_groups = [""]
    resources  = ["namespaces"]
    verbs      = ["create", "delete", "get"]
  }
  rule {
    api_groups = [""]
    resources  = ["configmaps"]
    verbs      = ["list", "watch", "create", "update", "patch", "get", "delete"]
  }
  rule {
    api_groups = [""]
    resources  = ["pods"]
    verbs      = ["list", "watch", "create", "update", "patch", "get", "delete"]
  }
  rule {
    api_groups = [""]
    resources  = ["secrets"]
    verbs      = ["list", "watch", "create", "update", "patch", "get", "delete"]
  }
  rule {
    api_groups = [""]
    resources  = ["services"]
    verbs      = ["list", "watch", "create", "update", "patch", "get", "delete"]
  }
  rule {
    api_groups = ["apps"]
    resources  = ["statefulsets"]
    verbs      = ["list", "watch", "create", "update", "patch", "get", "delete"]
  }
  rule {
    api_groups = ["mongodbcommunity.mongodb.com"]
    resources  = ["mongodbcommunity"]
    verbs      = ["list", "watch", "update", "patch", "get"]
  }
  rule {
    api_groups = ["mongodbcommunity.mongodb.com"]
    resources  = ["mongodbcommunity/finalizers"]
    verbs      = ["list", "watch", "update", "patch", "get"]
  }
  rule {
    api_groups = ["mongodbcommunity.mongodb.com"]
    resources  = ["mongodbcommunity/spec"]
    verbs      = ["list", "watch", "update", "patch", "get"]
  }
  rule {
    api_groups = ["mongodbcommunity.mongodb.com"]
    resources  = ["mongodbcommunity/status"]
    verbs      = ["list", "watch", "update", "patch", "get"]
  }
}

resource "kubernetes_cluster_role_binding" "cicd_bot" {
  metadata {
    name = kubernetes_service_account.cicd_bot.metadata[0].name
  }
  subject {
    kind      = "ServiceAccount"
    namespace = kubernetes_service_account.cicd_bot.metadata[0].namespace
    name      = kubernetes_service_account.cicd_bot.metadata[0].name
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = kubernetes_cluster_role.cicd_bot.metadata[0].name
  }
}

不幸的是，管道仍然失败并出现同样的错误。过去我已经能够克服类似的错误，但这次不行。我错过了什么？

更新:我能够通过将角色 roles/container.admin 附加到我的服务帐户来成功部署。所以现在我需要弄清楚哪些权限 roles/container.admin 具有我的自定义角色没有的权限。

Answer 1

可悲的是，缺少的一个权限是

container.roles.escalate

即使包括所有其他 container.* 权限也是不够的；仍然需要 container.roles.escalate。

这很不幸，因为它使集群更容易受到权限升级攻击。如果有更安全的方法来实现这一点，我很乐意听到。我不会将自己的答案标记为“正确”，因为我对此不满意。但是，嘿，至少它在工作......