kubernetes - serviceAccount token 量投影 - list 文件中 "path"中的投影 token 不正确

Question

我正在按照描述使用 serviceAccount token 量预测 here .这是我正在使用的 list 文件:

kind: Pod
apiVersion: v1
metadata:
  name: pod-acr
spec:
  containers:
  - image: yorchaksacr.azurecr.io/busybox:1.0
    name: busybox
    command: ["/bin/sh","-c","sleep 36000"]
    volumeMounts:
    - mountPath: /var/run/secrets/tokens
      name: vault-token
  serviceAccountName: pull-acr-images
  volumes:
  - name: vault-token
    projected:
      sources:
      - serviceAccountToken:
          path: vault-token
          expirationSeconds: 7200
          audience: vault

正如预期的那样， token 被挂载到 /var/run/secrets/tokens/vault-token 下的容器中:

/ # ls -la /var/run/secrets/tokens
total 4
drwxrwxrwt    3 root     root           100 Jul 24 21:35 .
drwxr-xr-x    4 root     root          4096 Jul 24 21:35 ..
drwxr-xr-x    2 root     root            60 Jul 24 21:35 ..2019_07_24_21_35_15.018111081
lrwxrwxrwx    1 root     root            31 Jul 24 21:35 ..data -> ..2019_07_24_21_35_15.018111081
lrwxrwxrwx    1 root     root            18 Jul 24 21:35 vault-token -> ..data/vault-token

问题是，如果我尝试使用此 token 向 API 服务器进行身份验证，API 会拒绝使用 401 Unauthorized 的调用:

/ # wget --header="Authorization: Bearer $(cat /var/run/secrets/tokens/vault-token)" --no-check-certificate https://10.2.1.19:6443
Connecting to 10.2.1.19:6443 (10.2.1.19:6443)
wget: server returned error: HTTP/1.1 401 Unauthorized

但是，如果我使用默认路径和 token ，其中为所有 pod 投影服务帐户 token /var/run/secrets/kubernetes.io/serviceacconts/token 有效:

/ # wget --header="Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --no-check-certificate https://10.2.1.19:6443
Connecting to 10.2.1.19:6443 (10.2.1.19:6443)
saving to 'index.html'
index.html           100% |************************************************************************************************************************************************************|  2738  0:00:00 ETA
'index.html' saved

如果我 cat 这两个标记，我可以看到它们实际上是不同的:

# cat /var/run/secrets/tokens/vault-token
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJhdWQiOlsidmF1bHQiXSwiZXhwIjoxNTY0MDEzMjcwLCJpYXQiOjE1NjQwMDYwNzAsImlzcyI6Imh0dHBzOi8vMTAuMi4xLjE5Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0IiwicG9kIjp7Im5hbWUiOiJwb2QtYWNyIiwidWlkIjoiNThiNjI5YWEtZGU4Ni00YTAzLWI3YmQtMTI4ZGFiZWVkYmQ5In0sInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJwdWxsLWFjci1pbWFnZXMiLCJ1aWQiOiJlZGE0NDlmYS1iODE2LTQ0ZGMtYTFjNi0yMWJhZWUwZmVkN2YifX0sIm5iZiI6MTU2NDAwNjA3MCwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6cHVsbC1hY3ItaW1hZ2VzIn0.UjPaTgWHPwCeeh1ltBb64hv0yEjKkRxKw_BZ3PLDA3HsJK-keXN40Khp-8mNnLQ-uYIfMgW4FXwYIm0SVeQUhM4sh4rwjAYDEfEHDah9AvhEL8I65T_jhnhT10E1M7mzk1x0RFGvjZAECd1RlYM7IuXIkEfZCI_6GRVAbX3Vmk6XF0sRh2T8DZzw8kj_Z54J2gYCt2beBnn7hC9rOC9LW9J0AFEAAQQE_UJME5y4jZD6hfJMSGOouyQm70nVGytqKVsLbzyorH5pugEqrs1Z_dLx6E3Ta9kELRPvyDZgeNiS44fEYlRApn6fZawsppc1oRNoeyMqiIPRdgQekBVfTA/ #


# cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InB1bGwtYWNyLWltYWdlcy10b2tlbi1oYjU0NyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwdWxsLWFjci1pbWFnZXMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJlZGE0NDlmYS1iODE2LTQ0ZGMtYTFjNi0yMWJhZWUwZmVkN2YiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwdWxsLWFjci1pbWFnZXMifQ.nqqhZVmBUuKVi6E3L9MEn8oW1dKd-DV4c9jcVy5mXAuEMZ1WgLlaaHFF1ibnVMjEK6VUJyJhp7w08hgSmyyh-KY4BQ5oJf1jmSySvmttJxjXW-KsMpf5rHF0ZDmgaqZwbi7FvowtoTECstFBVNoszKJUn1iV5mU_6MQkEtGTNyE4KuZ9LEvPuZxiNZ5UyW3UaHXLqF63-w_xlkfa_75E-cgXqvSSGTCb6RsTuOmVyCqganx5SpIb5EU-3Mu7hUWEhSRAh3tpcPIwjS7-NkuO0ReH7Z40rPHqkIokshUUO75WM_oPq7tlu6PSCTwOK-Jw66kzi-jqKNyKvMeWJUq4WQ/ #

有人知道我为什么会看到这种行为吗？我希望这两个 token 都能正常工作，但显然情况并非如此。

API 服务器的配置:

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.2.1.19
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --basic-auth-file=/etc/kubernetes/pki/passwordfile
    - --service-account-issuer=https://10.2.1.19
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    image: k8s.gcr.io/kube-apiserver:v1.15.1
    imagePullPolicy: IfNotPresent

Answer 1

令人困惑的是，401 Unauthorized 指示身份验证问题而不是授权问题(参见 here )。这意味着 Kubernetes 服务帐户 token 身份验证器不喜欢 /var/run/secrets/tokens/vault-token 中的 token (我们称它为 vault-token) .这两个 token 在几个方面有所不同。这是解码后的 vault-token:

{
  "aud": [
    "vault"
  ],
  "exp": 1564013270,
  "iat": 1564006070,
  "iss": "https://10.2.1.19",
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "pod-acr",
      "uid": "58b629aa-de86-4a03-b7bd-128dabeedbd9"
    },
    "serviceaccount": {
      "name": "pull-acr-images",
      "uid": "eda449fa-b816-44dc-a1c6-21baee0fed7f"
    }
  },
  "nbf": 1564006070,
  "sub": "system:serviceaccount:default:pull-acr-images"
}

注意受众(["vault"])、发行者("https://10.2.1.19")和主题("系统: serviceaccount:default:pull-acr-images").

这是默认路径标记:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "default",
  "kubernetes.io/serviceaccount/secret.name": "pull-acr-images-token-hb547",
  "kubernetes.io/serviceaccount/service-account.name": "pull-acr-images",
  "kubernetes.io/serviceaccount/service-account.uid": "eda449fa-b816-44dc-a1c6-21baee0fed7f",
  "sub": "system:serviceaccount:default:pull-acr-images"
}

相同的主题，但不同的发行者(“kubernetes/serviceaccount”)并且没有观众。

我不确定为什么 default-path-token 有不同的颁发者，或者为什么它能正确验证，但你的 vault-token 没有正确验证，因为观众与发行人不匹配。

更具体地说，线索在您链接的文档中 here .它表示此功能是否正常工作取决于您如何为 kube-apiserver 设置以下标志:

• --service-account-issuer
• --service-account-signing-key-file
• --service-account-api-audiences

不确定这些文档是否有误或只是过时了，但由于您使用的是 v1.15.1，因此这些标志现在称为:

• --service-account-issuer
• --service-account-signing-key-file string
• --api-audiences

flag documentation说到 --api-audiences 标志:

Identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. If the --service-account-issuer flag is configured and this flag is not, this field defaults to a single element list containing the issuer URL.

由于您没有设置此标志并且您有--service-account-issuer=https://10.2.1.19，并且您有audience: vault 在你的 pod spec 中，你的 token 将声明它绑定(bind)到 vault 观众，并且 token 认证器将尝试将它与 --service-account-issuer 标志，显然它们不匹配。

您可以通过在 Pod 规范中指定 audience: https://10.2.1.19 而不是 audience: vault 来使它们匹配。一个警告:这个解决方案在技术上可以确保 token 进行身份验证，但我不确定真正正确的答案是什么，因为在 pod 规范中使用这些标志和字段是真正的目的, 只是让这些字符串匹配可能有点 hacky。