kubernetes - podAntiAffinity 不适用于 nodeAffinity

Question

我在 k8s 上进行了一系列部署，这些部署部署了相同的服务器二进制文件，但在内存限制等规范上有所不同。每个部署只运行 1 个 pod，并希望为这些部署安排 pod，以便每个 VM 仅安排单个 pod .

因为一些部署需要比其他部署更大的内存，我们结合 nodeAffinity 和 podAntiAffinity 来满足以下要求。

• 每种类型的实例将只分配一个标签为 pod_group_affinity == per_node
的 pod
• 每个具有不同内存需求的pod分配给具有相应内存大小的节点

下面是我的配置。

    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values:
                - ${INSTANCE_TYPE}
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: pod_group_affinity
                operator: In
                values:
                - per_node
            topologyKey: "kubernetes.io/hostname"

我首先尝试将本地 minikube 作为单个 VM(提供适当的 node.kubernetes.io/instance-type 值)，但所有部署的 pod 都已安排。它不应该发生，因为所有 Pod 都有 pod_group_affinity == per_node 标签。即使我删除了 nodeAffinity 部分，podAntiAffinity 似乎仍然不起作用。

我错过了什么吗？例如，podAntiAffinity 不适用于来自多个部署的 inter pod？但只要我看了文章https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#more-practical-use-cases ，podAntiAffinity 可以从另一个部署中引用 pod 的标签。

问候，

编辑:由于@confused genius 的建议，我添加了更多信息来调查原因。

• kubectl get nodes --show-labes

$ kubectl get nodes --show-labels
NAME       STATUS   ROLES                  AGE     VERSION   LABELS
minikube   Ready    control-plane,master   4d19h   v1.22.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=minikube,kubernetes.io/os=linux,minikube.k8s.io/commit=0a0ad764652082477c00d51d2475284b5d39ceed,minikube.k8s.io/name=minikube,minikube.k8s.io/updated_at=2022_01_19T12_04_46_0700,minikube.k8s.io/version=v1.23.2,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=,node.kubernetes.io/instance-type=Standard_B2ms,topology.hostpath.csi/node=minikube

我手动设置 node.kubernetes.io/instance-type=Standard_B2ms 来模拟真实环境，每个虚拟机都有相应实例类型的标签。

• 部署的模板部分由于工作的保密协议(protocol)，我无法部署真实版本，但大致如下。

spec:
  replicas: 1
  selector:
    matchLabels:
      app: ${NODE_NAME}
      pod_group_affinity: per_node
  template:
    metadata:
      labels:
        app: ${NODE_NAME}
        team: backend
        release: stable
        environment: ${ENV_LABEL}
        pod_group_affinity: per_node
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values:
                - ${INSTANCE_TYPE}
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: pod_group_affinity
                operator: In
                values:
                - per_node
            topologyKey: "kubernetes.io/hostname"                
      initContainers:
      - name: registration
        image: ${REGISTRATION_CONTAINER_IMAGE}
        env:
          - name: PASSWORD
            value: "${PASSWORD}"
        imagePullPolicy: IfNotPresent
        command: ["/bin/bash", "-c", "/workdir/run.sh"]
        volumeMounts:
        - name: node-config
          mountPath: /workdir/run.sh
          subPath: run.sh
        resources:
          requests:
            memory: "${MEMORY_REQUESTS}"
            cpu: "${CPU_REQUESTS}"
          limits:
            memory: "${MEMORY_LIMITS}"
      containers:
      - name: web
        image: ${WEB_CONTAINER_IMAGE}
        env:
          - name: USERNAME
            value: "${USERNAME}"
          - name: PASSWORD
            value: "${PASSWORD}"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: rest
          protocol: TCP
      - name: cordapp
        image: ${NODE_CONTAINER_IMAGE}
        env:
          - name: QUEUE_REGION
            valueFrom:
              configMapKeyRef:
                name: ${NODE_NAME}-config
                key: QUEUE_REGION
        imagePullPolicy: IfNotPresent
        command: ["/bin/bash", "-c", "/workdir/run.sh"]
        lifecycle:
          postStart:
            exec:
              command: ["/workdir/lifecycle.sh", "startup", "${GRACE_PERIOD_SECONDS}"]
          preStop:
            exec:
              command: ["/workdir/lifecycle.sh", "shutdown", "${GRACE_PERIOD_SECONDS}"]
        ports:
        - containerPort: 10000
          name: rpc
        volumeMounts:
        resources:
          requests:
            memory: "${MEMORY_REQUESTS}"
            cpu: "${CPU_REQUESTS}"
          limits:
            memory: "${MEMORY_LIMITS}"
      terminationGracePeriodSeconds: ${GRACE_PERIOD_SECONDS}
      volumes:
      - name: ${NODE_NAME}-config
        configMap:
          name: ${NODE_NAME}-config

Answer 1

我可以解决这个问题。

上述设置无法正常工作的原因是我将每个部署放在不同的命名空间中，如 https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity 中所述

最终的 yaml 看起来像:

      podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: pod_group_affinity
                operator: In
                values:
                - per_node
            topologyKey: "kubernetes.io/hostname"
            # this setting is important
            namespaces: ["node1", "node2", "node3"]