kubernetes - k8s : getting error 1 Insufficient memory, 1 个节点上的 HashicorpVault 与 Pod 亲和性/反亲和性不匹配-6ren

kubernetes - k8s : getting error 1 Insufficient memory, 1 个节点上的 HashicorpVault 与 Pod 亲和性/反亲和性不匹配

转载作者：行者123 更新时间：2023-12-03 08:28:36

我正在 k8s (EKS) 上部署 haVault，并在其中一个 Vault Pod 上收到此错误，我认为这也会导致其他 Pod 失败:这是 kubectl get events 的输出:
搜索:节点可用:1 内存不足

26m         Normal    Created                        pod/vault-1                                 Created container vault
26m         Normal    Started                        pod/vault-1                                 Started container vault
26m         Normal    Pulled                         pod/vault-1                                 Container image "hashicorp/vault-enterprise:1.5.0_ent" already present on machine
7m40s       Warning   BackOff                        pod/vault-1                                 Back-off restarting failed container
2m38s       Normal    Scheduled                      pod/vault-1                                 Successfully assigned vault-foo/vault-1 to ip-10-101-0-103.ec2.internal
2m35s       Normal    SuccessfulAttachVolume         pod/vault-1                                 AttachVolume.Attach succeeded for volume "pvc-acfc7e26-3616-4075-ab79-0c3f7b0f6470"
2m35s       Normal    SuccessfulAttachVolume         pod/vault-1                                 AttachVolume.Attach succeeded for volume "pvc-19d03d48-1de2-41f8-aadf-02d0a9f4bfbd"
48s         Normal    Pulled                         pod/vault-1                                 Container image "hashicorp/vault-enterprise:1.5.0_ent" already present on machine
48s         Normal    Created                        pod/vault-1                                 Created container vault
99s         Normal    Started                        pod/vault-1                                 Started container vault
60s         Warning   BackOff                        pod/vault-1                                 Back-off restarting failed container
27m         Normal    TaintManagerEviction           pod/vault-2                                 Cancelling deletion of Pod vault-foo/vault-2
28m         Warning   FailedScheduling               pod/vault-2                                 0/4 nodes are available: 1 Insufficient memory, 4 Insufficient cpu.
28m         Warning   FailedScheduling               pod/vault-2                                 0/5 nodes are available: 1 Insufficient memory, 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 4 Insufficient cpu.
27m         Normal    Scheduled                      pod/vault-2                                 Successfully assigned vault-foo/vault-2 to ip-10-101-0-103.ec2.internal
27m         Normal    SuccessfulAttachVolume         pod/vault-2                                 AttachVolume.Attach succeeded for volume "pvc-fb91141d-ebd9-4767-b122-da8c98349cba"
27m         Normal    SuccessfulAttachVolume         pod/vault-2                                 AttachVolume.Attach succeeded for volume "pvc-95effe76-6e01-49ad-9bec-14e091e1a334"
27m         Normal    Pulling                        pod/vault-2                                 Pulling image "hashicorp/vault-enterprise:1.5.0_ent"
27m         Normal    Pulled                         pod/vault-2                                 Successfully pulled image "hashicorp/vault-enterprise:1.5.0_ent"
26m         Normal    Created                        pod/vault-2                                 Created container vault
26m         Normal    Started                        pod/vault-2                                 Started container vault
26m         Normal    Pulled                         pod/vault-2                                 Container image "hashicorp/vault-enterprise:1.5.0_ent" already present on machine
7m26s       Warning   BackOff                        pod/vault-2                                 Back-off restarting failed container
2m36s       Warning   FailedScheduling               pod/vault-2                                 0/7 nodes are available: 1 Insufficient memory, 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules, 1 node(s) had volume node affinity conflict, 1 node(s) were unschedulable, 4 Insufficient cpu.
114s        Warning   FailedScheduling               pod/vault-2                                 0/8 nodes are available: 1 Insufficient memory, 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules, 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 1 node(s) had volume node affinity conflict, 1 node(s) were unschedulable, 4 Insufficient cpu.
104s        Warning   FailedScheduling               pod/vault-2                                 0/9 nodes are available: 1 Insufficient memory, 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules, 1 node(s) had volume node affinity conflict, 1 node(s) were unschedulable, 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 4 Insufficient cpu.
93s         Normal    Scheduled                      pod/vault-2                                 Successfully assigned vault-foo/vault-2 to ip-10-101-0-82.ec2.internal
88s         Normal    SuccessfulAttachVolume         pod/vault-2                                 AttachVolume.Attach succeeded for volume "pvc-fb91141d-ebd9-4767-b122-da8c98349cba"
88s         Normal    SuccessfulAttachVolume         pod/vault-2                                 AttachVolume.Attach succeeded for volume "pvc-95effe76-6e01-49ad-9bec-14e091e1a334"
83s         Normal    Pulling                        pod/vault-2                                 Pulling image "hashicorp/vault-enterprise:1.5.0_ent"
81s         Normal    Pulled                         pod/vault-2                                 Successfully pulled image "hashicorp/vault-enterprise:1.5.0_ent"
38s         Normal    Created                        pod/vault-2                                 Created container vault
37s         Normal    Started                        pod/vault-2                                 Started container vault
38s         Normal    Pulled                         pod/vault-2                                 Container image "hashicorp/vault-enterprise:1.5.0_ent" already present on machine
4s          Warning   BackOff                        pod/vault-2                                 Back-off restarting failed container
2m38s       Normal    Scheduled                      pod/vault-agent-injector-d54bdc675-qwsmz    Successfully assigned vault-foo/vault-agent-injector-d54bdc675-qwsmz to ip-10-101-2-91.ec2.internal
2m37s       Normal    Pulling                        pod/vault-agent-injector-d54bdc675-qwsmz    Pulling image "hashicorp/vault-k8s:latest"
2m36s       Normal    Pulled                         pod/vault-agent-injector-d54bdc675-qwsmz    Successfully pulled image "hashicorp/vault-k8s:latest"
2m36s       Normal    Created                        pod/vault-agent-injector-d54bdc675-qwsmz    Created container sidecar-injector
2m35s       Normal    Started                        pod/vault-agent-injector-d54bdc675-qwsmz    Started container sidecar-injector
28m         Normal    Scheduled                      pod/vault-agent-injector-d54bdc675-wz9ws    Successfully assigned vault-foo/vault-agent-injector-d54bdc675-wz9ws to ip-10-101-0-87.ec2.internal
28m         Normal    Pulled                         pod/vault-agent-injector-d54bdc675-wz9ws    Container image "hashicorp/vault-k8s:latest" already present on machine
28m         Normal    Created                        pod/vault-agent-injector-d54bdc675-wz9ws    Created container sidecar-injector
28m         Normal    Started                        pod/vault-agent-injector-d54bdc675-wz9ws    Started container sidecar-injector
3m22s       Normal    Killing                        pod/vault-agent-injector-d54bdc675-wz9ws    Stopping container sidecar-injector
3m22s       Warning   Unhealthy                      pod/vault-agent-injector-d54bdc675-wz9ws    Readiness probe failed: Get https://10.101.0.73:8080/health/ready: dial tcp 10.101.0.73:8080: connect: connection refused
3m18s       Warning   Unhealthy                      pod/vault-agent-injector-d54bdc675-wz9ws    Liveness probe failed: Get https://10.101.0.73:8080/health/ready: dial tcp 10.101.0.73:8080: connect: no route to host
28m         Normal    SuccessfulCreate               replicaset/vault-agent-injector-d54bdc675   Created pod: vault-agent-injector-d54bdc675-wz9ws
2m38s       Normal    SuccessfulCreate               replicaset/vault-agent-injector-d54bdc675   Created pod: vault-agent-injector-d54bdc675-qwsmz
28m         Normal    ScalingReplicaSet              deployment/vault-agent-injector             Scaled up replica set vault-agent-injector-d54bdc675 to 1
2m38s       Normal    ScalingReplicaSet              deployment/vault-agent-injector             Scaled up replica set vault-agent-injector-d54bdc675 to 1
28m         Normal    EnsuringLoadBalancer           service/vault-ui                            Ensuring load balancer
28m         Normal    EnsuredLoadBalancer            service/vault-ui                            Ensured load balancer
26m         Normal    UpdatedLoadBalancer            service/vault-ui                            Updated load balancer with new hosts
3m24s       Normal    DeletingLoadBalancer           service/vault-ui                            Deleting load balancer
3m23s       Warning   PortNotAllocated               service/vault-ui                            Port 32476 is not allocated; repairing
3m23s       Warning   ClusterIPNotAllocated          service/vault-ui                            Cluster IP 172.20.216.143 is not allocated; repairing
3m22s       Warning   FailedToUpdateEndpointSlices   service/vault-ui                            Error updating Endpoint Slices for Service vault-foo/vault-ui: failed to update vault-ui-crtg4 EndpointSlice for Service vault-foo/vault-ui: Operation cannot be fulfilled on endpointslices.discovery.k8s.io "vault-ui-crtg4": the object has been modified; please apply your changes to the latest version and try again
3m16s       Warning   FailedToUpdateEndpoint         endpoints/vault-ui                          Failed to update endpoint vault-foo/vault-ui: Operation cannot be fulfilled on endpoints "vault-ui": the object has been modified; please apply your changes to the latest version and try again
2m52s       Normal    DeletedLoadBalancer            service/vault-ui                            Deleted load balancer
2m39s       Normal    EnsuringLoadBalancer           service/vault-ui                            Ensuring load balancer
2m36s       Normal    EnsuredLoadBalancer            service/vault-ui                            Ensured load balancer
96s         Normal    UpdatedLoadBalancer            service/vault-ui                            Updated load balancer with new hosts
28m         Normal    NoPods                         poddisruptionbudget/vault                   No matching pods found
28m         Normal    SuccessfulCreate               statefulset/vault                           create Pod vault-0 in StatefulSet vault successful
28m         Normal    SuccessfulCreate               statefulset/vault                           create Pod vault-1 in StatefulSet vault successful
28m         Normal    SuccessfulCreate               statefulset/vault                           create Pod vault-2 in StatefulSet vault successful
2m40s       Normal    NoPods                         poddisruptionbudget/vault                   No matching pods found
2m38s       Normal    SuccessfulCreate               statefulset/vault                           create Pod vault-0 in StatefulSet vault successful
2m38s       Normal    SuccessfulCreate               statefulset/vault                           create Pod vault-1 in StatefulSet vault successful
2m38s       Normal    SuccessfulCreate               statefulset/vault                           create Pod vault-2 in StatefulSet vault successful

这是我的 Helm :

# Vault Helm Chart Value Overrides
global:
  enabled: true
  tlsDisable: false

injector:
  enabled: true
  # Use the Vault K8s Image https://github.com/hashicorp/vault-k8s/
  image:
    repository: "hashicorp/vault-k8s"
    tag: "latest"

  resources:
      requests:
        memory: 256Mi
        cpu: 250m
      limits:
        memory: 256Mi
        cpu: 250m

server:
  # Use the Enterprise Image
  image:
    repository: "hashicorp/vault-enterprise"
    tag: "1.5.0_ent"

  # These Resource Limits are in line with node requirements in the
  # Vault Reference Architecture for a Small Cluster
  resources:
    requests:
      memory: 8Gi
      cpu: 2000m
    limits:
      memory: 16Gi
      cpu: 2000m

  # For HA configuration and because we need to manually init the vault,
  # we need to define custom readiness/liveness Probe settings
  readinessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204"
  livenessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true"
    initialDelaySeconds: 60

  # extraEnvironmentVars is a list of extra environment variables to set with the stateful set. These could be
  # used to include variables required for auto-unseal.
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca

  # extraVolumes is a list of extra volumes to mount. These will be exposed
  # to Vault in the path .
  #extraVolumes:
  #  - type: secret
  #    name: tls-server
  #  - type: secret
  #    name: tls-ca
  #  - type: secret
  #    name: kms-creds
  extraVolumes:
    - type: secret
      name: vault-server-tls   
  
  # This configures the Vault Statefulset to create a PVC for audit logs.
  # See https://www.vaultproject.io/docs/audit/index.html to know more
  auditStorage:
    enabled: true

  standalone:
    enabled: false

  # Run Vault in "HA" mode.
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true

      config: |
        ui = true
        listener "tcp" {
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
          tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          tls_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
        }

        storage "raft" {
          path = "/vault/data"
            retry_join {
            leader_api_addr = "http://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
            leader_api_addr = "http://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
            leader_api_addr = "http://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
        }

        service_registration "kubernetes" {}

# Vault UI
ui:
  enabled: true
  serviceType: "LoadBalancer"
  serviceNodePort: null
  externalPort: 8200

  # For Added Security, edit the below
  #loadBalancerSourceRanges:
  #   - < Your IP RANGE Ex. 10.0.0.0/16 >
  #   - < YOUR SINGLE IP Ex. 1.78.23.3/32 >

我哪里配置不对？

最佳答案

这里有几个问题，它们都由如下错误消息表示:

0/9 nodes are available: 1 Insufficient memory, 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules, 1 node(s) had volume node affinity conflict, 1 node(s) were unschedulable, 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 4 Insufficient cpu.

您有 9 个节点，但由于条件不同，没有一个节点可用于调度。请注意，每个节点都可能受到多个问题的影响，因此这些数字加起来可能会超过节点总数。

让我们一一分解:

Insufficient memory :执行kubectl describe node <node-name>检查有多少可用内存。检查 pod 的请求和限制。请注意，Kubernetes 将阻止 pod 请求的全部内存，无论该 pod 使用了多少内存。
Insufficient cpu :同上类推。
node(s) didn't match pod affinity/anti-affinity :检查您的affinity/anti-affinity规则。
node(s) didn't satisfy existing pods anti-affinity rules :同上。
node(s) had volume node affinity conflict :当 pod 由于无法从另一个可用区连接到卷而无法调度时会发生。您可以通过创建 storageclass 来解决此问题对于单个区域，然后使用 storageclass在你的 PVC 中。
node(s) were unschedulable :这是因为该节点被标记为 Unschedulable 。这将我们引向下面的下一期:
node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate :这对应于NodeCondition Ready =False 。您可以使用kubectl描述节点来检查污点和kubectl taint nodes <node-name> <taint-name>-以便删除它们。检查Taints and Tolerations了解更多详情。

还有一个GitHub thread遇到类似的问题，您可能会觉得有用。

尝试逐一检查/消除这些问题(从上面列出的第一个问题开始)，因为它们在某些情况下可能会产生“链式 react ”。

关于kubernetes - k8s : getting error 1 Insufficient memory, 1 个节点上的 HashicorpVault 与 Pod 亲和性/反亲和性不匹配，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/65841399/

文章推荐： design-patterns - 在 Erlang 中实现取消

文章推荐： r - 向字符串/字符向量中的每个元素添加可变长度填充

java - 反/序列化二进制数据
我目前正在对一个 mmorpg 的二进制网络协议(protocol)进行逆向工程。我正在用 java 实现该协议(protocol)。对于每个数据包类型，我将创建一个表示二进制数据的类。例如，聊天
c# - (反)序列化对象的属性而不获取对象的其余部分
我正在尝试围绕现有类编写半透明包装器，我希望它能够模仿其他类的序列化。例如，给定以下类: class Foo { [JsonConverter(CustomConverter)] s
java - (反)序列化枚举集
是否有使用 Jackson 序列化和反序列化枚举集的简单方法？ private enum Type { YES, NO } @JacksonXmlProperty(localName = "t
Java(反)序列化过程
我很想知道当我们反序列化一个对象时会发生什么。例如，如果我的类对象由许多其他对象组成，对象创建过程如何在反序列化过程中发生最佳答案对象是用默认的初始化字段创建的，然后用从串行流中获取的属性值填充
c++ - (反)序列化枚举类
我正在尝试序列化和反序列化(使用 QDataStream 但这与这里无关)一个 enum class变量: enum class Type : char { Trivial, Comp
c++ - 编译麻烦c转c++反
我不确定这到底有什么问题...它不会为我编译，我将它从 c 翻译成 C++(或尝试)...是的，我是初学者。谢谢! #include #include using namespace std; i
json4s - json4s中是否有针对UUID的开箱即用的(反)序列化器？
我遇到的问题与此处描述的问题非常相似:Combining type and field serializers case class(id: Option[UUID], otherValue:Stri
r - 在(反)对角线上应用函数
我们知道base中的apply()可以对数组的边距应用一个函数，边距应该是行或列。我想将边距扩大到“对角线” 和“反对角线”。结构看起来像 diagApply <- function(x, FUN,
json - 如何(反)序列化对象列表？
我找到了 JSON serialization and deserialization to objects in Flutter 的例子但是如何使用像这样的人员列表来做到这一点: [ {
elasticsearch - elasticsearch必须与MUST_NOT(反)差异
我有一个相当大的terms聚合结果，这些结果被加载到下拉列表中以提供filter功能。可以说，我的下拉列表中有4000多种动物。我的另一个下拉列表有4种动物颜色。例， animal --> ["d
c# - 如何(反)序列化具有字符串数组值的可序列化字典？
我需要将 C# (.NET Framework 4.5.2) 中的一个类与 XML 序列化(反序列化)，该类具有 string 的字典属性。键和 string[]数组值。我正在使用 Serializa
c# - 使用json(反)序列化多个对象时的空问题
[已解决]应用给定的解决方案，效果很好! 程序的目的:在用户打开和关闭程序时保存/重新加载以前的数据。我曾经用一个对象(obj)成功(反)序列化，现在我有两个不同类的不同对象。我试图通过查看其他帖
c# - 通过属性在(反)序列化期间控制枚举值格式
问题假设我有一个代表某事或其他的枚举: public enum ResultState { Found, Deleted, NotFound } 在我的序列化 json 中，
javascript - 反 JSON 字符串
是否有取消 JSON 字符串的功能？我猜它不会内置到 JQuery 中，但它可以通过编写一个操纵字符串的脚本来实现吗？我在下面遇到了这个问题。我正在使用 NYTimes API，但它不支持 JSON
java - Java(反)序列化是原子的吗？
对于这个问题，假设当对象完全写入流并成功读出时，或者当对象部分写入流并且读回对象时发生异常时，序列化/反序列化是原子的。假设写操作可能无法成功完成，例如因为停电了。在Serializable的描述中
c - 反 VM 的时间检查效果不佳？
有谁知道时序检查是否仍在检测虚拟环境？我尝试使用 rdtsc 指令来获取 cpu 周期并比较真实 linux 机器和在 virtualbox 上运行的 linux 之间的结果。但结果似乎不稳定。有时，
c# - XML-(反)序列化整数的列表属性作为简单的分隔符分隔序列
我正在对一个(外部给定的)XML 文件进行操作，该文件具有以下形式的元素 10 20 30 40 50 60 70 80 我知道如何将属性作为属性处理(通过使用 [XmlAttri
c# - XML(反)序列化错误命名空间
我有一个通用的序列化器和反序列化器，用于通过网络连接发送的消息: public static async Task SerializeObject(Object obj) {
c# - (反)序列化特定子类而不使用 [ProtoInclude]
我正在考虑将当前基于 WCF 的应用程序迁移到 protobuf-net.Grpc。这似乎是可行的，但是我无法在不包含所有具有 [ProtoInclude] 属性的派生类的情况下使(DTO 类)基类的
c# - 保存的文件包含(反)斜杠
我正在尝试将一些数据保存到文件中，但文件保存到的目录不正确。 using (StreamWriter sw = new StreamWriter(dir + "\\temp" + x + ".txt"

行者123

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

kubernetes - k8s : getting error 1 Insufficient memory, 1 个节点上的 HashicorpVault 与 Pod 亲和性/反亲和性不匹配