Elasticsearch 7.2.0 : master not discovered or elected yet, 一次选举至少需要X个节点

Question

我正在尝试自动化 kubernetes 集群中 Elasticsearch 节点的水平扩展和缩减过程。

最初，我在 Kubernetes 集群上部署了一个 elasticsearch 集群(3 个主节点、3 个数据和 3 个摄取节点)。哪里，cluster.initial_master_nodes曾是:

cluster.initial_master_nodes:
  - master-a
  - master-b
  - master-c

然后，我进行了缩减操作，将主节点 3 的数量减少到 1 个(意外，但出于测试目的)。这样做时，我删除了 master-c , master-b节点并重新启动 master-a具有以下设置的节点:

cluster.initial_master_nodes:
  - master-a

由于 elasticsearch 节点(即 pods)使用持久卷，在重启节点后， master-a减慢以下日志:

"message": "master not discovered or elected yet, an election requires at least 2 nodes with ids from [TxdOAdryQ8GAeirXQHQL-g, VmtilfRIT6KDVv1R6MHGlw, KAJclUD2SM6rt9PxCGACSA], have discovered [] which is not a quorum; discovery will continue using [] from hosts providers and [{master-a}{VmtilfRIT6KDVv1R6MHGlw}{g29haPBLRha89dZJmclkrg}{10.244.0.95}{10.244.0.95:9300}{ml.machine_memory=12447109120, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 5, last-accepted version 40 in term 5"  }

似乎它正在试图找到 master-b和 master-c .

问题:

如何覆盖集群设置，以便 master-a不会搜索这些已删除的节点吗？

Answer 1

cluster.initial_master_nodes设置仅在集群第一次启动时有效，但为了避免一些非常罕见的极端情况，一旦设置它就不应更改其值，通常应尽快将其从配置文件中删除。来自 the reference manual关于 cluster.initial_master_nodes :

You should not use this setting when restarting a cluster or adding a new node to an existing cluster.

除此之外，Elasticsearch 使用 quorum-based election protocol并说以下内容:

To be sure that the cluster remains available you must not stop half or more of the nodes in the voting configuration at the same time.

您同时停止了三个符合主节点条件的节点中的两个(超过一半)，因此预计集群将不再工作。

引用手册还包含 instructions for removing master-eligible nodes你没有遵循:

As long as there are at least three master-eligible nodes in the cluster, as a general rule it is best to remove nodes one-at-a-time, allowing enough time for the cluster to automatically adjust the voting configuration and adapt the fault tolerance level to the new set of nodes.

If there are only two master-eligible nodes remaining then neither node can be safely removed since both are required to reliably make progress. To remove one of these nodes you must first inform Elasticsearch that it should not be part of the voting configuration, and that the voting power should instead be given to the other node.

它继续描述如何使用 POST /_cluster/voting_config_exclusions/node_name 从投票配置中安全地删除不需要的节点。当缩小到单个节点时。