cluster-computing - 错误 1047 (08S01) : WSREP has not yet prepared node for application use when I create database or use database

Question

问这个问题之前，我搜索了stackoverflow，发现了一个类似的帖子:
mariadb galera - Error when a node shutdown ERROR 1047 WSREP has not yet prepared node for application use

但是，帖子是双节点集群，我的是三节点集群，并且在每个节点中，mariadb 都运行如下:

Controller 1:

[root@controller1 mysql]# systemctl status mariadb
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: active (running) since Tue 2017-07-04 19:34:22 CST; 19h ago
 Main PID: 3785 (mysqld)
   Status: "Taking your SQL requests now..."
   CGroup: /system.slice/mariadb.service
           └─3785 /usr/sbin/mysqld --wsrep_start_position=de05f291-60a9-11e7-a641-13dca5e08098:0

Jul 05 14:52:54 controller1 mysqld[3785]: 2017-07-05 14:52:54 139817407719168 [Note] WSREP: applier thread exiting (code:6)
Jul 05 14:52:54 controller1 mysqld[3785]: 2017-07-05 14:52:54 139817408489216 [Note] WSREP: New cluster view: global state: 00000000-0000-0000-0000-000000000000:0, v... version -1
Jul 05 14:52:54 controller1 mysqld[3785]: 2017-07-05 14:52:54 139817408489216 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Jul 05 14:52:54 controller1 mysqld[3785]: 2017-07-05 14:52:54 139817408489216 [Note] WSREP: applier thread exiting (code:6)
Jul 05 14:55:41 controller1 mysqld[3785]: 2017-07-05 14:55:41 139817330894592 [Warning] WSREP: Setting parameter 'pc.bootstrap' to 'YES' failed: Setting 'pc.bootstra...tion abort)
Jul 05 14:55:41 controller1 mysqld[3785]: at galera/src/galera_gcs.hpp:param_set():223
Jul 05 14:55:41 controller1 mysqld[3785]: 2017-07-05 14:55:41 139817330894592 [ERROR] WSREP: Set options returned 7
Jul 05 15:04:39 controller1 mysqld[3785]: 2017-07-05 15:04:39 139817330894592 [Warning] WSREP: Setting parameter 'pc.bootstrap' to 'YES' failed: Setting 'pc.bootstra...tion abort)
Jul 05 15:04:39 controller1 mysqld[3785]: at galera/src/galera_gcs.hpp:param_set():223
Jul 05 15:04:39 controller1 mysqld[3785]: 2017-07-05 15:04:39 139817330894592 [ERROR] WSREP: Set options returned 7
Hint: Some lines were ellipsized, use -l to show in full.

Controller 2:

[root@controller2 ~]# systemctl status mariadb
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: active (running) since Tue 2017-07-04 19:32:33 CST; 19h ago
 Main PID: 3181 (mysqld)
   Status: "Taking your SQL requests now..."
   CGroup: /system.slice/mariadb.service
           └─3181 /usr/sbin/mysqld --wsrep_start_position=de05f291-60a9-11e7-a641-13dca5e08098:0

Jul 05 14:52:54 controller2 mysqld[3181]: 2017-07-05 14:52:54 140476124489472 [Note] WSREP: Closing replication queue.
Jul 05 14:52:54 controller2 mysqld[3181]: 2017-07-05 14:52:54 140476124489472 [Note] WSREP: Closing slave action queue.
Jul 05 14:52:54 controller2 mysqld[3181]: 2017-07-05 14:52:54 140476124489472 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 0)
Jul 05 14:52:54 controller2 mysqld[3181]: 2017-07-05 14:52:54 140476124489472 [Note] WSREP: RECV thread exiting -103: Software caused connection abort
Jul 05 14:52:54 controller2 mysqld[3181]: 2017-07-05 14:52:54 140478689122048 [Note] WSREP: applier thread exiting (code:6)
Jul 05 14:52:54 controller2 mysqld[3181]: 2017-07-05 14:52:54 140478689654528 [Note] WSREP: applier thread exiting (code:6)
Jul 05 14:52:54 controller2 mysqld[3181]: 2017-07-05 14:52:54 140478690424576 [Note] WSREP: applier thread exiting (code:6)
Jul 05 14:52:54 controller2 mysqld[3181]: 2017-07-05 14:52:54 140478689388288 [Note] WSREP: New cluster view: global state: 00000000-0000-0000-0000-000000000000:0, v... version -1
Jul 05 14:52:54 controller2 mysqld[3181]: 2017-07-05 14:52:54 140478689388288 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Jul 05 14:52:54 controller2 mysqld[3181]: 2017-07-05 14:52:54 140478689388288 [Note] WSREP: applier thread exiting (code:6)
Hint: Some lines were ellipsized, use -l to show in full.

Controller 3:

[root@controller3 mysql]# systemctl status mariadb
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: active (running) since Wed 2017-07-05 04:32:36 CST; 10h ago
 Main PID: 3176 (mysqld)
   Status: "Taking your SQL requests now..."
   CGroup: /system.slice/mariadb.service
           └─3176 /usr/sbin/mysqld --wsrep_start_position=de05f291-60a9-11e7-a641-13dca5e08098:0

Jul 05 15:11:12 controller3 mysqld[3176]: 2017-07-05 15:11:12 140609394308864 [Note] WSREP: (e95d1384, 'tcp://0.0.0.0:4567') reconnecting to 77...empt 1080
Jul 05 15:11:36 controller3 mysqld[3176]: 2017-07-05 15:11:36 140609394308864 [Note] WSREP: (e95d1384, 'tcp://0.0.0.0:4567') reconnecting to 7c...empt 1110
Jul 05 15:11:43 controller3 mysqld[3176]: 2017-07-05 15:11:43 140609394308864 [Note] WSREP: (e95d1384, 'tcp://0.0.0.0:4567') reconnecting to 77...empt 1110
Jul 05 15:12:06 controller3 mysqld[3176]: 2017-07-05 15:12:06 140609394308864 [Note] WSREP: (e95d1384, 'tcp://0.0.0.0:4567') reconnecting to 7c...empt 1140
Jul 05 15:12:13 controller3 mysqld[3176]: 2017-07-05 15:12:13 140609394308864 [Note] WSREP: (e95d1384, 'tcp://0.0.0.0:4567') reconnecting to 77...empt 1140
Jul 05 15:12:39 controller3 mysqld[3176]: 2017-07-05 15:12:39 140609394308864 [Note] WSREP: (e95d1384, 'tcp://0.0.0.0:4567') reconnecting to 7c...empt 1170
Jul 05 15:12:44 controller3 mysqld[3176]: 2017-07-05 15:12:44 140609394308864 [Note] WSREP: (e95d1384, 'tcp://0.0.0.0:4567') reconnecting to 77...empt 1170
Jul 05 15:13:09 controller3 mysqld[3176]: 2017-07-05 15:13:09 140609394308864 [Note] WSREP: (e95d1384, 'tcp://0.0.0.0:4567') reconnecting to 7c...empt 1200
Jul 05 15:13:15 controller3 mysqld[3176]: 2017-07-05 15:13:15 140609394308864 [Note] WSREP: (e95d1384, 'tcp://0.0.0.0:4567') reconnecting to 77...empt 1200
Jul 05 15:13:39 controller3 mysqld[3176]: 2017-07-05 15:13:39 140609394308864 [Note] WSREP: (e95d1384, 'tcp://0.0.0.0:4567') reconnecting to 7c...empt 1230
Hint: Some lines were ellipsized, use -l to show in full.

作为我登录到每个 mariadb 的标题，我使用或创建数据库我会得到错误:

ERROR 1047 (08S01): WSREP has not yet prepared node for application use when I create database or use database.

我不知道该怎么处理。

Answer 1

检查所有节点上的 cat /var/lib/mysql/grastate.dat。集群启动时，seqno 设置为 -1(即未知)，safe_to_bootstrap 设置为 0。当 mariadb 正常关闭时，seqno 设置为节点已知的最后一个事务。如果所有节点不干净地关闭，所有节点都会有 seqno: -1 和 safe_to_bootstrap: 0，这使得 galera 无法为任何加入者选择捐赠者。如果这是你的情况，这是解决方案:

在所有节点上停止 mariadb。必要时残忍地杀死它。

将 wsrep_recover=1 添加到所有节点上的 galera 配置文件中，然后启动它们。它们将短暂运行并退出并出现错误。

检查 systemctl status mariadb -l 或您的日志。您会找到类似“WSREP:恢复位置:01234567-89ab-cdef-0123-456789abcdef:999”的条目。所有节点上的 UUID 应该相同。冒号后面的数字是该节点的最后一个已知事务。

在所有节点上将 wsrep_recover 更改为 =0。

编辑每个节点的 grastate.dat 以包含正确的集群 UUID 和该节点的正确 seqno(在本例中为 999)。在具有最高 seqno 的节点上，也将 safe_to_bootstrap 更改为 1。

使用 galera_new_cluster 引导具有最高 seqno 的节点。跟踪日志并查看它是否已启动并稳定。然后正常启动( systemctl start mariadb )具有下一个最高序列号的节点。跟踪其日志并验证它是否与引导节点同步。

在这两个节点上分别登录mysql数据库，运行show global status like 'wsrep_cluster_size';和select user from user limit 10;。如果第一个结果是 2 而第二个结果不是错误，则您有一个工作集群。

按照各自 seqno 的降序正常启动剩余节点。

如果步骤 3 中的 UUID 在所有节点上都不相同，请仅在具有最高 seqno 的节点上修复 grastate.dat。删除其他节点上/var/lib/mysql/目录中的所有内容，包括隐藏的 .sst 目录(如果存在)。继续步骤 6-8。最后，想想你的守护天使，他在这个困惑发生之前对你的数据库进行了完整备份。