hadoop - 升级配置单元时重复的 key 名称

Question

在尝试将配置单元从2.1.0升级到2.3.0时遇到以下错误
(实际上将emr 5.7.0升级到5.8.0)
init hive-metastore schema]/returns: 0: jdbc:mysql://connection_string-> CREATE INDEX PCS_STATS_IDX ON PART_COL_STATS (DB_NAME,TABLE_NAME,COLUMN_NAME,PARTITION_NAME) USING BTREENotice: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]/returns: Error: Duplicate key name 'PCS_STATS_IDX' (state=42000,code=1061)Notice: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]/returns:Notice: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]/returns: Closing: 0: jdbc:mysql://connection_string:3306/hive?createDatabaseIfNotExist=trueNotice: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]/returns: org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!Notice: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]/returns: at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:586)Notice: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]/returns: ... 8 moreNotice: /Stage[main]/Hadoop_hive::Init_metastore_schema/Exec[init hive-metastore schema]/returns: *** schemaTool failed ***
有没有一种方法可以解决，而无需在配置单元外部metastore中手动删除已经创建的索引PCS_STATS_IDX？

Answer 1

我在EMR群集的RDS实例上的外部配置单元元数据库中也观察到了这一点。对于任何emr-5 +版本，似乎AWS都不是一个很好的解决方法-如果已经创建了元存储架构，则好像它应该能够跳过这些init hive-metastore脚本。解决此问题的一种可能方法是让EMR启动其自己的配置单元本地元存储(因此，在不更改初始hive-site.xml的情况下启动EMR群集)，并且在该群集启动后，运行一个EMR步骤，该步骤将停止hive-server2 sudo stop hive-server2，通过插入以下属性，修改/etc/hive/conf/hive-site.xml以将Hive Metastore指向外部RDS实例:

      <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>username</value>
      </property>

      <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>password</value>
      </property>

      <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://<rds instance connection string>:3306/db_name?createIfNotExist=true</value>
      </property>

并重新启动hive-server2( sudo start hive-server2)。