gpt4 book ai didi

hadoop - org.apache.hadoop.hbase.RegionTooBusyException

转载 作者:行者123 更新时间:2023-12-02 20:24:20 24 4
gpt4 key购买 nike

我正在尝试使用hive-HBase集成从Hive到Hbase加载 30亿个记录(ORC文件)。

配置单元创建表DDL

CREATE EXTERNAL TABLE cs.account_dim_hbase(`account_number` string,`encrypted_account_number` string,`affiliate_code` string,`alternate_party_name` string, `alternate_party_name` string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping"=":key,account_dim:encrypted_account_number,account_dim:affiliate_code,account_dim:alternate_party_name")TBLPROPERTIES ("hbase.table.name" = "default:account_dim");

Hive插入查询到HBase,我正在运行128插入命令,类似于以下示例。
insert  into table cs.account_dim_hbase  select account_number ,encrypted_account_number ,    affiliate_code ,alternate_party_name,mod_account_number from cds.account_dim where mod_account_number=1;

当我尝试同时运行所有128个插入时,出现以下错误
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 438 actions: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=2.0G, regionName=jhgjhsdgfjgsdjf, server=cldf0007.com

帮助我解决此问题,并让我知道如果做错了什么。我正在使用HDP 3

最佳答案

在行键字段上使用MD5哈希从hive加载数据,并使用区域拆分创建HBASE表。现在每个分区仅需5分钟即可加载数据(之前是20分钟,但有异常(exception),但现在已修复)

create ‘users, ‘usercf’, SPLITS=›
['10000000000000000000000000000000',
'20000000000000000000000000000000',
'30000000000000000000000000000000',
'40000000000000000000000000000000',
'50000000000000000000000000000000',
'60000000000000000000000000000000',
'70000000000000000000000000000000',
'80000000000000000000000000000000',
'90000000000000000000000000000000',
'a0000000000000000000000000000000',
'b0000000000000000000000000000000',
'c0000000000000000000000000000000',
'd0000000000000000000000000000000',
'e0000000000000000000000000000000',
'f0000000000000000000000000000000']

关于hadoop - org.apache.hadoop.hbase.RegionTooBusyException,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/58100515/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com