hadoop - HIVE-JSON错误失败:添加请求失败:INSERT INTO `COLUMNS_V2` (`CD_ID` ,`COMMENT` ,`COLUMN_NAME` ,`TYPE_NAME` ,`INTEGER_IDX`)值

Question

我正在尝试使用具有500多个colyumns的Complex JSON定义HIVE EXTERNAL表。当我达到列的某个限制并且想知道如何克服它时，我得到一个错误。

完整的JSON具有548列的结构。我现在无法上传完整的JSON

CREATE STATEMENT成功，直到第243行。在我添加了SCHEMA定义的其余部分后，IT会在下面标记错误

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:javax.jdo.JDODataStoreException: Add request failed : INSERT INTO `COLUMNS_V2` (`CD_ID`,`COMMENT`,`COLUMN_NAME`,`TYPE_NAME`,`INTEGER_IDX`) VALUES (?,?,?,?,?) 
        at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
        at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
        at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
        at org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:855)
        at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98)
        at com.sun.proxy.$Proxy2.createTable(Unknown Source)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1394)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1427)
        at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
        at com.sun.proxy.$Proxy4.create_table_with_environment_context(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:8926)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:8910)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
NestedThrowablesStackTrace:
com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data too long for column 'TYPE_NAME' at row 1

Answer 1

Data too long for column 'TYPE_NAME'

列类型存储在 columns_v2.type_name的metastore中，该文件定义为varchar(4000)。
它太小，无法容纳一个或多个列类型表达式。

列数在这里不是问题。
该错误最有可能是由包含许多字段的 struct引起的。

例如 mycol struct<col1:int,col2:string,...,col1000:date>
在这种情况下，列类型为 struct<col1:int,col2:string,...,col1000:date>，并且超过4000个字符。

解决方法是，查看是否可以避免声明某些字段。

至于OP评论
JIRA
HiveHIVE-12274
Increase width of columns used for general configuration in the metastore.