xml - 在 Pig 中使用 Hcatalog 加载配置单元表时出错-6ren

xml - 在 Pig 中使用 Hcatalog 加载配置单元表时出错

转载作者：可可西里更新时间：2023-11-01 15:27:21

我正在尝试使用 pig 中的 Hcatalog 加载我的配置单元表，为此我编写了下面的代码，但出现错误。我正在使用 pig -useHCatalog

打开我的 pig 壳

代码:

A = LOAD 'patient_info' USING org.apache.hive.hcatalog.pig.HCatLoader();

错误:

ERROR hive.ql.metadata.Table - Unable to get field from serde: com.ibm.spss.hive.serde2.xml.XmlSerDe java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class com.ibm.spss.hive.serde2.xml.XmlSerDe not found) at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:275) at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:255) at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:602) at org.apache.hive.hcatalog.common.HCatUtil.getTableSchemaWithPtnCols(HCatUtil.java:184) at org.apache.hive.hcatalog.pig.HCatLoader.getSchema(HCatLoader.java:216) at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175) at org.apache.pig.newplan.logical.relational.LOLoad.(LOLoad.java:89) at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:866) at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188) at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1688) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1635) at org.apache.pig.PigServer.registerQuery(PigServer.java:587) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1093) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) at org.apache.pig.Main.run(Main.java:547) at org.apache.pig.Main.main(Main.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: MetaException(message:java.lang.ClassNotFoundException Class com.ibm.spss.hive.serde2.xml.XmlSerDe not found) at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:400)

更新:

我在 hive 中存储数据的命令如下。

add jar /home/cloudera/hivexmlserde-1.0.5.3.jar;
CREATE EXTERNAL TABLE patient_info (
statusCode string,
title string,
startTime string,
endTime string,
frequencyValue string,
frequencyUnits string

)
ROW FORMAT SERDE 'com.ibm.spss.hive.serde2.xml.XmlSerDe'
WITH SERDEPROPERTIES (
"column.xpath.statusCode"="medicationsInfo/entryInfo/statusCode/text()",
"column.xpath.title"="medications/code/code/text()",
"column.xpath.startTime"="medications/xxx/startTime/text()",
"column.xpath.endTime"="medications/xxx/endTime/text()",
"column.xpath.frequencyValue"="medications/xxx/frequencyValue/text()",
"column.xpath.frequencyUnits"="medications/xxx/frequencyUnits/text()",
)
STORED AS
INPUTFORMAT 'com.ibm.spss.hive.serde2.xml.XmlInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
TBLPROPERTIES (
"xmlinput.start"="<medicationsInfo",
"xmlinput.end"="</medicationsInfo>");


load data inpath '/user/cloudera/xml' into table patient_info ;

示例:

 <Document>
    <ProductCode>
     <code>10160-0</code>
     <entryInfo> 
        <statusCode>completed</statusCode>
        <startTime>20110729</startTime>
        <endTime>20110822</endTime>
        <strengthValue>24</strengthValue>
        <strengthUnits>h</strengthUnits>
     </entryInfo> 
     <entryInfo>
        <statusCode>completed</statusCode>
        <startTime>20120130</startTime>
        <endTime>20120326</endTime>
        <strengthValue>12</strengthValue>
        <strengthUnits>h</strengthUnits>
     </entryInfo>
     <entryInfo>
        <statusCode>completed</statusCode>
        <startTime>20100412</startTime>
        <endTime>20110822</endTime>
        <strengthValue>8</strengthValue>
        <strengthUnits>d</strengthUnits>
     </entryInfo> 
    </ProductCode>
    <ProductCode>
     <code>10160-0</code>
     <entryInfo> 
        <statusCode>completed</statusCode>
        <startTime>20110729</startTime>
        <endTime>20110822</endTime>
        <strengthValue>24</strengthValue>
        <strengthUnits>h</strengthUnits>
     </entryInfo> 
     <entryInfo>
        <statusCode>completed</statusCode>
        <startTime>20120130</startTime>
        <endTime>20120326</endTime>
        <strengthValue>12</strengthValue>
        <strengthUnits>h</strengthUnits>
     </entryInfo>
     <entryInfo>
        <statusCode>completed</statusCode>
        <startTime>20100412</startTime>
        <endTime>20110822</endTime>
        <strengthValue>8</strengthValue>
        <strengthUnits>d</strengthUnits>
     </entryInfo> 
    </ProductCode> 
   <Medicationsinfo>
     <code>10160-0</code>
     <entryInfo> 
        <statusCode>completed</statusCode>
        <startTime>20110729</startTime>
        <endTime>20110822</endTime>
        <strengthValue>24</strengthValue>
        <strengthUnits>h</strengthUnits>
     </entryInfo> 
     <entryInfo>
        <statusCode>completed</statusCode>
        <startTime>20120130</startTime>
        <endTime>20120326</endTime>
        <strengthValue>12</strengthValue>
        <strengthUnits>h</strengthUnits>
     </entryInfo>
     <entryInfo>
        <statusCode>completed</statusCode>
        <startTime>20100412</startTime>
        <endTime>20110822</endTime>
        <strengthValue>8</strengthValue>
        <strengthUnits>d</strengthUnits>
     </entryInfo> 
    </Medicationsinfo>
    <Medicationsinfo>
     <code>10160-0</code>
     <entryInfo> 
        <statusCode>completed</statusCode>
        <startTime>20110729</startTime>
        <endTime>20110822</endTime>
        <strengthValue>24</strengthValue>
        <strengthUnits>h</strengthUnits>
     </entryInfo> 
     <entryInfo>
        <statusCode>completed</statusCode>
        <startTime>20120130</startTime>
        <endTime>20120326</endTime>
        <strengthValue>12</strengthValue>
        <strengthUnits>h</strengthUnits>
     </entryInfo>
     <entryInfo>
        <statusCode>completed</statusCode>
        <startTime>20100412</startTime>
        <endTime>20110822</endTime>
        <strengthValue>8</strengthValue>
        <strengthUnits>d</strengthUnits>
     </entryInfo> 
    </Medicationsinfo> 
    </Document>

最佳答案

外部表的定义无效。以下是一些选项:

选项 1

create external table patient_info 
(
    code        string
   ,entryInfo   string
)
row format serde 'com.ibm.spss.hive.serde2.xml.XmlSerDe'
with serdeproperties 
(
    "column.xpath.code"      = "/Medicationsinfo/code/text()"
   ,"column.xpath.entryInfo" = "/Medicationsinfo/entryInfo"
)
stored as
inputformat 'com.ibm.spss.hive.serde2.xml.XmlInputFormat'
outputformat 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
location '/user/hive/warehouse/patient_info'
tblproperties 
(
    "xmlinput.start" = "<Medicationsinfo"
   ,"xmlinput.end"   = "</Medicationsinfo>"
)
;

select * from patient_info
;

+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| patient_info.code | patient_info.entryinfo                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 10160-0           | <string><entryInfo><statusCode>completed</statusCode><startTime>20110729</startTime><endTime>20110822</endTime><strengthValue>24</strengthValue><strengthUnits>h</strengthUnits></entryInfo><entryInfo><statusCode>completed</statusCode><startTime>20120130</startTime><endTime>20120326</endTime><strengthValue>12</strengthValue><strengthUnits>h</strengthUnits></entryInfo><entryInfo><statusCode>completed</statusCode><startTime>20100412</startTime><endTime>20110822</endTime><strengthValue>8</strengthValue><strengthUnits>d</strengthUnits></entryInfo></string> |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 10160-0           | <string><entryInfo><statusCode>completed</statusCode><startTime>20110729</startTime><endTime>20110822</endTime><strengthValue>24</strengthValue><strengthUnits>h</strengthUnits></entryInfo><entryInfo><statusCode>completed</statusCode><startTime>20120130</startTime><endTime>20120326</endTime><strengthValue>12</strengthValue><strengthUnits>h</strengthUnits></entryInfo><entryInfo><statusCode>completed</statusCode><startTime>20100412</startTime><endTime>20110822</endTime><strengthValue>8</strengthValue><strengthUnits>d</strengthUnits></entryInfo></string> |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

选项 2

create external table patient_info 
(
    code        string
   ,entryInfo   array<map<string,map<string,string>>>
)
row format serde 'com.ibm.spss.hive.serde2.xml.XmlSerDe'
with serdeproperties 
(
    "column.xpath.code"      = "/Medicationsinfo/code/text()"
   ,"column.xpath.entryInfo" = "/Medicationsinfo/entryInfo"
)
stored as
inputformat 'com.ibm.spss.hive.serde2.xml.XmlInputFormat'
outputformat 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
location '/user/hive/warehouse/patient_info'
tblproperties 
(
    "xmlinput.start" = "<Medicationsinfo"
   ,"xmlinput.end"   = "</Medicationsinfo>"
)
;

select * from patient_info
;

+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| patient_info.code |                                                                                                                                                                                   patient_info.entryinfo                                                                                                                                                                                   |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 10160-0           | [{"entryInfo":{"statusCode":"completed","startTime":"20110729","strengthUnits":"h","endTime":"20110822","strengthValue":"24"}},{"entryInfo":{"statusCode":"completed","startTime":"20120130","strengthUnits":"h","endTime":"20120326","strengthValue":"12"}},{"entryInfo":{"statusCode":"completed","startTime":"20100412","strengthUnits":"d","endTime":"20110822","strengthValue":"8"}}] |
| 10160-0           | [{"entryInfo":{"statusCode":"completed","startTime":"20110729","strengthUnits":"h","endTime":"20110822","strengthValue":"24"}},{"entryInfo":{"statusCode":"completed","startTime":"20120130","strengthUnits":"h","endTime":"20120326","strengthValue":"12"}},{"entryInfo":{"statusCode":"completed","startTime":"20100412","strengthUnits":"d","endTime":"20110822","strengthValue":"8"}}] |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

选项 3

create external table patient_info 
(
    code        string
   ,entryInfo   array<map<string,struct<statusCode:string,startTime:string,endTime:string,strengthValue:int,strengthUnits:string>>>
)
row format serde 'com.ibm.spss.hive.serde2.xml.XmlSerDe'
with serdeproperties 
(
    "column.xpath.code"      = "/Medicationsinfo/code/text()"
   ,"column.xpath.entryInfo" = "/Medicationsinfo/entryInfo"
)
stored as
inputformat 'com.ibm.spss.hive.serde2.xml.XmlInputFormat'
outputformat 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
location '/user/hive/warehouse/patient_info'
tblproperties 
(
    "xmlinput.start" = "<Medicationsinfo"
   ,"xmlinput.end"   = "</Medicationsinfo>"
)
;

select * from patient_info
;

+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| patient_info.code |                                                                                                                                                                                patient_info.entryinfo                                                                                                                                                                                |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 10160-0           | [{"entryInfo":{"statuscode":"completed","starttime":"20110729","endtime":"20110822","strengthvalue":24,"strengthunits":"h"}},{"entryInfo":{"statuscode":"completed","starttime":"20120130","endtime":"20120326","strengthvalue":12,"strengthunits":"h"}},{"entryInfo":{"statuscode":"completed","starttime":"20100412","endtime":"20110822","strengthvalue":8,"strengthunits":"d"}}] |
| 10160-0           | [{"entryInfo":{"statuscode":"completed","starttime":"20110729","endtime":"20110822","strengthvalue":24,"strengthunits":"h"}},{"entryInfo":{"statuscode":"completed","starttime":"20120130","endtime":"20120326","strengthvalue":12,"strengthunits":"h"}},{"entryInfo":{"statuscode":"completed","starttime":"20100412","endtime":"20110822","strengthvalue":8,"strengthunits":"d"}}] |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

分解选项 3

select  pi.code
       ,ei.i + 1    as i
       ,ei.entryInfo["entryInfo"].statusCode
       ,ei.entryInfo["entryInfo"].startTime
       ,ei.entryInfo["entryInfo"].endTime
       ,ei.entryInfo["entryInfo"].strengthValue
       ,ei.entryInfo["entryInfo"].strengthUnits

from    patient_info    pi
        lateral view  posexplode (entryInfo) ei as i,entryInfo
;

+---------+---+------------+-----------+----------+---------------+---------------+
| pi.code | i | statuscode | starttime | endtime  | strengthvalue | strengthunits |
+---------+---+------------+-----------+----------+---------------+---------------+
| 10160-0 | 1 | completed  | 20110729  | 20110822 | 24            | h             |
+---------+---+------------+-----------+----------+---------------+---------------+
| 10160-0 | 2 | completed  | 20120130  | 20120326 | 12            | h             |
+---------+---+------------+-----------+----------+---------------+---------------+
| 10160-0 | 3 | completed  | 20100412  | 20110822 | 8             | d             |
+---------+---+------------+-----------+----------+---------------+---------------+
| 10160-0 | 1 | completed  | 20110729  | 20110822 | 24            | h             |
+---------+---+------------+-----------+----------+---------------+---------------+
| 10160-0 | 2 | completed  | 20120130  | 20120326 | 12            | h             |
+---------+---+------------+-----------+----------+---------------+---------------+
| 10160-0 | 3 | completed  | 20100412  | 20110822 | 8             | d             |
+---------+---+------------+-----------+----------+---------------+---------------+

关于xml - 在 Pig 中使用 Hcatalog 加载配置单元表时出错，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/42227789/

文章推荐： r - Spark + rsparkling : Error while connecting to a cluster

文章推荐： http - 克服与不同端口的最大持久连接？

文章推荐： java - MapReduce 2 作业在本地运行而不是提交到 yarn

文章推荐： hadoop - 如何重命名配置单元中的所有分区列

javascript - 加载 gif 不等待 iFrame 加载？
我想要显示正在加载的 .gif，直到所有内容都已加载，包括嵌入的 iframe。但是，目前加载 gif 会在除 iframe 之外的所有内容都已加载后消失。我怎样才能让它等到 iframe 也加载完毕
javascript - AngularJS 加载 JSON 数据然后从中解析/加载 HTML
首先，这是我第一次接触 Angular。我想要实现的是，我有一个通知列表，我必须以某种方式限制 limitTo，因此元素被限制为三个，在我单击按钮后，其余的应该加载。我不明白该怎么做: 设置“ V
java - 瓦片不会随 map View 加载，但会随 fragment 加载
我正在尝试在我的设备上运行这个非常简单的应用程序(使用 map API V2)，并且出于某种原因尝试使用 MapView 时: 使用 java 文件: public class MainMap e
python - 通过 PyXLL 加载 scipy 时出现问题 - 是否有人成功通过 PyXLL 加载 Scipy？
我正在使用 Python 2.6、Excel 2007 Professional 和最新版本的 PyXLL。在 PyXLL 中加载具有 import scipy 抛出异常，模块未加载。有没有人能够在
unreal-engine4 - 虚幻引擎 4 : What is correct way to PAK files, 加载/挂载它们并在打包游戏中使用 AssetRegistry 加载 Assets ？
我想做这个: 创建并打包原始游戏。然后我想根据原始游戏中的蓝图创建具有新网格/声音/动画和蓝图的其他 PAK 文件。原始游戏不应该知道有关其他网格/动画/等的任何信息。因此，我需要在原始游戏中使用 A
你会几种读取/加载 properties配置文件方法
**摘要：**在java项目中经常会使用到配置文件，这里就介绍几种加载配置文件的方法。本文分享自华为云社区《【Java】读取/加载 properties配置文件的几种方法》，作者：Copy工程师。
class - 条件类导入/加载
在 Groovy 脚本中是否可以执行条件导入语句？ if (test){ import this.package.class } else { import that.package.
CUDA:加载/存储效率与全局内存指令重放之间的关系
我正在使用 NVidia 视觉分析器(来自 CUDA 5.0 beta 版本的基于 eclipse 的版本)和 Fermi 板，我不了解其中两个性能指标: 全局加载/存储效率表示实际内存事务数与请求事
加载 View 时angularjs清除历史记录
有没有办法在通过 routeProvider 加载特定 View 时清除 Angular JS 存储的历史记录？ ? 我正在使用 Angular 创建一个公共(public)安装，并且历史会积累很多，
initialization - 加载 Storyboard时首先调用什么方法？
使用 Xcode 4.2，在我的应用程序中， View 加载由 segue 事件触发。在 View Controller 中首先调用什么方法？ -(void) viewWillAppear:(BOO
Django JSONField转储/加载
我在某些Django模型中使用JSONField，并希望将此数据从Oracle迁移到Postgres。到目前为止，当使用Django的dumpdata和loaddata命令时，我仍然没有运气来保持J
Cocoa 加载 ViewNib
创建 Nib 时，我需要创建两种类型:WindowNib 或 ViewNib。我看到的区别是，窗口 Nib 有一个窗口和一个 View 。如何将 View Nib 加载到另一个窗口中？我是否必须创建
rust - 加载.env并使用辅助函数将其转换为一般结构
我想将多个env.variables转换为静态结构。我可以手动进行: Env { is_development: env::var("IS_DEVELOPMENT")
c++ - 加载/存储宽松原子变量和普通变量有什么区别？
正如我从一个测试用例中看到的:https://godbolt.org/z/K477q1 生成的程序集加载/存储原子松弛与普通变量相同:ldr 和 str 那么，宽松的原子变量和普通变量之间有什么区别吗
javascript - 加载/重定向到外部网站时如何添加加载屏幕？
我有一个重定向到外部网站的按钮/链接，但是外部网站需要一些时间来加载。所以我想添加一个加载屏幕，以便外部页面在显示之前完全加载。我无法控制外部网站，并且外部网站具有同源策略，因此我无法在 iFrame
bash - 加载.env的Dockerfile入口点bash文件在容器中不可见
我正在尝试为我的应用程序开发一个Dockerfile，该文件在初始化后加载大量环境变量。不知何故，当我稍后执行以下命令时，这些变量是不可用的: docker exec -it container_na
javascript - 加载 JavaScript
很难说出这里问的是什么。这个问题是含糊的、模糊的、不完整的、过于宽泛的或修辞性的，无法以目前的形式得到合理的回答。如需帮助澄清此问题以便重新打开它，visit the help center 。已关
JavaScript 加载 html
我刚刚遇到一个问题，我有一个带有一些不同选项的选择标签。现在我想检查用户选择了哪些选项。然后我想将一个新的 html 文件加载到该网站(取决于用户选中的选项)宽度 javascript，我该怎么做
黑莓 - 应用程序设置保存/加载
我知道两种保存/加载应用程序设置的方法: 使用PersistentStore 使用文件系统(存储，因为 SDCard 是可选的) 我想知道您使用应用程序设置的做法是什么？使用 PersistentS
Vulkan 加载 vkCreateDebugReportCallbackEXT
我开始使用 Vulkan 时偶然发现了我的第一个问题。尝试创建调试报告回调时(验证层和调试扩展在我的英特尔 hd vulkan 驱动程序上可用，至少它是这么说的)，它没有告诉我 vkCreateDeb

可可西里

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

xml - 在 Pig 中使用 Hcatalog 加载配置单元表时出错

选项 1

选项 2

选项 3

分解选项 3