java - 配置单元/ElasticMapreduce : How bring JsonSerDe to ignore malformed JSON?

Question

我对 Hive 和 ElasticMapreduce 还很陌生，目前我遇到了一个特定的问题。在包含数十亿行 JSON 对象的表上运行 Hive 语句时，只要其中一行是无效的/格式错误的 JSON，MapReduce 作业就会崩溃。

异常:

java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"ip":"39488130","cdate":"2012-08-09","cdate_ts":"2012-08-09 17:06:41","country":"SA","city":"Riyadh","mid":"6666276582211270592","osversion":"5.1.
1
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:441)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:377)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"ip":"39488130","cdate":"2012-08-09","cdate_ts":"2012-08-09 17:06:41","country":"SA","city":"Riyadh","mid":"6666276582211270592","osversion":"5.1.1
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
... 8 more
Caused by: com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated string near
at com.google.gson.Streams.parse(Streams.java:51)
at com.google.gson.JsonParser.parse(JsonParser.java:83)
at com.google.gson.JsonParser.parse(JsonParser.java:58)
at com.google.gson.JsonParser.parse(JsonParser.java:44)
at com.amazon.elasticmapreduce.JsonSerde.deserialize(Unknown Source)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:510)
... 9 more
Caused by: com.google.gson.stream.MalformedJsonException: Unterminated string near
at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1110)
at com.google.gson.stream.JsonReader.nextString(JsonReader.java:967)
at com.google.gson.stream.JsonReader.nextValue(JsonReader.java:802)
at com.google.gson.stream.JsonReader.objectValue(JsonReader.java:782)
at com.google.gson.stream.JsonReader.quickPeek(JsonReader.java:377)
at com.google.gson.stream.JsonReader.peek(JsonReader.java:340)
at com.google.gson.Streams.parseRecursive(Streams.java:60)
at com.google.gson.Streams.parseRecursive(Streams.java:83)
at com.google.gson.Streams.parse(Streams.java:40)
... 14 more

我这样创建表格:

CREATE EXTERNAL TABLE IF NOT EXISTS table1 (
column1 string,
column2 string
)
PARTITIONED BY (year string, month string)
ROW FORMAT SERDE 'com.amazon.elasticmapreduce.JsonSerde'
WITH SERDEPROPERTIES ('paths'='c1, c2')
LOCATION 's3://mybucket/table1';

我该怎么做才能防止崩溃？忽略格式错误的 JSON 对象/字符串会很好，因为它是格式错误的十亿分之一。

提前感谢您的帮助。最好的，萨沙

Answer 1

通过更改行格式中使用的类并添加“格式错误”属性，您可以使您的表创建工作在格式错误的 JSON 上:

ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES ("ignore.malformed.json" = "true")
LOCATION ...

使用“hive-site.xml”中的“hive.aux.jars.path”属性或“ADD JAR”Hive 指令包含 JAR。您可以找到 JAR here , 或者从 this source 编译它.