json - 解析json时由于输入结束而没有要映射的内容

Question

我在 Spark 中使用 play JSON 库工具解析 JSON 数据，并收到以下错误消息。有没有人对这个错误的可能原因有任何线索？如果这是由于错误的 JSON 记录造成的，我如何识别错误的记录？谢谢!

这是我用来解析 JSON 数据的主要脚本:

import play.api.libs.json._
val jsonData = distdata.map(line => Json.parse(line)) //line 194 of script parseJson_v14.scala
val filteredData = jsonData.map(json => (json \ "QueryStringParameters" \ "pr").asOpt[String].orNull).countByValue()

变量distdata是文本格式JSON数据的rdd，变量jsonData是JsValue数据的rdd。由于Spark转换是惰性的，直到执行第二条命令创建变量filteredData时错误才跳出，根据错误信息，错误来自我创建变量jsonData的第一条命令。
[2017-03-29 14:55:39.616]-[Logging$class.logWarning]-[WARN]: Lost task 42.0 in stage 1.0 (TID 90, 10.119.126.114): com.fasterxml.jackson.databind.JsonMappingException: No content to map due to end-of-input
at [Source: ; line: 1, column: 1]
at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148)
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3110)
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3024)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1652)
at play.api.libs.json.jackson.JacksonJson$.parseJsValue(JacksonJson.scala:226)
at play.api.libs.json.Json$.parse(Json.scala:21)
at parseJson_v14$$anonfun$1$$anonfun$3$$anonfun$apply$1.apply(parseJson_v14.scala:194)
at parseJson_v14$$anonfun$1$$anonfun$3$$anonfun$apply$1.apply(parseJson_v14.scala:194)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:389)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$6.apply$mcV$sp(PairRDDFunctions.scala:1197)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$6.apply(PairRDDFunctions.scala:1197)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$6.apply(PairRDDFunctions.scala:1197)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1250)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1205)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1185)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


Answer 1

检查 distdata 中是否没有空行并且您在一行中拥有所有 JSON 对象，例如

{"id":"121", "name":"robot 1"}
{"id":"122", "name":"robot 2"}

对面

{"id":"121", "name":
"robot 1"}
{"id":"122", "name":
"robot 2"}