scala - org.apache.spark.sql.AnalysisException : Can't extract value from probability

Question

我正在使用朴素贝叶斯算法对文章进行分类，并希望访问部分结果的“概率”列:

val Array(trainingDF, testDF) = rawDataDF.randomSplit(Array(0.6, 0.4))
    val ppline = MyUtil.createTrainPpline(rawDataDF)
    val model = ppline.fit(trainingDF)
    val testRes = model.transform(testDF)
    testRes.filter($"probability"(0).as[Double] === 1).show()

int 最后一行，中断

    Exception in thread "main" org.apache.spark.sql.AnalysisException: Can't extract value from probability#133;
            at org.apache.spark.sql.catalyst.expressions.ExtractValue$.apply(complexTypeExtractors.scala:73)
            at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$5.applyOrElse(Analyzer.scala:616)
            at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$5.applyOrElse(Analyzer.scala:608)
            at 

Answer 1

您始终可以获取底层 RDD 并过滤:

val filteredRes = results.rdd.filter(row => row.getAs[Vector]("probability")(0) == 1)

然后，如果需要，您可以将其转换回 dataframe:

val df = spark.createDataFrame(filteredRes, results.schema)