java - Spark SQL RowFactory 返回空行

Question

我有一个具有这种模式的数据集:

{"user":"A10T7BS07XCWQ1","recommendations":[{"iID":34142,"rating":22.998692},{"iID":24963,"rating":22.678337},{"iID":47761,"rating":22.31455},{"iID":28694,"rating":21.269365},{"iID":36890,"rating":21.143366},{"iID":48522,"rating":20.678747},{"iID":20032,"rating":20.330639},{"iID":57315,"rating":20.099955},{"iID":18148,"rating":20.07064},{"iID":7321,"rating":19.754635}]}

我尝试通过这种方式对我的数据集进行平面映射:

    StructType struc = new StructType();
    struc.add("user", DataTypes.StringType, false);
    struc.add("item", DataTypes.IntegerType, false);
    struc.add("relevance", DataTypes.DoubleType, false);
    ExpressionEncoder<Row> encoder = RowEncoder.apply(struc);

    Dataset<Row> recomenderResult = userRecs.flatMap((FlatMapFunction<Row, Row>) row -> {
        String user = row.getString(0);
        List<Row> recsWithIntItemID = row.getList(1);
        Integer item;
        Double relevance;
        List<Row> rows = new ArrayList<>();

        for (Row rec : recsWithIntItemID) {

            item = rec.getInt(0);
            relevance = (double) rec.getFloat(1);
            System.out.println(user + " : " + item + " : " + relevance);

            Row newRow = RowFactory.create(user, item, relevance);
            rows.add(newRow);
        }
        System.out.println("++++++++++++++++++++++++++++++++");
        return rows.iterator();
    }, encoder);

    recomenderResult.write().json("temp2");
    recomenderResult.show();

系统输出如下:

...

A1049B0RS95K7B : 24708 : 17.146669387817383
A1049B0RS95K7B : 2825 : 16.809375762939453
A1049B0RS95K7B : 36503 : 16.758258819580078
++++++++++++++++++++++++++++++++

...

但是 Row 实例是空的，show() 方法给出这样的输出:

++
||
++
||
||

我不知道为什么我的结果数据集是空的。我已经看过这个网站上与我的问题相关的所有主题并使用了谷歌，但我还没有找到我的问题的解决方案。有人可以帮助我吗？

Answer 1

这是非常愚蠢的错误 :( 简单的答案，错误就在这里:

    StructType struc = new StructType();
    struc = struc.add("user", DataTypes.StringType, false);
    struc = struc.add("item", DataTypes.IntegerType, false);
    struc = struc.add("relevance", DataTypes.DoubleType, false);
    ExpressionEncoder<Row> encoder = RowEncoder.apply(struc);

花了我2天1夜...