scala - 在 Spark RDD Println Error 中，如何显示 [Ljava.lang.String;@8e6606 等数据

Question

我在使用 Scala 时遇到 Apache Spark 的一个问题。我正在尝试创建一个 Spark 应用程序，它根据用户输入打印一个 RDD。输入数据是这样的:

List("aaaa","aaaa","dfddf","aaaa","aaaa","dfddf","aaaa","aaaa","dfddf","aaaa","aaaa","dfddf","aaaa","aaaa","dfddf")

代码是这样的:

    val wSchemaString = "col1 col2 col3 col4";
    val wSchema = StructType(wSchemaString.split(" ").map(fieldName =>StructField(fieldName, StringType, true)));
    val wRow = sc.parallelize(wInput.map(_.split(",")));
    val wRowRDD = wRow.map(x=>{
        val wNum = wSchemaString.split(" ").size;
        var out = new Array[String](wNum+1);
        for(i <- 0 to wNum)
        {
            out(i) = x(i);
        }
        Row(out);
    });

wRowRDD.collect.foreach(println);
...

结果是这样的:

27 [[Ljava.lang.String;@5f1ec010]
28 [[Ljava.lang.String;@5bd38b39]
29 [[Ljava.lang.String;@5d6b1c05]
30 [[Ljava.lang.String;@7ea6404c]
31 [[Ljava.lang.String;@75447fda]
32 [[Ljava.lang.String;@6425fd5b]
33 [[Ljava.lang.String;@7a1c94ba]
34 [[Ljava.lang.String;@6a687df7]
35 [[Ljava.lang.String;@722619b4]
36 [[Ljava.lang.String;@117d1979]
37 [[Ljava.lang.String;@304a45f4]
38 [[Ljava.lang.String;@5c36aef0]
39 [[Ljava.lang.String;@a173ddc]
40 [[Ljava.lang.String;@7bde3bb0]
41 [[Ljava.lang.String;@3b20df58]
42 [[Ljava.lang.String;@981f1f2

但是，我想要的是:

"aaaa","aaaa","dfddf","aaaa"
"aaaa","dfddf","aaaa","aaaa"
"dfddf","aaaa","aaaa","dfddf"    
"aaaa","aaaa","dfddf","aaaa"

Answer 1

Row 是一个 Array[Any]，因此在 prinln 函数中生成的字符串是 Array 的 toString 方法生成的字符串。

要获得这些输出，您必须执行以下操作:

val wSchemaString = "col1 col2 col3 col4";
val wSchema = StructType(wSchemaString.split(" ").map(fieldName =>StructField(fieldName, StringType, true)));
val wRow = sc.parallelize(wInput.map(_.split(",")));
val wRowRDD = wRow.map(x=>{
    val wNum = wSchemaString.split(" ").size;
    var out = new Array[String](wNum+1);
    for(i <- 0 to wNum)
    {
        out(i) = x(i);
    }
    Row(out);
});

wRowRDD.collect.foreach(row => println(row.mkString(",")));

希望对你有用