java - Spark SQL Java - 无法创建嵌套 Row 对象

Question

这是我尝试使用 Spark SQL 实现的最终架构

|-- references: array (nullable = true)                                                                                                                                                                                                                                        
 |-- element: struct (containsNull = true)                                                                                                                                                                                                                                 
    |-- name: string (nullable = true)                                                                                                                                                                                                                                   
    |-- type: string (nullable = true)                                                                                                                                                                                                                                         
    |-- url: string (nullable = true) 

我正在尝试在 Parquet 中插入数据，但无法创建与上述架构匹配的嵌套 JSON Row 对象。

以下是我尝试过但不起作用的方法 -

Tried inserting the data as - Object[] references = new Object[]{"1", "2", "3"}

Tried this Object[] references - new Object[0] (Only this works)

Tried this Object[] references - new Object[]{new Object[]{"1", "2", "3"}}

然后我将其返回为

Row.createFactory(references)

我尝试作为 Row 对象返回的位置

我需要使用 Spark SQL Java 创建架构的帮助。我无法在网上找到任何解决方案。

Answer 1

看起来数组列表可用，函数“array”和“struct”可用于创建所需的架构:

    List<Row> data = Lists.newArrayList(
            RowFactory.create(new String[]{"1", "2", "3"}),
            RowFactory.create(new String[]{"4", "5", "6"})
    );

    StructType schema = DataTypes.createStructType(
            new StructField[]{
                    DataTypes.createStructField("name", DataTypes.StringType, true),
                    DataTypes.createStructField("type", DataTypes.StringType, true),
                    DataTypes.createStructField("url", DataTypes.StringType, true),
            });
    Dataset<Row> plain = spark().createDataFrame(data, schema);

    Dataset<Row> result = plain.withColumn("references",
            array(
                    struct(col("name"), col("type"), col("url")))).
            select("references");
    result.show(false);
    result.printSchema();

输出是:

+----------+
|references|
+----------+
|[[1,2,3]] |
|[[4,5,6]] |
+----------+

root
 |-- references: array (nullable = false)
 |    |-- element: struct (containsNull = false)
 |    |    |-- name: string (nullable = true)
 |    |    |-- type: string (nullable = true)
 |    |    |-- url: string (nullable = true)