java - 如何从现有 Dataframe 创建 Dataframe 并将特定字段设为 Struct 类型？

Question

我需要从现有的 DataFrame 创建一个 DataFrame，我还需要在其中更改架构。

我有一个像这样的 DataFrame:

+-----------+----------+-------------+
|Id         |Position   |playerName  |
+-----------+-----------+------------+
|10125      |Forward    |Messi       |
|10126      |Forward    |Ronaldo     |
|10127      |Midfield   |Xavi        |
|10128      |Midfield   |Neymar      |

我是使用下面给出的案例类创建的:

case class caseClass (
                       Id: Int = "",
                       Position : String = "" ,
                       playerName : String = "" 
                     )

现在我需要在 Struct 类型下创建 Playername 和 position。

即，

我需要创建另一个带有模式的 DataFrame，

root

|-- Id: int (nullable = true)

|-- playerDetails: struct (nullable = true)

| |--playername: string (nullable = true)

| |--Position: string (nullable = true)

我通过引用链接做了以下代码来创建一个新的数据框 https://medium.com/@mrpowers/adding-structtype-columns-to-spark-dataframes-b44125409803

我的模式是

  List(
    StructField("Id", IntegerType, true),
    StructField("Position",StringType, true),
    StructField("playerName", StringType,true)
)

我试过下面的代码

  spark.sparkContext.parallelize(data),
  myschema
)

但我做不到。

我看到了类似的问题 Change schema of existing dataframe但我无法理解解决方案。

有没有直接在case类中实现StructType的解决方案？所以我认为我不需要创建自己的架构来创建结构类型值。

Answer 1

可以使用函数“struct”:

// data
val playersDF = Seq(
  (10125, "Forward", "Messi"),
  (10126, "Forward", "Ronaldo"),
  (10127, "Midfield", "Xavi"),
  (10128, "Midfield", "Neymar")
).toDF("Id", "Position", "playerName")

// action
val playersStructuredDF = playersDF.select($"Id", struct("playerName", "Position").as("playerDetails"))
// display
playersStructuredDF.printSchema()
playersStructuredDF.show(false)

输出:

root
 |-- Id: integer (nullable = false)
 |-- playerDetails: struct (nullable = false)
 |    |-- playerName: string (nullable = true)
 |    |-- Position: string (nullable = true)

+-----+------------------+
|Id   |playerDetails     |
+-----+------------------+
|10125|[Messi, Forward]  |
|10126|[Ronaldo, Forward]|
|10127|[Xavi, Midfield]  |
|10128|[Neymar, Midfield]|
+-----+------------------+