scala - 如何在Spark Scala中的变量中存储嵌套JSON的架构

Question

我可以使用以下方法获取嵌套JSON的结构:df.schema
我明白了:

StructType(StructField(CreatedBy,StringType,true), StructField(Description,StringType,true), StructField(ExpirationDate,LongType,true), StructField(ID,StringType,true), StructField(Name,StringType,true), StructField(Package,StringType,true), StructField(PackageDependencies,StructType(StructField(dataShape,StructType(StructField(fieldDefinitions,StructType(StructField(Name................

我只想获取具有数据类型的列。如果在Spark Scala上工作的人可以向我建议这一点，那就太好了。

谢谢

Answer 1

我认为您正在使用架构作为输入来寻找df.dtypes。
您可以使用一个空的数据框得到它。看一下这个

scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._

scala> val typ = StructType(Seq(
     | StructField("CreatedBy",StringType,true),
     | StructField("Description",StringType,true),
     | StructField("ExpirationDate",LongType,true),
     | StructField("ID",StringType,true),
     | StructField("Name",StringType,true),
     | StructField("Package",StringType,true)));
typ: org.apache.spark.sql.types.StructType = StructType(StructField(CreatedBy,StringType,true), StructField(Description,StringType,true), StructField(ExpirationDate,LongType,true), StructField(ID,StringType,true), StructField(Name,StringType,true), StructField(Package,StringType,true))

scala> import org.apache.spark.sql.Row
import org.apache.spark.sql.Row

scala> val df = spark.createDataFrame(spark.sparkContext.emptyRDD[Row],typ)
df: org.apache.spark.sql.DataFrame = [CreatedBy: string, Description: string ... 4 more fields]

scala> df.dtypes
res8: Array[(String, String)] = Array((CreatedBy,StringType), (Description,StringType), (ExpirationDate,LongType), (ID,StringType), (Name,StringType), (Package,StringType))

scala> df.dtypes.foreach(println)
(CreatedBy,StringType)
(Description,StringType)
(ExpirationDate,LongType)
(ID,StringType)
(Name,StringType)
(Package,StringType)

scala>