python - 如何使用 pyspark 从 CSV 设置 Spark 中的 parquet 中正确的数据类型

Question

我有一个类似于以下内容的 csv 文件:

39813458,13451345,14513,SomeText,344564,Some other text,328984,"[{""field_int_one"":""16784832510"",""second_int_field"":""84017"",""third_int_field"":""245"",""some_timestamp_one"":""2018-04-17T23:54:34.000Z"",""some_other_timestamp"":""2018-03-03T15:34:04.000Z"",""one_more_int_field"":0,},{""field_int_one"":""18447548326"",""second_int_field"":""04965"",""third_int_field"":""679"",""some_timestamp_one"":""2018-02-06T03:39:12.000Z"",""some_other_timestamp"":""2018-03-01T09:19:12.000Z"",""one_more_int_field"":0}]"

我将其转换为 Parquet

from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql.session import SparkSession
sc = SparkContext('local')
spark = SparkSession(sc)

if __name__ == "__main__":
    sqlContext = SQLContext(sc)

    schema = StructType([
              StructField("first_int", IntegerType(), True),
              StructField("second_int", IntegerType(), True),
              StructField("third_int", IntegerType(), True),
              StructField("first_string_field", StringType(), True),
              StructField("fourth_int", IntegerType(), True),
              StructField("second_string_field", StringType(), True),
              StructField("last_int_field", StringType(), True),
              StructField("json_field", StringType(), True)])

    rdd = spark.read.schema(schema).csv("source_file.csv")
    rdd.write.parquet('parquet_output')

它可以工作并转换它，但是如果您在查询它时执行.printSchema，它显然会将其定义打印为字符串。如何正确地将最后一个字段声明为 Json？

Answer 1

我认为嵌套的ArrayType适用于这种类型的模式

schema = StructType([
          StructField("first_int", IntegerType(), True),
          StructField("second_int", IntegerType(), True),
          StructField("third_int", IntegerType(), True),
          StructField("first_string_field", StringType(), True),
          StructField("fourth_int", IntegerType(), True),
          StructField("second_string_field", StringType(), True),
          StructField("last_int_field", StringType(), True),
          StructField("json_field", ArrayType(
                StructType() \
                   .add("field_int_one", IntegerType()) \
                   .add("field_string_one", StringType()) \
                   .addMoreFieldsHere), 
          True)])