pyspark - 列类型中的 Spark 错误

Question

我有一个名为“SupplierId”的数据框列，类型为字符串，有很多数字，但也有一些字符链。(例如:['123','456','789',......,'abc'])。我使用

将此列格式化为字符串

from pyspark.sql.types import StringType
df=df.withColumn('SupplierId',df['SupplierId'].cast(StringType())

所以我检查它是否被视为一个字符串，使用:

df.printSchema()

我得到:

root
 |-- SupplierId: string (nullable = true)

但是当我尝试转换为 Pandas，或者只是使用 df.collect() 时，我收到以下错误:

An error occurred while calling o516.collectToPython. : org.apache.spark.SparkException: Job aborted due to stage failure:

Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 11, servername.ops.somecompany.local, executor 3): 
ava.lang.RuntimeException: Error while encoding: java.lang.RuntimeException:
Exception parsing 'CPD160001' into a IntegerType$ for column "SupplierId":
Unable to deserialize value using com.somecompany.spark.parsers.text.converters.IntegerConverter.
The value being deserialized was: CPD160001 

所以 Spark 似乎将此列的值视为整数。我试过使用 UDF 强制转换为 python 的字符串，但它仍然不起作用。您知道什么会导致此错误吗？

Answer 1

请分享您的实际数据样本，因为您的问题无法用玩具数据重现:

spark.version
#  u'2.2.0'

from pyspark.sql import Row
df = spark.createDataFrame([Row(1, 2, '3'),
                            Row(4, 5, 'a'),
                            Row(7, 8, '9')],
                            ['x1', 'x2', 'id'])  

df.printSchema()
# root
#  |-- x1: long (nullable = true)
#  |-- x2: long (nullable = true) 
#  |-- id: string (nullable = true) 

df.collect()
# [Row(x1=1, x2=2, id=u'3'), Row(x1=4, x2=5, id=u'a'), Row(x1=7, x2=8, id=u'9')]

import pandas as pd
df_pandas = df.toPandas()
df_pandas
#   x1 x2 id 
# 0  1  2  3
# 1  4  5  a
# 2  7  8  9