pyspark - pyspark 数据框的区分大小写的列删除操作？

Question

从一些简短的测试来看，pyspark 数据帧的列删除功能似乎不区分大小写，例如。

from pyspark.sql import SparkSession
from pyspark.sql.functions import *
import sys

sparkSession = SparkSession.builder.appName("my-session").getOrCreate()

dff = sparkSession.createDataFrame([(10,123), (14,456), (16,678)], ["age", "AGE"])

>>> dff.show()
+---+---+
|age|AGE|
+---+---+
| 10|123|
| 14|456|
| 16|678|
+---+---+

>>> dff.drop("AGE")
DataFrame[]

>>> dff_dropped = dff.drop("AGE")
>>> dff_dropped.show()
++
||
++
||
||
||
++

"""
What I'd like to see here is:
+---+
|age|
+---+
| 10|
| 14|
| 16|
+---+
"""

有没有办法以区分大小写的方式删除数据框列？ (在 spark JIRA discussions 中看到了一些与此相关的评论，但正在寻找仅适用于 drop() 操作的临时方式(不是全局/持久设置)).

Answer 1

#Add this before using drop
sqlContext.sql("set spark.sql.caseSensitive=true")

You need to set casesensitivity as true if you have two columns having same name