hadoop - 应用逻辑后，regex模式在pyspark中不起作用

Question

我有以下数据:

>>> df1.show()
+-----------------+--------------------+
|     corruptNames|       standardNames|
+-----------------+--------------------+
|Sid is (Good boy)|     Sid is Good Boy|
|    New York Life| New York Life In...|
+-----------------+--------------------+

因此，根据上述数据，我需要应用正则表达式，创建一个新列并获取第二列中的数据，即 standardNames。我尝试下面的代码:

spark.sql("select *, case when corruptNames rlike '[^a-zA-Z ()]+(?![^(]*))' or corruptNames rlike 'standardNames' then standardNames else 0 end as standard from temp1").show()  

它抛出以下错误:

pyspark.sql.utils.AnalysisException: "cannot resolve '`standardNames`' given input columns: [temp1.corruptNames, temp1. standardNames];

Answer 1

尝试不使用select sql的示例。我假设您要在正则表达式模式为true的情况下基于贪婪的名称创建一个称为standardNames的新列，否则“执行其他操作……”。

注意:您的模式无法编译，因为您需要使用\转义倒数第二个)。

pattern = '[^a-zA-Z ()]+(?![^(]*))' #this won't compile
pattern = r'[^a-zA-Z ()]+(?![^(]*\))' #this will

码

import pyspark.sql.functions as F

df_text = spark.createDataFrame([('Sid is (Good boy)',),('New York Life',)], ('corruptNames',))

pattern = r'[^a-zA-Z ()]+(?![^(]*\))'

df = (df_text.withColumn('standardNames', F.when(F.col('corruptNames').rlike(pattern), F.col('corruptNames'))
             .otherwise('Do something else'))
             .show()
     )

df.show()

#+-----------------+---------------------+
#|     corruptNames|        standardNames|
#+-----------------+---------------------+
#|Sid is (Good boy)|    Do something else|
#|    New York Life|    Do something else|
#+-----------------+---------------------+