scala - 如何在 spark-sql 中使用 "not rlike"？

Question

rlike工作正常，但 not rlike抛出错误:

scala> sqlContext.sql("select * from T where columnB rlike '^[0-9]*$'").collect()
res42: Array[org.apache.spark.sql.Row] = Array([412,0], [0,25], [412,25], [0,25])

scala> sqlContext.sql("select * from T where columnB not rlike '^[0-9]*$'").collect()
java.lang.RuntimeException: [1.35] failure: ``in'' expected but `rlike' found


val df = sc.parallelize(Seq(
  (412, 0),
  (0, 25), 
  (412, 25), 
  (0, 25)
)).toDF("columnA", "columnB")

或者是问题的延续 https://issues.apache.org/jira/browse/SPARK-4207 ?

Answer 1

在 PySpark 中执行此操作的简洁方法是:

df.filter(~df.column.rlike(pattern))