pyspark - 如何将 FPGrowth 项集限制为 2 或 3

Question

我正在使用 jupyter notebook 在 python3.6 中使用 pyspark 运行 FPGrowth 算法。当我试图保存生成的规则的关联规则输出是巨大的。所以我想限制结果的数量。这是我试过的代码。我还更改了 spark 上下文参数。

Maximum Pattern Length fpGrowth (Apache) PySpark

from pyspark.sql.functions import col, size
from pyspark.ml.fpm import FPGrowth
from pyspark.sql import Row
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
from pyspark import SparkConf

conf = SparkConf().setAppName("App")
conf = (conf.setMaster('local[*]')
        .set('spark.executor.memory', '100G')
        .set('spark.driver.memory', '400G')
        .set('spark.driver.maxResultSize', '200G'))
sc = SparkContext.getOrCreate(conf=conf)
spark = SparkSession(sc)
R = Row('ID', 'items')
df=spark.createDataFrame([R(i, x) for i, x in enumerate(lol)])
fpGrowth = FPGrowth(itemsCol="items", minSupport=0.7, minConfidence=0.9)

model = fpGrowth.fit(df)
ar=model.associationRules.where(size(col('antecedent')) == 2).where(size(col('cosequent')) == 1)

ar.cache()
ar.toPandas().to_csv('output.csv')

     It gives an error


   TypeError Traceback (most recent call last)
   <ipython-input-1-f90c7a9f11ae> in <module>

   ---> 73 ar=model.associationRules.where(size(col('antecedent')) == 
  2).where(size(col('consequent')) == 1)
   TypeError: 'str' object is not callable

谁能帮我解决这个问题。

这里 lol 是交易列表的列表:[['a','b'],['c','a','e']...]

python :3.6.5派斯帕克Windows 10

Answer 1

根据上述讨论和点击此链接，它帮助我解决了问题。

'str' object is not callable TypeError

   import pyspark.sql.functions as func
   model.associationRules.where(func.size(func.col('antecedent')) == 1).where(func.size(func.col('consequent')) == 1).show()