machine-learning - 使用 pyspark.ml 作为数据帧的随机森林

Question

我正在尝试使用 pyspark.ml 库为数据帧构建随机森林分类器(不是 RDD 的 mllib)。我必须使用文档中给出的管道吗？我只是想构建一个简单的模型，

rf = RandomForestClassifier(labelCol = labs, featuresCol = rawdata) 

我遇到以下错误

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/spark/python/pyspark/__init__.py", line 104, in wrapper
    return func(self, **kwargs)
  File "/usr/lib/spark/python/pyspark/ml/classification.py", line 910, in __init
__
    self.setParams(**kwargs)
  File "/usr/lib/spark/python/pyspark/__init__.py", line 104, in wrapper
    return func(self, **kwargs)
  File "/usr/lib/spark/python/pyspark/ml/classification.py", line 928, in setPar
ams
    return self._set(**kwargs)
  File "/usr/lib/spark/python/pyspark/ml/param/__init__.py", line 421, in _set
    raise TypeError('Invalid param value given for param "%s". %s' % (p.name, e)
)
TypeError: Invalid param value given for param "labelCol". Could not convert <cl
ass 'pyspark.sql.dataframe.DataFrame'> to string type

我的标签示例

+---+
| _2|
+---+
|0.0|
|1.0|
|0.0|
|0.0|
|0.0|
|0.0|
|1.0|
|1.0|
|1.0|
|0.0|
|0.0|
|0.0|
|0.0|
|0.0|
|0.0|
|0.0|
|0.0|
|0.0|
|1.0|
|1.0|
+---+

我的数据类似，有 180 列。

Answer 1

Spark 数据帧的使用方式与 Spark ML 不同；您的所有特征都需要是单列中的向量，通常(但不一定)名为features。另外，labelcol=labs 意味着您的标签需要位于名为 labs 的列中，而不是 _2。

下面是一个带有玩具数据的示例:

spark.version
# u'2.2.0'

from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.linalg import Vectors
df = sqlContext.createDataFrame([
     (0.0, Vectors.dense(0.0, 1.0)),
     (1.0, Vectors.dense(1.0, 0.0))], 
     ["label", "features"])

df.show() # notice there are only 2 columns, and 'features' is a 2-d vector
# +-----+---------+ 
# |label| features|
# +-----+---------+ 
# |  0.0|[0.0,1.0]|
# |  1.0|[1.0,0.0]|
# +-----+---------+

rf = RandomForestClassifier(numTrees=3, maxDepth=2, labelCol="label", seed=42)
rf_model = rf.fit(df)

This answer of mine可能对如何将数据转换为所需的格式有所帮助。