apache-spark - 列特征必须是 org.apache.spark.ml.linalg.VectorUDT 类型

Question

我想在 pyspark (spark 2.1.1) 中运行此代码:

from pyspark.ml.feature import PCA

bankPCA = PCA(k=3, inputCol="features", outputCol="pcaFeatures") 
pcaModel = bankPCA.fit(bankDf)    
pcaResult = pcaModel.transform(bankDF).select("label", "pcaFeatures")    
pcaResult.show(truncate= false)

但我收到此错误:

requirement failed: Column features must be of type org.apache.spark.ml.linalg.Vect orUDT@3bfc3ba7 but was actually org.apache.spark.mllib.linalg.VectorUDT@f71b0bce.

Answer 1

您可以找到的示例 here :

from pyspark.ml.feature import PCA
from pyspark.ml.linalg import Vectors

data = [(Vectors.sparse(5, [(1, 1.0), (3, 7.0)]),),
    (Vectors.dense([2.0, 0.0, 3.0, 4.0, 5.0]),),
    (Vectors.dense([4.0, 0.0, 0.0, 6.0, 7.0]),)]
df = spark.createDataFrame(data, ["features"])

pca = PCA(k=3, inputCol="features", outputCol="pcaFeatures")
model = pca.fit(df)

... other code ...

正如您在上面看到的，df 是一个数据框，其中包含导入的 Vectors.sparse() 和 Vectors.dense() 来自 pyspark.ml.linalg .

可能，你的 bankDf 包含导入的矢量 来自 pyspark.mllib.linalg .

因此，您必须设置导入数据框中的 Vectors

from pyspark.ml.linalg import Vectors 

代替:

from pyspark.mllib.linalg import Vectors

也许你会发现这个有趣的 stackoverflow question .