python - pyspark - LinearRegression.load() 抛出 NoSuchMethodException

Question

在尝试加载线性回归模型时出现以下错误:

Traceback (most recent call last):
  File "server.py", line 5, in <module>
    linReg = Model()
  File "/home/pyspark/Desktop/building_py_rec/lin_reg/ml_algo/model.py", line 23, in __init__
    self.model = LinearRegression.load('model_lin_reg')
  File "/home/pyspark/spark-2.1.0-bin-hadoop2.7/python/pyspark/ml/util.py", line 252, in load
    return cls.read().load(path)
  File "/home/pyspark/spark-2.1.0-bin-hadoop2.7/python/pyspark/ml/util.py", line 193, in load
    java_obj = self._jread.load(path)
  File "/home/pyspark/spark-2.1.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/home/pyspark/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/home/pyspark/spark-2.1.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o24.load.
: java.lang.NoSuchMethodException: org.apache.spark.ml.regression.LinearRegressionModel.<init>(java.lang.String)
    at java.lang.Class.getConstructor0(Class.java:3082)
    at java.lang.Class.getConstructor(Class.java:1825)
    at org.apache.spark.ml.util.DefaultParamsReader.load(ReadWrite.scala:325)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:280)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:748)

相关代码如下:

import findspark;
findspark.init()

import pyspark;
from pyspark.sql import SparkSession;
from pyspark.ml.regression import LinearRegression;
from pyspark.ml.feature import VectorAssembler;
from pyspark.sql.types import *;

X_Cols = ["Freq_Hz", "AoA_Deg", "Chord_m", "V_inf_mps", "displ_thick_m"]


class Model:

    spark = None;
    model = None;
    airfoil_assembler = None;

    def __init__(self):
        print('[Model] Creating spark session...');
        self.spark = SparkSession.builder.appName('lin_reg_reader').getOrCreate();
        print('[Model] Loading model...');      
        self.model = LinearRegression.load('model_lin_reg')
        print('[Model] Loading complete...');
        self.airfoil_assembler = VectorAssembler(inputCols=X_Cols, outputCol='features')
        return ;

    def _getSchema(self):
        schema = StructType({
            StructField("Freq_Hz", IntegerType(), False),
            StructField("AoA_Deg", IntegerType(), False),
            StructField("Chord_m", DoubleType(), False),
            StructField("V_inf_mps", DoubleType(), False),
            StructField("displ_thick_m", DoubleType(), False),
        });
        return schema

    def _prepare_df(self):
        schema = self._getSchema();
        return df;

    def assemble(self, tup):
        schema = self._getSchema();
        df = self.spark.createDataFrame(tup, schema)
        assembled_vector = self.airfoil_assembler.transform(df);
        return assembled_vector;

    def predict(self, airfoil):
        assembled_vector = self.assemble(tup=airfoil)
        return self.model.predict(assembled_vector)

注意

• 星火版本:2.1.0
• 引用这个link表明它们应该是一个可用的load函数。

Answer 1

您使用了错误的类。要加载模型，请使用 LinearRegressionModel:

from pyspark.ml.regression import LinearRegressionModel

LinearRegressionModel.load('model_lin_reg')