Python 在使用函数时无法接受输入

Question

我正在研究 Kaggle 上托管的房价问题。在构建模型时，我认为在测试集上重用我在训练数据集上使用的一些代码是有意义的，因此我将执行相互操作的代码放入一个函数定义中。在此函数中，我处理缺失值并使用其返回来执行单热编码并将其用于随机森林回归。但是，它引发以下错误:

Traceback (most recent call last):
  File "C:/Users/security/Downloads/AP/Boston-Kaggle/Model.py", line 56, in <module>
    sel.fit(x_train, y_train)
  File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\feature_selection\from_model.py", line 196, in fit
    self.estimator_.fit(X, y, **fit_params)
  File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\ensemble\forest.py", line 249, in fit
    X = check_array(X, accept_sparse="csc", dtype=DTYPE)
  File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\utils\validation.py", line 542, in check_array
    allow_nan=force_all_finite == 'allow-nan')
  File "C:\Users\security\AppData\Roaming\Python\Python37\site-packages\sklearn\utils\validation.py", line 56, in _assert_all_finite
    raise ValueError(msg_err.format(type_err, X.dtype))
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

当我使用相同的代码而不将其组织成函数时，没有遇到这个问题。 def feature_selection_and_engineering(df) 是有问题的函数。以下是我的全部代码:

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor

train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")

def feature_selection_and_engineering(df):
    # Creating a series of how many NaN's are in each column
    nan_counts = df.isna().sum()

    # Creating a template list
    nan_columns = []

    # Iterating over the series and if the value is more than 0 (i.e there are some NaN's present)
    for i in range(0, len(nan_counts)):
        if nan_counts[i] > 0:
            nan_columns.append(df.columns[i])

    # Iterating through all the columns which are known to have NaN's
    for i in nan_columns:
        if df[nan_columns][i].dtypes == 'float64':
            df[i] = df[i].fillna(df[i].mean())
        elif df[nan_columns][i].dtypes == 'object':
            df[i] = df[i].fillna('XX')

    # Creating a template list
    categorical_columns = []

    # Iterating across all the columns,
    # checking if they're of the object datatype and if they are, appending them to the categorical list
    for i in range(0, len(df.dtypes)):
        if df.dtypes[i] == 'object':
            categorical_columns.append(df.columns[i])

    return categorical_columns

# take one-hot encoding
OHE_sdf = pd.get_dummies(feature_selection_and_engineering(train))

# drop the old categorical column from original df
train.drop(columns = feature_selection_and_engineering(train), axis = 1, inplace = True)

# attach one-hot encoded columns to original data frame
train = pd.concat([train, OHE_sdf], axis = 1, ignore_index = False)

# Dividing the training dataset into train/test sets with the test size being 20% of the overall dataset.
x_train, x_test, y_train, y_test = train_test_split(train, train['SalePrice'], test_size = 0.2, random_state = 42)

randomForestRegressor = RandomForestRegressor(n_estimators=1000)

# Invoking the Random Forest Classifier with a 1.25x the mean threshold to select correlating features
sel = SelectFromModel(RandomForestClassifier(n_estimators = 100), threshold = '1.25*mean')
sel.fit(x_train, y_train)

selected = sel.get_support()

# linearRegression.fit(x_train, y_train)
randomForestRegressor.fit(x_train, y_train)

# Assigning the accuracy of the model to the variable "accuracy"
accuracy = randomForestRegressor.score(x_train, y_train)

# Predicting for the data in the test set
predictions = randomForestRegressor.predict(feature_selection_and_engineering(test))

# Writing the predictions to a new CSV file
submission = pd.DataFrame({'Id': test['PassengerId'], 'SalePrice': predictions})
filename = 'Boston-Submission.csv'
submission.to_csv(filename, index=False)

print(accuracy*100, "%")

新错误:

    Traceback (most recent call last):
  File "/home/onur/Documents/Boston-Kaggle/Model.py", line 76, in <module>
    x_train, encoder = feature_selection_and_engineering(x_train)
  File "/home/onur/Documents/Boston-Kaggle/Model.py", line 57, in feature_selection_and_engineering
    encoder = train_one_hot_encoder(df, categorical_columns)
  File "/home/onur/Documents/Boston-Kaggle/Model.py", line 30, in train_one_hot_encoder
    return enc.fit(categorical_df)
  File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 493, in fit
    self._fit(X, handle_unknown=self.handle_unknown)
  File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 80, in _fit
    X_list, n_samples, n_features = self._check_X(X)
  File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py", line 67, in _check_X
    force_all_finite=needs_validation)
  File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/utils/validation.py", line 542, in check_array
    allow_nan=force_all_finite == 'allow-nan')
  File "/opt/anaconda/envs/lib/python3.7/site-packages/sklearn/utils/validation.py", line 60, in _assert_all_finite
    raise ValueError("Input contains NaN")
ValueError: Input contains NaN

Answer 1

重用代码是个好主意，但要注意当您将代码放入函数中时变量的范围如何变化。

您收到的错误是因为有 NaN 引起的您输入到随机森林的数组中的值。在你的feature_engineering_and_selection()函数，您将删除 NaN值，但是 df永远不会从函数返回，因此原始的、未经修改的 df用于模型中。

我建议拆分你的feature_engineering_and_selection()功能分为不同的组件。这里我做了一个函数，只删除 NaN s。

# Iterates through the columns and fixes any NaNs
def remove_nan(df):
    replace_dict = {}

    for col in df.columns:

        # If there are any NaN values in this column
        if pd.isna(df[col]).any():

            # Replace NaN in object columns with 'N/A'
            if df[col].dtypes == 'object':
                replace_dict[col] = 'N/A'

            # Replace NaN in float columns with 0
            elif df[col].dtypes == 'float64':
                replace_dict[col] = 0

    df = df.fillna(replace_dict)

    return df

我建议填写NaN用 0 代替平均值的数值。对于此数据，有 3 个具有 nan 值的数字列:LotFrontage (与特性相连的街道英尺数)，MasVnrArea (砌体贴面区域)，GarageYrBlt (车库建成年)。如果没有车库，则没有车库 build 年份，因此将年份设置为 0 而不是平均年份等是有意义的。

还需要使用您设置的单热编码器完成一些工作。创建 one-hot-encoding 可能很棘手，因为训练数据和测试数据需要具有相同的列。如果您有以下训练和测试数据

火车

| House Type |
| ---------- |
| Mansion    |
| Ranch      |

测试

| House Type |
| ---------- |
| Mansion    |
| Duplex     |

那么如果使用pd.get_dummies()火车列将是 [house_type_mansion, house_type_ranch]测试列将为 [house_type_mansion, house_type_duplex] ，这是行不通的。然而，使用 sklearn，您可以将一个热编码器安装到您的训练数据中。转换测试数据集时，它将创建与训练数据集相同的列。 handle_unknown参数将告诉编码器如何处理 duplex在测试集中，要么 ignore或error .

# Fits an sklearn one hot encoder
def train_one_hot_encoder(df, categorical_columns):
    # take one-hot encoding of categorical columns
    categorical_df = df[categorical_columns]
    enc = OneHotEncoder(sparse=False, handle_unknown='ignore')
    return enc.fit(categorical_df)

为了结合分类和非分类数据，我再次建议创建一个单独的函数

# One hot encodes the given dataframe
def one_hot_encode(df, categorical_columns, encoder):
    # Get dataframe with only categorical columns
    categorical_df = df[categorical_columns]
    # Get one hot encoding
    ohe_df = pd.DataFrame(encoder.transform(categorical_df), columns=encoder.get_feature_names())
    # Get float columns
    float_df = df.drop(categorical_columns, axis=1)
    # Return the combined array
    return pd.concat([float_df, ohe_df], axis=1)

最后，你的feature_engineering_and_selection() function 可以调用所有这些函数。

def feature_selection_and_engineering(df, encoder=None):
    df = remove_nan(df)
    categorical_columns = get_categorical_columns(df)
    # If there is no encoder, train one
    if encoder == None:
        encoder = train_one_hot_encoder(df, categorical_columns)
    # Encode Data
    df = one_hot_encode(df, categorical_columns, encoder)
    # Return the encoded data AND encoder
    return df, encoder

为了使代码运行，我必须修复一些问题，我已将整个修改后的脚本包含在此处的要点中 https://gist.github.com/kylelrichards11/6be90d92a7dd6a5cc9a5290dae3ff94e