具有替代性能指标的 R 插入符号模型评估

Question

我正在使用 R 的 caret 包进行一些网格搜索和模型评估。我有一个自定义评估指标，它是绝对误差的加权平均值。在观察级别分配权重。

X <- c(1,1,2,0,1) #feature 1
w <- c(1,2,2,1,1) #weights
Y <- 1:5 #target, continuous

#assume I run a model using X as features and Y as target and get a vector of predictions

mymetric <- function(predictions, target, weights){

v <- sum(abs(target-predictions)*weights)/sum(weights) 
return(v)
}

Here给出了一个关于如何使用 summaryFunction 为 caret 的 train() 定义自定义评估指标的示例。
报价:

trainControl 函数有一个名为 summaryFunction 的参数，它指定一个用于计算性能的函数。该函数应具有以下参数:

data is a reference for a data frame or matrix with columns called obs and pred for the observed and predicted outcome values (either numeric data for regression or character values for classification). Currently, class probabilities are not passed to the function. The values in data are the held-out predictions (and their associated reference values) for a single combination of tuning parameters. If the classProbs argument of the trainControl object is set to TRUE, additional columns in data will be present that contains the class probabilities. The names of these columns are the same as the class levels. lev is a character string that has the outcome factor levels taken from the training data. For regression, a value of NULL is passed into the function. model is a character string for the model being used (i.e. the value passed to the method argument of train).

我无法弄清楚如何将观察权重传递给 summaryFunction。

Answer 1

您不能将权重直接传递给汇总函数，这是一种疏忽，因为您可以将它们传递给建模函数。如果基础模型适应权重，它们将用于生成预测值。

我会把它添加到下一个版本中。

最大限度