r - Spark 错误 : No rows dropped by 'na.omit' call

Question

当我尝试将 ml_decision_tree 或 ml_logistic_regresion 与 Sparklyr 包一起使用时，出现以下错误。我在 cloudera 集群上使用 spark2.1.0。

> No rows dropped by 'na.omit' call.  Error in
> stop(simpleError(sprintf(fmt, ...), if (call.)
> sys.call(sys.parent()))) :   bad error message

下面是我运行的代码片段:

at<-data_select

for (col in c(colnames(data_select)))
 {
data_ft <-at%>%
        ft_string_indexer(input.col =col, output.col = paste0(col,"_in"))%>%
        ft_one_hot_encoder(input.col = paste0(col,"_in"), output.col = paste0(col,'_ohe'))
        at <-data_ft
}

# create the features vectors
data_col<-colnames(data_ft) # get the colnames names of the aiom_ft table
gp<- grep("*ohe", data_col) # select only columns ended with ohe
features <-c(data_col[gp]) # get the names of those columns
features<-features[features!="target_ohe"] # remove the target variables from the features columns

# create the feactures vectors
data_feac<- ft_vector_assembler(data_ft, input.col = features , output.col ='FeacturesVectors' )

# create a aprtition
#parition the table
partitions<- data_feac %>%
             sdf_partition(training = 0.6, test = 0.4, seed = 10099)
    fit.dec<-partitions$training %>%
        ml_decision_tree(data_feac, response= 'target_ohe', features='FeacturesVectors', type ="classification", ml_options(na.action = getOption("na.action", "na.pass")))

我试过 ml_options(na.action = getOption("na.action", "na.pass")) 和 ml_options(na.action = getOption("na.action", "na.omit"))和同样的错误信息

Answer 1

这是 sparklyr 的未决问题。看这个issue在 GitHub 上