java - 如何在数据框中动态地从列表中选择列加上固定列

Question

我正在使用带有 java8 的 Spark-sql-2.4.1v。

我有动态列列表被传递到我的函数中。

即

List<String> cols = Arrays.asList("col_1","col_2","col_3","col_4");
Dataset<Row> df = //which has above columns plus "id" ,"name" plus many other columns;

需要选择 cols + "id"+ "name"

我正在做如下

Dataset<Row> res_df = df.select("id", "name", cols.stream().toArray( String[]::new)); 

this is giving compilation error. so how to handle this use-case.

尝试过:

当我执行如下操作时:

List<String> cols = new ArrayList<>(Arrays.asList("col_1","col_2","col_3","col_4"));
cols.add("id");
cols.add("name");

给出错误

Exception in thread "main" java.lang.UnsupportedOperationException
    at java.util.AbstractList.add(AbstractList.java:148)
    at java.util.AbstractList.add(AbstractList.java:108)

Answer 1

您可以创建列数组并将其传递给 select 语句。

import org.apache.spark.sql.*;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

List<String> cols = new ArrayList<>(Arrays.asList("col_1","col_2","col_3","col_4"));
cols.add("id");
cols.add("name");
Column[] cols2 = cols.stream()
        .map(s->new Column(s)).collect(Collectors.toList())
        .toArray(new Column[0]);

settingsDataset.select(cols2).show();