java - 如何使用 Java 将 BigQuery 读取到 Apache Spark

Question

我想使用 Java 将 Google BigQuery 中的表中的数据读取到 Spark 中。我如何在 Java 中做到这一点，我需要什么依赖项以及生成的数据类型是什么？

我能找到的所有内容都在 Scala 中，但我需要 Java 中的内容。

Answer 1

下面是 Scala Shakespeare 示例的 Java 等效项:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

public class JavaShakespeare {

  public static void main(String[] args) {
    SparkSession spark = SparkSession.builder()
        .appName("spark-bigquery-demo")
        .getOrCreate();

    // Use the Cloud Storage bucket for temporary BigQuery export data used
    // by the connector. This assumes the Cloud Storage connector for
    // Hadoop is configured.
    String bucket = spark.sparkContext().hadoopConfiguration().get("fs.gs.system.bucket");
    spark.conf().set("temporaryGcsBucket", bucket);

    // Load data in from BigQuery.
    Dataset<Row> wordsDF = spark.read().format("bigquery")
        .option("table", "publicdata.samples.shakespeare").load().cache();
    wordsDF.show();
    wordsDF.printSchema();
    wordsDF.createOrReplaceTempView("words");

    // Perform word count.
    Dataset<Row> wordCountDF = spark.sql(
        "SELECT word, SUM(word_count) AS word_count FROM words GROUP BY word");

    // Saving the data to BigQuery
    wordCountDF.write().format("bigquery").option("table", "wordcount_dataset.wordcount_output")
        .save();
  }
}