scala - 通过 Spark 作业加载到 BigQuery 失败，为 parquet 找到多个来源的异常

Question

我有一个将数据加载到 BigQuery 的 spark 作业。 spark 作业在 dataproc 集群中运行。
这是片段

df.write
      .format("bigquery")
      .mode(writeMode)
      .option("table",tabName)
      .save()

我在 spark-submit 命令的 --jars 参数中指定了 spark bigquery 依赖 jar ( spark-bigquery-with-dependencies_2.12-0.19.1.jar )
当我运行代码时，我收到以下异常 java.lang.RuntimeException: Failed to write to BigQuery详细错误

Caused by: org.apache.spark.sql.AnalysisException: Multiple sources found for parquet (org.apache.spark.sql.execution.datasources.v2.parquet.ParquetDataSourceV2, org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat), please specify the fully qualified class name.
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:717)

这是我项目中的依赖项

<dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.12.14</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.12</artifactId>
            <version>2.4.8</version>
        </dependency>
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-bigquery</artifactId>
            <version>1.133.1</version>
        </dependency>
        <dependency>
            <groupId>com.google.cloud.spark</groupId>
            <artifactId>spark-bigquery_2.12</artifactId>
            <version>0.21.1</version>
        </dependency>
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-storage</artifactId>
            <version>1.116.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.1.1</version>
        </dependency>
        <dependency>
            <groupId>net.alchim31.maven</groupId>
            <artifactId>scala-maven-plugin</artifactId>
            <version>3.3.3</version>
        </dependency>
    </dependencies>

我正在构建一个 uber jar 来运行 spark 作业
如果，我删除了 --jars 参数，则在读取 bigquery 表时作业失败

java.lang.ClassNotFoundException: Failed to find data source: bigquery. Please find packages at http://spark.apache.org/third-party-projects.html
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:689)

Answer 1

看来您正在使用带有已编译并包含 spark 2.4.8 工件的 jar 的 Spark 3.x。解决方案很简单:将 scala-library 和 spark-sql 标记为范围 provided .此外，当您将 spark-bigquery-connector 带到外部时，您不需要将其添加到代码中(以及 google-cloud-* 依赖项，除非您直接使用它们)