java - Apache Beam 和 BigQuery

Question

我正在尝试执行 apache beam sdk 2.4 和库 com.google.cloud.bigquery 但出现异常

Exception in thread main java.lang.NoSuchMethodError com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(LjavalangString;)LcomgoogleapiclientgoogleapisservicesAbstractGoogleClient$Builder;
at com.google.api.services.bigquery.Bigquery$Builder.setBatchPath(Bigquery.java3519)

import com.google.cloud.bigquery.*;

<dependency>
    <groupId>com.google.cloud.dataflow</groupId>
    <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
    <version>[2.4.0, 2.99)</version>
</dependency>

<!-- slf4j API frontend binding with JUL backend -->
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <versi

on>1.7.14</version>
    </dependency>

   <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-jdk14</artifactId>
        <version>1.7.14</version>
    </dependency>

    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-bigquery</artifactId>
        <version>1.35.0</version>
    </dependency>

</dependencies>

the code example apache beam 2.4 using

import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;

If I use the the libraries om.google.api.services.bigquery. the codeexample run successfully .

What library is it using internally?

What library should I use? com.google.api.services.bigquery or com.google.cloud.bigquery?

Answer 1

Beam 2.4 正在使用 com.google.api.services.bigquery v2-rev374-1.22.0。这也是您应该与 Beam 2.4 一起使用的库。 com.google.api.services.bigquery 库已过时，有一个 open issue来升级它。遗憾的是，Beam 中的其他组件依赖于与 com.google.cloud.bigquery 不兼容的库，需要先升级这些库才能使用新库。

如果您还没有这样做，您应该查看 BigQuery IO ，它为 Beam 的常见 BigQuery 操作提供了包装器。