google-cloud-dataflow - com.google.common.base.Preconditions.checkState 的 NoSuchMethodError

Question

我将 Apache Beam 与 Cloud Dataflow Runner 一起使用，并在尝试运行管道时出现以下异常:

java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkState(ZLjava/lang/String;Ljava/lang/Object;)V
at org.apache.beam.sdk.io.gcp.pubsub.PubsubClient$TopicPath.getV1Beta1Path(PubsubClient.java:264)
at org.apache.beam.runners.dataflow.DataflowRunner$StreamingPubsubIOReadTranslator.translate(DataflowRunner.java:856)
at org.apache.beam.runners.dataflow.DataflowRunner$StreamingPubsubIOReadTranslator.translate(DataflowRunner.java:843)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:435)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:491)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$400(TransformHierarchy.java:235)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:210)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:409)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:381)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:171)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:523)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:153)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:295)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:281)
at com.souq.ApsalarEvent.<init>(ApsalarEvent.java:57)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.eclipse.jetty.server.handler.ContextHandler$Context.createInstance(ContextHandler.java:2521)
at org.eclipse.jetty.servlet.ServletContextHandler$Context.createServlet(ServletContextHandler.java:1326)
at org.eclipse.jetty.servlet.ServletHolder.newInstance(ServletHolder.java:1273)
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:614)
at org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:498)
at org.eclipse.jetty.servlet.ServletHolder.ensureInstance(ServletHolder.java:785)
at org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:770)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:538)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1593)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1239)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1562)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1141)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:118)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)

通过将其添加到我的 pom.xml 中，我确保我使用的是最新的 Guava 版本:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>21.0</version>
        </dependency>
    </dependencies>
</dependencyManagement>

解码堆栈跟踪中的方法签名给了我:

void checkState(boolean, String, Object)

我在我的项目中看到了这种方法:

下面是我完整的POM

<modelVersion>4.0.0</modelVersion>
<packaging>war</packaging>
<version>1.0-SNAPSHOT</version>

<groupId>com.example</groupId>
<artifactId>test</artifactId>

<properties>
    <appengine.version>1.9.51</appengine.version>
    <appengine.maven.plugin.version>1.0.0</appengine.maven.plugin.version>
    <jetty-maven-plugin-version>9.3.7.v20160115</jetty-maven-plugin-version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <maven.compiler.showDeprecation>true</maven.compiler.showDeprecation>
    <archiveClasses>true</archiveClasses>
    <failOnMissingWebXml>false</failOnMissingWebXml>
</properties>

<prerequisites>
    <maven>3.3.9</maven>
</prerequisites>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>20.0</version>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>javax.servlet</groupId>
        <artifactId>javax.servlet-api</artifactId>
        <version>3.1.0</version>
        <type>jar</type>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.beam</groupId>
        <artifactId>beam-sdks-java-core</artifactId>
        <version>2.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.beam</groupId>
        <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
        <version>2.0.0</version>
    </dependency>
</dependencies>

<build>
    <!-- for hot reload of the web application-->
    <outputDirectory>${project.build.directory}/${project.build.finalName}/WEB-INF/classes</outputDirectory>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>versions-maven-plugin</artifactId>
            <version>2.3</version>
            <executions>
                <execution>
                    <phase>compile</phase>
                    <goals>
                        <goal>display-dependency-updates</goal>
                        <goal>display-plugin-updates</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-war-plugin</artifactId>
            <version>2.6</version> <!-- required for Eclipse Mars -->
            <configuration>
                <archiveClasses>true</archiveClasses>
                <webResources>
                    <!-- in order to interpolate version from pom into appengine-web.xml -->
                    <resource>
                        <directory>${basedir}/src/main/webapp/WEB-INF</directory>
                        <filtering>true</filtering>
                        <targetPath>WEB-INF</targetPath>
                    </resource>
                </webResources>
            </configuration>
        </plugin>

        <plugin>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-maven-plugin</artifactId>
            <version>${jetty-maven-plugin-version}</version>
        </plugin>

        <plugin>
            <groupId>com.google.cloud.tools</groupId>
            <artifactId>appengine-maven-plugin</artifactId>
            <version>${appengine.maven.plugin.version}</version>
            <configuration>
            </configuration>
        </plugin>
    </plugins>
</build>

Answer 1

通常，您可以使用 mvn dependency:tree -Dverbose 找出您的项目使用的依赖项。 .根据您提供的 pom.xml ，您的项目正在使用 guava 20.0 .

但是，在这种情况下，如果您使用任何容器环境(如 AppEngine 或 IDE)来运行/测试作业创建，查看 Maven 依赖项只会告诉您项目包含的内容，而不是容器环境包含的内容。创建作业后，您应该能够通过调用 getFilesToStage() 获取为 Google Cloud Dataflow 暂存的文件列表。 :

Pipeline p = ...
...
p.run(options);
List<String> resources = options.as(DataflowPipelineWorkerPoolOptions.class).getFilesToStage();
System.out.println(resources);

您可以从 GCS 下载 jar 并提取它们。然后使用 javap 内省(introspection)类.这样做你会发现另一个版本的违规 Preconditions类(class)。删除有问题的类是特定于容器的，但可以通过在运行管道之前将文件专门设置为手动暂存来解决。