个人中心
gpt4 book ai didi

java - 在数据流2.3.0中使用谷歌 Spanner

转载 作者：行者123 更新时间：2023-12-02 02:16:36 24 4
gpt4 key购买 nike

我正在尝试写信来抓取来自 google pub/sub 的一些消息。写入 bigtable 是有效的，因此在基础层面上没有遗漏任何内容。

我已升级到 google-cloud-dataflow-java-sdk-all v2.3.0，其中包括 SpannerIO 接收器。

我基本上是按照 API docs of SpannerIO 中的示例进行编写的.

但是，我遇到了异常 internal_static_google_rpc_LocalizedMessage_fieldAccessorTable

完整堆栈跟踪:

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project second-dataflow: An exception occured while executing the Java class. internal_static_google_rpc_LocalizedMessage_fieldAccessorTable
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An exception occured while executing the Java class. internal_static_google_rpc_LocalizedMessage_fieldAccessorTable
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:339)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: java.lang.NoSuchFieldError: internal_static_google_rpc_LocalizedMessage_fieldAccessorTable
    at com.google.rpc.LocalizedMessage.internalGetFieldAccessorTable (LocalizedMessage.java:87)
    at com.google.protobuf.GeneratedMessageV3.getDescriptorForType (GeneratedMessageV3.java:121)
    at io.grpc.protobuf.ProtoUtils.keyForProto (ProtoUtils.java:126)
    at com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor.<clinit> (SpannerErrorInterceptor.java:47)
    at com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.<init> (GrpcSpannerRpc.java:128)
    at com.google.cloud.spanner.SpannerOptions$DefaultSpannerRpcFactory.create (SpannerOptions.java:70)
    at com.google.cloud.spanner.SpannerOptions$DefaultSpannerRpcFactory.create (SpannerOptions.java:65)
    at com.google.cloud.ServiceOptions.getRpc (ServiceOptions.java:443)
    at com.google.cloud.spanner.SpannerOptions.getSpannerRpcV1 (SpannerOptions.java:315)
    at com.google.cloud.spanner.SpannerImpl.<init> (SpannerImpl.java:138)
    at com.google.cloud.spanner.SpannerOptions$DefaultSpannerFactory.create (SpannerOptions.java:60)
    at com.google.cloud.spanner.SpannerOptions$DefaultSpannerFactory.create (SpannerOptions.java:55)
    at com.google.cloud.ServiceOptions.getService (ServiceOptions.java:430)
    at org.apache.beam.sdk.io.gcp.spanner.SpannerConfig.connectToSpanner (SpannerConfig.java:155)
    at com.crypto20.DeveloperSubscription.main (DeveloperSubscription.java:94)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:748)

指定 withProjectId() 并不能解决此问题。

也不会升级所有依赖项，包括那些传递性引入的依赖项 - 除非我犯了一个错误(我不习惯数据流的内部结构以及每个依赖项的作用)。更新依赖项后，甚至无法暂存已编译的项目。

我怀疑有一个特定的依赖项需要更新，我尝试了各种组合，但无济于事。

更新:

我尝试过的一件事是将 grpc-google-common-protos 固定到 0.1.9 甚至是最新的 1.2.0.

如果我这样做，甚至上传管道都不起作用，因为我在上传之前收到以下异常:

Mar 12, 2018 7:40:22 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 120 files from PipelineOptions.filesToStage to staging location to prepare for execution.
[WARNING] 
java.lang.RuntimeException: Error while staging packages
    at org.apache.beam.runners.dataflow.util.PackageUtil.stageClasspathElements (PackageUtil.java:396)
    at org.apache.beam.runners.dataflow.util.PackageUtil.stageClasspathElements (PackageUtil.java:272)
    at org.apache.beam.runners.dataflow.util.GcsStager.stageFiles (GcsStager.java:76)
    at org.apache.beam.runners.dataflow.util.GcsStager.stageDefaultFiles (GcsStager.java:64)
    at org.apache.beam.runners.dataflow.DataflowRunner.run (DataflowRunner.java:661)
    at org.apache.beam.runners.dataflow.DataflowRunner.run (DataflowRunner.java:174)
    at org.apache.beam.sdk.Pipeline.run (Pipeline.java:311)
    at org.apache.beam.sdk.Pipeline.run (Pipeline.java:297)
    at com.crypto20.DeveloperSubscription.main (DeveloperSubscription.java:106)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:748)
Caused by: java.io.IOException: Error executing batch GCS request
    at org.apache.beam.sdk.util.GcsUtil.executeBatches (GcsUtil.java:610)
    at org.apache.beam.sdk.util.GcsUtil.getObjects (GcsUtil.java:341)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystem.matchNonGlobs (GcsFileSystem.java:216)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystem.match (GcsFileSystem.java:85)
    at org.apache.beam.sdk.io.FileSystems.match (FileSystems.java:123)
    at org.apache.beam.sdk.io.FileSystems.matchSingleFileSpec (FileSystems.java:188)
    at org.apache.beam.runners.dataflow.util.PackageUtil.alreadyStaged (PackageUtil.java:159)
    at org.apache.beam.runners.dataflow.util.PackageUtil.stagePackageSynchronously (PackageUtil.java:183)
    at org.apache.beam.runners.dataflow.util.PackageUtil.lambda$stagePackage$1 (PackageUtil.java:173)
    at org.apache.beam.runners.dataflow.repackaged.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:111)
    at org.apache.beam.runners.dataflow.repackaged.com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:58)
    at org.apache.beam.runners.dataflow.repackaged.com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:75)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: com.google.api.client.http.HttpResponseException: 404 Not Found
Not Found
    at org.apache.beam.sdks.java.extensions.google.cloud.platform.core.repackaged.com.google.common.util.concurrent.AbstractFuture.getDoneValue (AbstractFuture.java:500)
    at org.apache.beam.sdks.java.extensions.google.cloud.platform.core.repackaged.com.google.common.util.concurrent.AbstractFuture.get (AbstractFuture.java:479)
    at org.apache.beam.sdks.java.extensions.google.cloud.platform.core.repackaged.com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get (AbstractFuture.java:76)
    at org.apache.beam.sdk.util.GcsUtil.executeBatches (GcsUtil.java:602)
    at org.apache.beam.sdk.util.GcsUtil.getObjects (GcsUtil.java:341)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystem.matchNonGlobs (GcsFileSystem.java:216)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystem.match (GcsFileSystem.java:85)
    at org.apache.beam.sdk.io.FileSystems.match (FileSystems.java:123)
    at org.apache.beam.sdk.io.FileSystems.matchSingleFileSpec (FileSystems.java:188)
    at org.apache.beam.runners.dataflow.util.PackageUtil.alreadyStaged (PackageUtil.java:159)
    at org.apache.beam.runners.dataflow.util.PackageUtil.stagePackageSynchronously (PackageUtil.java:183)
    at org.apache.beam.runners.dataflow.util.PackageUtil.lambda$stagePackage$1 (PackageUtil.java:173)
    at org.apache.beam.runners.dataflow.repackaged.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:111)
    at org.apache.beam.runners.dataflow.repackaged.com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:58)
    at org.apache.beam.runners.dataflow.repackaged.com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:75)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)
Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
Not Found
    at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:1070)
    at com.google.api.client.googleapis.batch.BatchRequest.execute (BatchRequest.java:241)
    at org.apache.beam.sdk.util.GcsUtil.lambda$executeBatches$0 (GcsUtil.java:596)
    at org.apache.beam.sdks.java.extensions.google.cloud.platform.core.repackaged.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:111)
    at org.apache.beam.sdks.java.extensions.google.cloud.platform.core.repackaged.com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:58)
    at org.apache.beam.sdks.java.extensions.google.cloud.platform.core.repackaged.com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:75)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)
Mar 12, 2018 7:40:22 AM org.apache.beam.runners.dataflow.repackaged.com.google.common.util.concurrent.AggregateFuture$RunningState handleException
SEVERE: Got more than one input Future failure. Logging failures after the first
java.io.IOException: Error executing batch GCS request
    at org.apache.beam.sdk.util.GcsUtil.executeBatches(GcsUtil.java:610)
    at org.apache.beam.sdk.util.GcsUtil.getObjects(GcsUtil.java:341)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystem.matchNonGlobs(GcsFileSystem.java:216)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystem.match(GcsFileSystem.java:85)
    at org.apache.beam.sdk.io.FileSystems.match(FileSystems.java:123)
    at org.apache.beam.sdk.io.FileSystems.matchSingleFileSpec(FileSystems.java:188)
    at org.apache.beam.runners.dataflow.util.PackageUtil.alreadyStaged(PackageUtil.java:159)
    at org.apache.beam.runners.dataflow.util.PackageUtil.stagePackageSynchronously(PackageUtil.java:183)
    at org.apache.beam.runners.dataflow.util.PackageUtil.lambda$stagePackage$1(PackageUtil.java:173)
    at org.apache.beam.runners.dataflow.repackaged.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
    at org.apache.beam.runners.dataflow.repackaged.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
    at org.apache.beam.runners.dataflow.repackaged.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

最佳答案

看看这个 Stack Overflow 帖子:Error using SpannerIO in apache beam 。它可能与 JIRA 问题 BEAM-2837 中描述的问题相关或相同。 .

Stack Overflow 帖子没有公认的答案，但我能够通过显式定义所需的 com.google.api.grpc 依赖项并从 中排除该版本来解决该问题>org.apache.beam.

<dependency>
    <groupId>com.google.api.grpc</groupId>
    <artifactId>grpc-google-common-protos</artifactId>
    <version>0.1.9</version>
</dependency>

<dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
    <version>${beam.version}</version>
    <exclusions>
        <exclusion>
            <groupId>com.google.api.grpc</groupId>
            <artifactId>grpc-google-common-protos</artifactId>
        </exclusion>
    </exclusions>
</dependency>

关于java - 在数据流2.3.0中使用谷歌 Spanner ，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/49222646/

24 4 0
文章推荐： django - 使用 RabbitMQ 的 Amazon Elastic Beanstalk 上的 Celery 超时异常
文章推荐： python-3.x - MacOS BigSur 上的 Pyinstaller 无法构建基本的 PyQt5 应用程序
文章推荐： sql - 如何在日期时间翻转月份和日期？
文章推荐： delphi - 为什么上传的照片比保存的小得多 - Delphi 10.3.2，firemonkey
行者123
个人简介

我是一名优秀的程序员,十分优秀！

滴滴打车优惠券免费领取
滴滴打车优惠券
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com