google-cloud-dataflow - com.google.cloud.spanner.SpannerException : DEADLINE_EXCEEDED

Question

我已经被这个问题困扰了一天多了，我想这里有人可能知道答案。基本上，作为一个简单的测试，我试图从表中读取数据并将结果简单地输出到日志文件中。该表相当大(约 1.67 亿行)。我不断收到以下错误

java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: com.google.cloud.spanner.SpannerException: DEADLINE_EXCEEDED: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 119997536405ns

接着是:

Workflow failed. Causes: S15:SpannerIO.ReadAll/Read from Cloud Spanner/Shuffle partitions/Reshuffle/GroupByKey/Read+SpannerIO.ReadAll/Read from Cloud Spanner/Shuffle partitions/Reshuffle/GroupByKey/GroupByWindow+SpannerIO.ReadAll/Read from Cloud Spanner/Shuffle partitions/Reshuffle/ExpandIterable+SpannerIO.ReadAll/Read from Cloud Spanner/Shuffle partitions/Values/Values/Map+SpannerIO.ReadAll/Read from Cloud Spanner/Read from Partitions+ParDo(FinishProcess) failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers: ricardocascadejob-m0771463-12021322-cpj6-harness-4nvn Root cause: Work item failed., ricardocascadejob-m0771463-12021322-cpj6-harness-c485 Root cause: Work item failed., ricardocascadejob-m0771463-12021322-cpj6-harness-kcgb Root cause: Work item failed., ricardocascadejob-m0771463-12021322-cpj6-harness-kcgb Root cause: Work item failed.

这里是在数据流上运行的主要代码

PipelineOptionsFactory.register(RicardoPriceLoadOptions.class);
    RicardoPriceLoadOptions opts = PipelineOptionsFactory.fromArgs(args)
        .withValidation().as(RicardoPriceLoadOptions.class);
    Pipeline pipeline = Pipeline.create(opts);

    SpannerConfig spannerConfig =
        SpannerConfig.create()
            .withProjectId(opts.getGcpProjectId())
            .withInstanceId(opts.getSpannerInstanceId())
            .withDatabaseId(opts.getSpannerDatabaseId());

    PCollectionView<Transaction> tx =
        pipeline.apply(SpannerIO.createTransaction().withSpannerConfig(spannerConfig));

    //Fetch All Price Events
    PCollection<Struct> pepLIst = pipeline.apply(Create.of(ReadOperation.create()
        .withColumns("DisabledFlag", "PriceEventPriceableId", "PriceableItemId",
            "OutgoingType", "PriceOriginal", "PriceIntermediate", "PriceRetail", "SaleValue", "SaleValueIntermediate",
            "SchedulableFlag", "SendToSiteFlag", "StartTime", "EndTime", "DisplayCode")
        .withTable("ABC")))
        .apply(SpannerIO.readAll().withTransaction(tx).withSpannerConfig(spannerConfig));

    pepLIst.apply(ParDo.of(new FinishProcessFn()));

最后一个 DoFn 是一个简单的 DoFn，它只记录 spanner 行。

public class FinishProcessFn extends DoFn<Struct, Void> {

@ProcessElement
public void process(@Element Struct elem) {
    log.debug(elem.toString());
}

我已经尝试了这里显示的谷歌建议

Common Errors

代码看起来很简单，但我不确定为什么我总是收到上面的错误。感谢任何输入或帮助。

谢谢!

这是表模式

    CREATE TABLE ABC (
    PriceEventPriceableId INT64 NOT NULL,
    Created TIMESTAMP NOT NULL,
    CreatedBy STRING(MAX) NOT NULL,
    DisabledFlag STRING(MAX) NOT NULL,
    DisplayCode STRING(MAX),
    EndTime TIMESTAMP,
    ErrorCode INT64,
    EstablishmentOverrideFlag STRING(MAX),
    LastUpdated TIMESTAMP NOT NULL,
    LastUpdatedBy STRING(MAX) NOT NULL,
    NotApplicableFlag STRING(MAX),
    OnSaleRatioOverrideFlag STRING(MAX),
    OutgoingType INT64,
    OwnedValue STRING(MAX),
    ParentPriceableItemId INT64,
    PriceableItemId INT64 NOT NULL,
    PriceEventId INT64 NOT NULL,
    PriceIntermediate STRING(MAX),
    PriceOriginal STRING(MAX),
    PriceRetail STRING(MAX),
    ReasonUnschedulable STRING(MAX),
    SaleValue STRING(MAX),
    SaleValueIntermediate STRING(MAX),
    SavingsMaxOverrideFlag STRING(MAX),
    SchedulableFlag STRING(MAX),
    SendToSiteFlag STRING(MAX),
    SentToSiteDate DATE,
    StartTime TIMESTAMP,
    StoredValue STRING(MAX),
    TenPercentOverrideFlag STRING(MAX),
    Timestamp TIMESTAMP NOT NULL OPTIONS (allow_commit_timestamp=true),
) PRIMARY KEY (PriceEventPriceableId)

Answer 1

DEADLINE_EXCEEDED error表示操作未在给定时间内完成。

对于更改系统状态的操作，即使操作已成功完成，也可能会返回此错误(例如，来自服务器的成功响应可能已延迟足够长的时间以达到截止日期).

我在 Google Cloud Status Dashboard 中看不到 Cloud Spanner 的任何问题| ，但是，最好联系 GCP 支持，以便他们可以检查您的项目并更深入地研究问题。