java - 如何从位于 Google Storage 的文本文件写入 Bitquery 表？

Question

我是 apache beam 和 Google 云数据流的新手，我想使用 Java SDK 编写一个程序，从位于 Google Storage 的文本文件中读取数据并将数据插入 Google Bigquery。

我已经为该问题编写了代码，但它显示错误读取数据时出错，错误消息:JSON表遇到太多错误，放弃。

public class StarterPipeline {
  private static final Logger LOG = LoggerFactory.getLogger(StarterPipeline.class);

  public static void main(String[] args) {
    PipelineOptions options = PipelineOptionsFactory.create();
    options.setTempLocation("gs://com_example_gcptraining/assignment_1/temp");
    Pipeline p = Pipeline.create(options);

    PCollection<String> input = p.apply(TextIO.read().from("gs://com_example_gcptraining/assignment_1/emp_details.txt"));

    PCollection<TableRow> output = input.apply(ParDo.of(new DoFn<String, TableRow>() {
      @ProcessElement
      public void processElement(ProcessContext c)  {
        StringTokenizer  tokenizer = new StringTokenizer(c.element(),",");
        c.output(new TableRow()
                .set("id", Integer.parseInt(tokenizer.nextToken()))
                .set("emp_name",tokenizer.nextToken())
                .set("designation", tokenizer.nextToken())
                .set("dob",Date.parseDate(tokenizer.nextToken()))
                .set("contact_no",Long.parseLong(tokenizer.nextToken()))
                .set("location",tokenizer.nextToken())
                .set("salary",Float.parseFloat(tokenizer.nextToken())));
      }
    }));

    String tableSpec = "gcp-training-246913:com_example_gcptraining.employee";
    output.apply(BigQueryIO.writeTableRows().to(tableSpec).withCreateDisposition(CreateDisposition.CREATE_NEVER).withWriteDisposition(WriteDisposition.WRITE_APPEND));

    p.run().waitUntilFinish();
  }
}

请帮我解决问题。

谢谢!!!

这是控制台输出日志，

Jul 21, 2019 2:02:47 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes
INFO: Filepattern gs://com_example_gcptraining/assignment_1/emp_details.txt matched 1 files with total size 682
Jul 21, 2019 2:02:47 PM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern gs://com_example_gcptraining/assignment_1/emp_details.txt into bundles of size 170 took 67 ms and produced 1 files and 4 bundles
Jul 21, 2019 2:02:48 PM org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$4 getTempFilePrefix
INFO: Writing BigQuery temporary files to gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/ before loading them.
Jul 21, 2019 2:02:49 PM org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter <init>
INFO: Opening TableRowWriter to gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/57558a8c-977e-478d-a44c-7863fb230af2.
Jul 21, 2019 2:02:49 PM org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter <init>
INFO: Opening TableRowWriter to gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5.
Jul 21, 2019 2:02:49 PM org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter <init>
INFO: Opening TableRowWriter to gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d73bb64e-cced-4016-9529-12df5bed8b9e.
Jul 21, 2019 2:02:49 PM org.apache.beam.sdk.io.gcp.bigquery.TableRowWriter <init>
INFO: Opening TableRowWriter to gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/be562c41-3aee-4c98-b1de-3e10691e3454.
Jul 21, 2019 2:02:50 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Loading 4 files into {datasetId=com_example_gcptraining, projectId=gcp-training-246913, tableId=employee} using job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0, location=US, projectId=gcp-training-246913}, attempt 0
Jul 21, 2019 2:02:50 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0, location=US, projectId=gcp-training-246913}.
bq show -j --format=prettyjson --project_id=gcp-training-246913 beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0
Jul 21, 2019 2:02:50 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0, location=US, projectId=gcp-training-246913} started
Jul 21, 2019 2:02:51 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0, currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=gcp-training-246913 beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0
Jul 21, 2019 2:02:52 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0, location=US, projectId=gcp-training-246913} completed in state DONE
Jul 21, 2019 2:02:52 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-0, location=US, projectId=gcp-training-246913} failed, will retry: {"errorResult":{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.","reason":"invalid"},"errors":[{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.","reason":"invalid"},{"message":"Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0","reason":"invalid"},{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: dob","reason":"invalid"}],"state":"DONE"}
Jul 21, 2019 2:02:52 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Loading 4 files into {datasetId=com_example_gcptraining, projectId=gcp-training-246913, tableId=employee} using job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1, location=US, projectId=gcp-training-246913}, attempt 1
Jul 21, 2019 2:02:53 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1, location=US, projectId=gcp-training-246913}.
bq show -j --format=prettyjson --project_id=gcp-training-246913 beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1
Jul 21, 2019 2:02:53 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1, location=US, projectId=gcp-training-246913} started
Jul 21, 2019 2:02:53 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1, currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=gcp-training-246913 beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1
Jul 21, 2019 2:02:54 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1, location=US, projectId=gcp-training-246913} completed in state DONE
Jul 21, 2019 2:02:54 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-1, location=US, projectId=gcp-training-246913} failed, will retry: {"errorResult":{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.","reason":"invalid"},"errors":[{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.","reason":"invalid"},{"message":"Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0","reason":"invalid"},{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: dob","reason":"invalid"}],"state":"DONE"}
Jul 21, 2019 2:02:54 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Loading 4 files into {datasetId=com_example_gcptraining, projectId=gcp-training-246913, tableId=employee} using job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2, location=US, projectId=gcp-training-246913}, attempt 2
Jul 21, 2019 2:02:54 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2, location=US, projectId=gcp-training-246913}.
bq show -j --format=prettyjson --project_id=gcp-training-246913 beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2
Jul 21, 2019 2:02:54 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2, location=US, projectId=gcp-training-246913} started
Jul 21, 2019 2:02:54 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: Still waiting for BigQuery job beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2, currently in status {"state":"RUNNING"}
bq show -j --format=prettyjson --project_id=gcp-training-246913 beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2
Jul 21, 2019 2:02:55 PM org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl pollJob
INFO: BigQuery job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2, location=US, projectId=gcp-training-246913} completed in state DONE
Jul 21, 2019 2:02:55 PM org.apache.beam.sdk.io.gcp.bigquery.WriteTables load
INFO: Load job {jobId=beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2, location=US, projectId=gcp-training-246913} failed, will not retry: {"errorResult":{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.","reason":"invalid"},"errors":[{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.","reason":"invalid"},{"message":"Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0","reason":"invalid"},{"location":"gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5","message":"Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: dob","reason":"invalid"}],"state":"DONE"}
Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.RuntimeException: Failed to create load job with id prefix beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000, reached max retries: 3, last failed load job: {
  "configuration" : {
    "load" : {
      "createDisposition" : "CREATE_NEVER",
      "destinationTable" : {
        "datasetId" : "com_example_gcptraining",
        "projectId" : "gcp-training-246913",
        "tableId" : "employee"
      },
      "schema" : {
        "fields" : [ {
          "mode" : "REQUIRED",
          "name" : "id",
          "type" : "INTEGER"
        }, {
          "name" : "emp_name",
          "type" : "STRING"
        }, {
          "name" : "designation",
          "type" : "STRING"
        }, {
          "name" : "dob",
          "type" : "DATE"
        }, {
          "name" : "contact_no",
          "type" : "NUMERIC"
        }, {
          "name" : "location",
          "type" : "STRING"
        }, {
          "name" : "salary",
          "type" : "FLOAT"
        } ]
      },
      "sourceFormat" : "NEWLINE_DELIMITED_JSON",
      "sourceUris" : [ "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/57558a8c-977e-478d-a44c-7863fb230af2", "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d73bb64e-cced-4016-9529-12df5bed8b9e", "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/be562c41-3aee-4c98-b1de-3e10691e3454", "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5" ],
      "writeDisposition" : "WRITE_APPEND"
    },
    "jobType" : "LOAD"
  },
  "etag" : "FFm2QxlI5DISFk5gFxWdbw==",
  "id" : "gcp-training-246913:US.beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2",
  "jobReference" : {
    "jobId" : "beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2",
    "location" : "US",
    "projectId" : "gcp-training-246913"
  },
  "kind" : "bigquery#job",
  "selfLink" : "https://www.googleapis.com/bigquery/v2/projects/gcp-training-246913/jobs/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2?location=US",
  "statistics" : {
    "creationTime" : "1563717774452",
    "endTime" : "1563717775186",
    "startTime" : "1563717774633"
  },
  "status" : {
    "errorResult" : {
      "location" : "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5",
      "message" : "Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.",
      "reason" : "invalid"
    },
    "errors" : [ {
      "location" : "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5",
      "message" : "Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.",
      "reason" : "invalid"
    }, {
      "message" : "Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0",
      "reason" : "invalid"
    }, {
      "location" : "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5",
      "message" : "Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: dob",
      "reason" : "invalid"
    } ],
    "state" : "DONE"
  },
  "user_email" : "shubham.naphade@outlook.in"
}.
    at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:349)
    at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:319)
    at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:210)
    at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:66)
    at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
    at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
    at com.example.gcptraining.StarterPipeline.main(StarterPipeline.java:75)
Caused by: java.lang.RuntimeException: Failed to create load job with id prefix beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000, reached max retries: 3, last failed load job: {
  "configuration" : {
    "load" : {
      "createDisposition" : "CREATE_NEVER",
      "destinationTable" : {
        "datasetId" : "com_example_gcptraining",
        "projectId" : "gcp-training-246913",
        "tableId" : "employee"
      },
      "schema" : {
        "fields" : [ {
          "mode" : "REQUIRED",
          "name" : "id",
          "type" : "INTEGER"
        }, {
          "name" : "emp_name",
          "type" : "STRING"
        }, {
          "name" : "designation",
          "type" : "STRING"
        }, {
          "name" : "dob",
          "type" : "DATE"
        }, {
          "name" : "contact_no",
          "type" : "NUMERIC"
        }, {
          "name" : "location",
          "type" : "STRING"
        }, {
          "name" : "salary",
          "type" : "FLOAT"
        } ]
      },
      "sourceFormat" : "NEWLINE_DELIMITED_JSON",
      "sourceUris" : [ "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/57558a8c-977e-478d-a44c-7863fb230af2", "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d73bb64e-cced-4016-9529-12df5bed8b9e", "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/be562c41-3aee-4c98-b1de-3e10691e3454", "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5" ],
      "writeDisposition" : "WRITE_APPEND"
    },
    "jobType" : "LOAD"
  },
  "etag" : "FFm2QxlI5DISFk5gFxWdbw==",
  "id" : "gcp-training-246913:US.beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2",
  "jobReference" : {
    "jobId" : "beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2",
    "location" : "US",
    "projectId" : "gcp-training-246913"
  },
  "kind" : "bigquery#job",
  "selfLink" : "https://www.googleapis.com/bigquery/v2/projects/gcp-training-246913/jobs/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951_0d1a010156cfb76d262315de02b986ab_00001_00000-2?location=US",
  "statistics" : {
    "creationTime" : "1563717774452",
    "endTime" : "1563717775186",
    "startTime" : "1563717774633"
  },
  "status" : {
    "errorResult" : {
      "location" : "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5",
      "message" : "Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.",
      "reason" : "invalid"
    },
    "errors" : [ {
      "location" : "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5",
      "message" : "Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.",
      "reason" : "invalid"
    }, {
      "message" : "Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0",
      "reason" : "invalid"
    }, {
      "location" : "gs://com_example_gcptraining/assignment_1/temp/BigQueryWriteTemp/beam_load_starterpipelineshubhamn0721140248cbe18bb0_d76616a7bafb416fa8e4ff9db0f8a951/d5c2db36-ec77-48bf-968c-e27e8c06d9c5",
      "message" : "Error while reading data, error message: JSON parsing error in row starting at position 0: JSON object specified for non-record field: dob",
      "reason" : "invalid"
    } ],
    "state" : "DONE"
  },
  "user_email" : "shubham.naphade@outlook.in"
}.
    at org.apache.beam.sdk.io.gcp.bigquery.WriteTables.load(WriteTables.java:308)
    at org.apache.beam.sdk.io.gcp.bigquery.WriteTables.access$600(WriteTables.java:80)
    at org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.processElement(WriteTables.java:159)

Answer 1

您的数据(字段 dob)似乎有问题。您需要一个日期，但由于某种原因它无法存储到 BigQuery 表中。

JSON parsing error in row starting at position 0: JSON object specified for non-record field: dob"

检查表架构，或者日期格式可能存在问题。