gpt4 book ai didi

go - "No filesystem found for scheme gs"在谷歌云平台运行数据流时

转载 作者:行者123 更新时间:2023-12-03 22:05:40 24 4
gpt4 key购买 nike

我在谷歌云平台(GCP)中运行我的谷歌数据流作业。
当我在本地运行此作业时,它运行良好,但是在 GCP 上运行时,出现此错误
“java.lang.IllegalArgumentException:未找到方案 gs 的文件系统”。
我可以访问那个谷歌云 URI,我可以将我的 jar 文件上传到那个 URI,我可以看到我的本地工作的一些临时文件。

我在 GCP 中的工作 ID:

2019-08-08_21_47_27-162804342585245230(梁版:2.12.0)

2019-08-09_16_41_15-11728697820819900062(梁版:2.14.0)

我尝试过 2.12.0 和 2.14.0 的梁版本,它们都有相同的错误。


java.lang.IllegalArgumentException: No filesystem found for scheme gs
at org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:456)
at org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:526)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.resolveTempLocation(BigQueryHelpers.java:689)
at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.extractFiles(BigQuerySourceBase.java:125)
at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.split(BigQuerySourceBase.java:148)
at org.apache.beam.runners.dataflow.worker.WorkerCustomSources.splitAndValidate(WorkerCustomSources.java:284)
at org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplitTyped(WorkerCustomSources.java:206)
at org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplitWithApiLimit(WorkerCustomSources.java:190)
at org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplit(WorkerCustomSources.java:169)
at org.apache.beam.runners.dataflow.worker.WorkerCustomSourceOperationExecutor.execute(WorkerCustomSourceOperationExecutor.java:78)
at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:412)
at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:381)
at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306)
at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

最佳答案

如果您构建一个捆绑所有依赖项的“胖 jar”,这可能是由几个问题引起的。

  • 您必须包含依赖项 org.apache.beam:google-cloud-platform-core 拥有 Beam GCS 文件系统。
  • 在远 jar 中,您必须保留 META-INF/services/org.apache.beam.sdk.io.FileSystemRegistrar带一行的文件 org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar .您可以在步骤 1 的 jar 中找到此文件。您的依赖项中可能有许多同名文件,注册了不同的 Beam 文件系统。您需要配置 maven 或 gradle 以将它们组合为构建的一部分,否则它们会相互覆盖而无法正常工作。
  • 关于go - "No filesystem found for scheme gs"在谷歌云平台运行数据流时,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/57438349/

    24 4 0
    Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
    广告合作:1813099741@qq.com 6ren.com