google-cloud-platform - 由于网络无法访问 googleapis.com，Dataproc 配置超时

Question

我正在尝试在 GCP 项目中创建一个基本的(我使用默认值)dataproc 集群，VM 已创建但集群仍处于配置状态直到超时

• 我尝试了控制台和命令行。
• 我尝试了不同的镜像版本(2.0-debian、2.0-ubuntu、1.5-debian、1.5-ubuntu)
• 未选择组件(它将用于 spark 作业)

在所有这些情况下，我都会遇到以下错误(在/var/log/google-dataproc-agent.0.log 上找到 SSHing master)

网络无法访问:dataproccontrol-europe-west1.googleapis.com/2a00:1450:400c:c04:0:0:0:5f:443

完整的错误跟踪:

ul 24, 2021 11:02:53 AM com.google.cloud.hadoop.services.repackaged.com.google.cloud.hadoop.util.ResilientOperation nextSleep INFO: Transient exception caught. Sleeping for 1120, then retrying.
com.google.cloud.hadoop.services.repackaged.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 29.974635818s. [buffered_nanos=30006131805, waiting_for_connection]
        at com.google.cloud.hadoop.services.repackaged.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:244)
        at com.google.cloud.hadoop.services.repackaged.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:225)
        at com.google.cloud.hadoop.services.repackaged.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:142)
        at com.google.cloud.dataproc.control.v1.AgentServiceGrpc$AgentServiceBlockingStub.createAgent(AgentServiceGrpc.java:735)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater$1.call(AgentApiAsyncUpdater.java:238)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater$1.call(AgentApiAsyncUpdater.java:235)
        at com.google.cloud.hadoop.services.repackaged.com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:67)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.executeWithBackoff(AgentApiAsyncUpdater.java:345)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.createAgent(AgentApiAsyncUpdater.java:234)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.getOrCreateAgent(AgentApiAsyncUpdater.java:203)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.run(AgentApiAsyncUpdater.java:183)
        at com.google.cloud.hadoop.services.repackaged.com.google.common.util.concurrent.MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.run(MoreExecutors.java:679)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Jul 24, 2021 11:03:23 AM com.google.cloud.hadoop.services.repackaged.com.google.cloud.hadoop.util.ResilientOperation nextSleep INFO: Transient exception caught. Sleeping for 1958, then retrying.
com.google.cloud.hadoop.services.repackaged.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at com.google.cloud.hadoop.services.repackaged.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:244)
        at com.google.cloud.hadoop.services.repackaged.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:225)
        at com.google.cloud.hadoop.services.repackaged.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:142)
        at com.google.cloud.dataproc.control.v1.AgentServiceGrpc$AgentServiceBlockingStub.createAgent(AgentServiceGrpc.java:735)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater$1.call(AgentApiAsyncUpdater.java:238)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater$1.call(AgentApiAsyncUpdater.java:235)
        at com.google.cloud.hadoop.services.repackaged.com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:67)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.executeWithBackoff(AgentApiAsyncUpdater.java:345)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.createAgent(AgentApiAsyncUpdater.java:234)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.getOrCreateAgent(AgentApiAsyncUpdater.java:203)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.run(AgentApiAsyncUpdater.java:183)
        at com.google.cloud.hadoop.services.repackaged.com.google.common.util.concurrent.MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.run(MoreExecutors.java:679)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.cloud.hadoop.services.repackaged.io.netty.channel.AbstractChannel$AnnotatedSocketException: Network is unreachable: dataproccontrol-europe-west1.googleapis.com/2a00:1450:400c:c04:0:0:0:5f:443
Caused by: java.net.SocketException: Network is unreachable
        at sun.nio.ch.Net.connect0(Native Method)
        at sun.nio.ch.Net.connect(Net.java:482)
        at sun.nio.ch.Net.connect(Net.java:474)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:91)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:88)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.internal.SocketUtils.connect(SocketUtils.java:88)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:315)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:248)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:533)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:54)
        at com.google.cloud.hadoop.services.repackaged.io.grpc.netty.WriteBufferingAndExceptionHandler.connect(WriteBufferingAndExceptionHandler.java:150)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.AbstractChannelHandlerContext.access$1000(AbstractChannelHandlerContext.java:61)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.AbstractChannelHandlerContext$9.run(AbstractChannelHandlerContext.java:538)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

请帮忙!

提前谢谢你

编辑 :我的防火墙和VPC

集群配置:

Answer 1

基于错误信息 Network is unreachable: dataproccontrol-europe-west1.googleapis.com/2a00:1450:400c:c04:0:0:0:5f:443以及您的网络设置，您似乎缺少通往互联网的路径。

您可以通过向 0.0.0.0/0 添加路由来解决此问题用于 IPv4 和 ::/0用于带 --next-hop-gateway=default-internet-gateway 的 IPv6 ，请参阅此 doc 中的更多详细信息.该路由应该是为新的 VPC 网络自动创建的，但我猜你删除了它，请参阅 doc .

需要路由的原因是虚拟机上的 Dataproc 代理需要访问 Dataproc 控制 API 以获取作业和报告状态。 API域名dataproccontrol-<region>.googleapis.com被解析为外部 IP，因此虚拟机需要有一个到互联网的路由(或 IP ranges )，但是当 Private Google Access启用后，流量将不会离开 Google 数据中心。 建议始终有通往互联网的路由，并使用防火墙规则进行更精细的访问控制。另请注意，默认情况下，没有外部 IP 的虚拟机无法访问互联网，即使路由和防火墙规则允许它，请参阅此 doc如果你想要一个解决方案。顺便说一句，您可以使用 Connectivity Test故障排除工具。