amazon-web-services - Storm 管理器连接错误从 nimbus 下载 jar

Question

我正在建立一个多节点 Storm 集群。所以我有 3 个 zookeeper 节点、1 个 nimbus、2 个主管和 1 个storm 客户端节点。因此，当我查看我与 zookeeper 和 nimbus & zookeeper 和 supervisor 的配置时，一切似乎都很好。但是当主管试图从 nimbus 数据目录中拉取 jar 文件时，主管会收到“连接被拒绝”。出于沮丧，我什至在盒子之间打开了 tcp 和 udp 端口​​(0-65535)，但我仍然以连接被拒绝而告终。

我已经验证了 nimbus 数据目录的权限是相当开放的，主管应该能够进入该目录并将文件拉下来。这是日志。

Nimbus.log:

2014-11-23 07:07:50 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2014-11-23 07:07:50 o.a.z.ClientCnxn [INFO] EventThread shut down
2014-11-23 07:07:50 o.a.z.ZooKeeper [INFO] Session: 0x249d964a3c20008 closed
2014-11-23 07:07:50 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2014-11-23 07:07:50 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=172.31.40.214:2181,172.31.45.110:2181,172.31.47.13:2181/storm sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@40160f3d
2014-11-23 07:07:50 o.a.z.ClientCnxn [INFO] Opening socket connection to server /172.31.40.214:2181
2014-11-23 07:07:50 o.a.z.ClientCnxn [INFO] Socket connection established to ip-172-31-40-214.us-west-2.compute.internal/172.31.40.214:2181, initiating session
2014-11-23 07:07:50 o.a.z.ClientCnxn [INFO] Session establishment complete on server ip-172-31-40-214.us-west-2.compute.internal/172.31.40.214:2181, sessionid = 0x149d964a86c001d, negotiated timeout = 20000
2014-11-23 07:07:50 b.s.d.nimbus [INFO] Delaying event :remove for 30 secs for TestingStormClusterTopology-1-1416724578
2014-11-23 07:07:50 b.s.d.nimbus [INFO] Starting Nimbus server...

2014-11-23 07:08:20 b.s.d.nimbus [INFO] Killing topology: TestingStormClusterTopology-1-1416724578
2014-11-23 07:08:22 b.s.d.nimbus [INFO] Cleaning up TestingStormClusterTopology-1-1416724578

2014-11-23 07:09:39 b.s.d.nimbus [INFO] Uploading file from client to /home/ubuntu/data/storm/nimbus/inbox/stormjar-dc265069-ebde-482f-abee-ccb7915fa663.jar
2014-11-23 07:09:39 b.s.d.nimbus [INFO] Finished uploading file from client: /home/ubuntu/data/storm/nimbus/inbox/stormjar-dc265069-ebde-482f-abee-ccb7915fa663.jar
2014-11-23 07:09:39 b.s.d.nimbus [INFO] Received topology submission for TestingStormClusterTopology with conf {"topology.max.task.parallelism" nil, "topology.acker.executors" nil, "topology.kryo.register" nil, "topology.kryo.decorators" (), "topology.name" "TestingStormClusterTopology", "storm.id" "TestingStormClusterTopology-1-1416726579", "topology.workers" 3}
2014-11-23 07:09:39 b.s.d.nimbus [INFO] Activating TestingStormClusterTopology: TestingStormClusterTopology-1-1416726579
2014-11-23 07:09:39 b.s.s.EvenScheduler [INFO] Available slots: (["30d36d53-ee60-4667-8a37-44c674da23e7" 6703] ["30d36d53-ee60-4667-8a37-44c674da23e7" 6702] ["30d36d53-ee60-4667-8a37-44c674da23e7" 6701] ["30d36d53-ee60-4667-8a37-44c674da23e7" 6700])
2014-11-23 07:09:39 b.s.d.nimbus [INFO] Setting new assignment for topology id TestingStormClusterTopology-1-1416726579: #backtype.storm.daemon.common.Assignment{:master-code-dir "/home/ubuntu/data/storm/nimbus/stormdist/TestingStormClusterTopology-1-1416726579", :node->host {"30d36d53-ee60-4667-8a37-44c674da23e7" "ip-172-31-43-254.us-west-2.compute.internal"}, :executor->node+port {[2 2] ["30d36d53-ee60-4667-8a37-44c674da23e7" 6702], [3 3] ["30d36d53-ee60-4667-8a37-44c674da23e7" 6701], [4 4] ["30d36d53-ee60-4667-8a37-44c674da23e7" 6703], [5 5] ["30d36d53-ee60-4667-8a37-44c674da23e7" 6702], [6 6] ["30d36d53-ee60-4667-8a37-44c674da23e7" 6701], [7 7] ["30d36d53-ee60-4667-8a37-44c674da23e7" 6703], [8 8] ["30d36d53-ee60-4667-8a37-44c674da23e7" 6702], [9 9] ["30d36d53-ee60-4667-8a37-44c674da23e7" 6701], [1 1] ["30d36d53-ee60-4667-8a37-44c674da23e7" 6703]}, :executor->start-time-secs {[1 1] 1416726579, [9 9] 1416726579, [8 8] 1416726579, [7 7] 1416726579, [6 6] 1416726579, [5 5] 1416726579, [4 4] 1416726579, [3 3] 1416726579, [2 2] 1416726579}}
2014-11-23 07:11:42 b.s.d.nimbus [INFO] Executor TestingStormClusterTopology-1-1416726579:[2 2] not alive
2014-11-23 07:11:42 b.s.d.nimbus [INFO] Executor TestingStormClusterTopology-1-1416726579:[3 3] not alive
2014-11-23 07:11:42 b.s.d.nimbus [INFO] Executor TestingStormClusterTopology-1-1416726579:[4 4] not alive
2014-11-23 07:11:42 b.s.d.nimbus [INFO] Executor TestingStormClusterTopology-1-1416726579:[5 5] not alive
2014-11-23 07:11:42 b.s.d.nimbus [INFO] Executor TestingStormClusterTopology-1-1416726579:[6 6] not alive
2014-11-23 07:11:42 b.s.d.nimbus [INFO] Executor TestingStormClusterTopology-1-1416726579:[7 7] not alive
2014-11-23 07:11:42 b.s.d.nimbus [INFO] Executor TestingStormClusterTopology-1-1416726579:[8 8] not alive
2014-11-23 07:11:42 b.s.d.nimbus [INFO] Executor TestingStormClusterTopology-1-1416726579:[9 9] not alive
2014-11-23 07:11:42 b.s.d.nimbus [INFO] Executor TestingStormClusterTopology-1-1416726579:[1 1] not alive
2014-11-23 07:11:42 b.s.d.nimbus [INFO] Setting new assignment for topology id TestingStormClusterTopology-1-1416726579: #backtype.storm.daemon.common.Assignment{:master-code-dir "/home/ubuntu/data/storm/nimbus/stormdist/TestingStormClusterTopology-1-1416726579", :node->host {}, :executor->node+port {}, :executor->start-time-secs {[1 1] 1416726579, [9 9] 1416726579, [8 8] 1416726579, [7 7] 1416726579, [6 6] 1416726579, [5 5] 1416726579, [4 4] 1416726579, [3 3] 1416726579, [2 2] 1416726579}}
2014-11-23 07:11:52 b.s.d.nimbus [INFO] Executor TestingStormClusterTopology-1-1416726579:[2 2] not alive
2014-11-23 07:11:52 b.s.d.nimbus [INFO] Executor TestingStormClusterTopology-1-1416726579:[3 3] not alive

这是 superisor.log 文件。

主管日志

2014-11-23 07:08:55 b.s.d.supervisor [INFO] Starting Supervisor with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/opt/jdk", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/home/ubuntu/data/storm", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["172.31.40.214" "172.31.45.110" "172.31.47.13"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "nimbus.host.ip" "172.31.47.40", "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed", "topology.optimize" true, "topology.max.task.parallelism" nil}
2014-11-23 07:08:56 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2014-11-23 07:08:56 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=172.31.40.214:2181,172.31.45.110:2181,172.31.47.13:2181 sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@76a78717
2014-11-23 07:08:56 o.a.z.ClientCnxn [INFO] Opening socket connection to server /172.31.47.13:2181
2014-11-23 07:08:56 o.a.z.ClientCnxn [INFO] Socket connection established to ip-172-31-47-13.us-west-2.compute.internal/172.31.47.13:2181, initiating session
2014-11-23 07:08:56 o.a.z.ClientCnxn [INFO] Session establishment complete on server ip-172-31-47-13.us-west-2.compute.internal/172.31.47.13:2181, sessionid = 0x349d964c0d30018, negotiated timeout = 20000
2014-11-23 07:08:56 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2014-11-23 07:08:56 o.a.z.ClientCnxn [INFO] EventThread shut down
2014-11-23 07:08:56 o.a.z.ZooKeeper [INFO] Session: 0x349d964c0d30018 closed
2014-11-23 07:08:56 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2014-11-23 07:08:56 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=172.31.40.214:2181,172.31.45.110:2181,172.31.47.13:2181/storm sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@603043f6
2014-11-23 07:08:56 o.a.z.ClientCnxn [INFO] Opening socket connection to server /172.31.40.214:2181
2014-11-23 07:08:56 o.a.z.ClientCnxn [INFO] Socket connection established to ip-172-31-40-214.us-west-2.compute.internal/172.31.40.214:2181, initiating session
2014-11-23 07:08:56 o.a.z.ClientCnxn [INFO] Session establishment complete on server ip-172-31-40-214.us-west-2.compute.internal/172.31.40.214:2181, sessionid = 0x149d964a86c001f, negotiated timeout = 20000
2014-11-23 07:08:56 b.s.d.supervisor [INFO] Starting supervisor with id 30d36d53-ee60-4667-8a37-44c674da23e7 at host ip-172-31-43-254.us-west-2.compute.internal


2014-11-23 07:09:39 b.s.d.supervisor [INFO] Downloading code for storm id TestingStormClusterTopology-1-1416726579 from /home/ubuntu/data/storm/nimbus/stormdist/TestingStormClusterTopology-1-1416726579
2014-11-23 07:09:39 b.s.event [ERROR] Error when processing event
java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
    at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:21) ~[storm-core-0.9.0.1.jar:na]
    at backtype.storm.utils.Utils.downloadFromMaster(Utils.java:226) ~[storm-core-0.9.0.1.jar:na]
    at backtype.storm.daemon.supervisor$fn__6326.invoke(supervisor.clj:396) ~[storm-core-0.9.0.1.jar:na]
    at clojure.lang.MultiFn.invoke(MultiFn.java:172) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__6251.invoke(supervisor.clj:290) ~[storm-core-0.9.0.1.jar:na]
    at backtype.storm.event$event_manager$fn__3072.invoke(event.clj:24) ~[storm-core-0.9.0.1.jar:na]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
    at org.apache.thrift7.transport.TSocket.open(TSocket.java:183) ~[libthrift7-0.7.0-2.jar:0.7.0-2]
    at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81) ~[libthrift7-0.7.0-2.jar:0.7.0-2]
    at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:66) ~[storm-core-0.9.0.1.jar:na]
    at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46) ~[storm-core-0.9.0.1.jar:na]
    at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:30) ~[storm-core-0.9.0.1.jar:na]
    at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:26) ~[storm-core-0.9.0.1.jar:na]
    at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:19) ~[storm-core-0.9.0.1.jar:na]
    ... 7 common frames omitted
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_65]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[na:1.7.0_65]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[na:1.7.0_65]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[na:1.7.0_65]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.7.0_65]
    at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_65]
    at org.apache.thrift7.transport.TSocket.open(TSocket.java:178) ~[libthrift7-0.7.0-2.jar:0.7.0-2]
    ... 13 common frames omitted
2014-11-23 07:09:39 b.s.util [INFO] Halting process: ("Error when processing an event")

所以我想了解我是否必须在这些盒子之间共享公钥和/或私钥。我知道如何生成公钥(ssh-keygen)，但我不确定跨盒子共享 key 的策略应该是什么。

我什至不确定这是否是问题所在，我只是对连接被拒绝错误意味着什么感到困惑。为这篇长文道歉，但我想提供尽可能多的信息。

Answer 1

由于 nimbus 出现问题，主管无法在提供给主管进行连接的端口上找到连接。