apache-storm - 以编程方式获取 Storm 拓扑统计信息

Question

我正在围绕我的 Storm 拓扑构建一个监控服务，并希望能够获取各个时间窗口周围的失败元组数量，类似于 Storm UI 如何在 10m、3h 和 1d 窗口中显示失败元组的数量。

我的监控服务目前是用 python 构建的，所以如果答案涉及 python 库或语言不可知的东西，比如炮弹到 CLI 或击中 REST 端点，我将不胜感激。我查看了 Storm CLI 和文档，但到目前为止，我对 Storm UI 实际从何处获取信息一无所知。

编辑:
- 运行 Storm 0.8.2 版本(不幸的是我无法控制)，因此在进行升级之前，storm-ui-rest-api(在 0.9.2 中发布)不是一个选项。

Answer 1

用 Storm UI Rest API

sqlInjection@foo:~$ curl http://$STORM_UI_HOST_AND_PORT/api/v1/topology/summary

{"topologies":[{"id":"topology-1-1436004781","encodedId":"topology-1-1436004781","encodedId","name":"topology-1","status":"ACTIVE","uptime":"40d 21h 51m 59s","tasksTotal":16,"workersTotal":1,"executorsTotal":10}]}

sqlInjection@foo:~$ curl http://$STORM_UI_HOST_AND_PORT/api/v1/topology/topology-1-1436004781

{"msgTimeout":30,"spouts":[{"executors":3,"emitted":22336820,"errorLapsedSecs":755996,"completeLatency":"232.052","transferred":22336820,"acked":22340300,"errorPort":6703,"spoutId":"KafkaSpout-removed","tasks":3,"errorHost":"removed","lastError":"java.lang.RuntimeException: java.lang.NullPointerException\n\tat backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)\n\tat backtype.storm.utils.DisruptorQueue.consumeBatch(Di","errorWorkerLogLink":"http://host:port/log?file=worker-6703.log","failed":0,"encodedSpoutId":"KafkaSpout-removed"}],"executorsTotal":8,"uptime":"67d 21h 15m 2s","encodedId":"topology-1-1436004781","visualizationTable":[{":row":[{":stream":"default",":sani-stream":"default1544803905",":checked":true},{":stream":"__ack_init",":sani-stream":"s__ack_init973324006",":checked":false},{":stream":"__ack_ack",":sani-stream":"s__ack_ack1278315507",":checked":false},{":stream":"__ack_fail",":sani ... removed

如您所见，您甚至可以捕捉到您的 bolts/spouts 中发生的最后一个错误。