azure - Elk Logstash 无法连接到事件中心 Azure

Question

我正在尝试从 ElasticSearch Logstash 连接到 Azure 事件中心。这是我在/etc/logstash/logstash.yml 文件中提到的 azure 模块的详细信息。

modules:
  - name: azure
    var.elasticsearch.hosts: localhost:9200
    var.kibana.host: localhost:5601
    var.input.azure_event_hubs.consumer_group: "default"
    var.input.azure_event_hubs.storage_connection: "DefaultEndpointsProtocol=https;AccountName=ABC;AccountKey=XXXXX==;EndpointSuffix=core.windows.net"
    var.input.azure_event_hubs.threads: 9
    var.input.azure_event_hubs.event_hub_connections:
      - "Endpoint=sb://elk-event.servicebus.windows.net/;SharedAccessKeyName=share-log;SharedAccessKey=YYYYY=;EntityPath=elk-log-event"

但是从我使用命令启动logstash时开始:

./logstash --path.settings=/etc/logstash

我使用consumer_group作为logstash以及默认值。这是一个问题吗？

它抛出错误:

[2019-10-04T09:43:46,332][ERROR][com.microsoft.azure.eventprocessorhost.PartitionPump] host logstash-8331671e-0dea-4f8d-88de-ba3ebc974dca: 1: PartitionReceiver creation failed
java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.IllegalEntityException: The messaging entity 'elk-event:eventhub:elk-log-event~32766|default' could not be found. TrackingId:c380d842-e946-4f12-ba12-daecdf60df2d_B29, SystemTracker:elk-event:eventhub:elk-log-event~32766|default, Timestamp:2019-10-04T09:43:45 TrackingId:778a3beb42734bc098c7419f5f4b8c21_G5, SystemTracker:gateway5, Timestamp:2019-10-04T09:43:45, errorContext[NS: elk-event.servicebus.windows.net, PATH: elk-log-event/ConsumerGroups/default/Partitions/1, REFERENCE_ID: e393e0_8c21_G5_1570182225737]
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_222]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_222]
        at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:647) ~[?:1.8.0_222]
        at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) ~[?:1.8.0_222]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_222]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_222]
        at com.microsoft.azure.eventhubs.impl.ExceptionUtil.completeExceptionally(ExceptionUtil.java:98) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.MessageReceiver.cancelOpen(MessageReceiver.java:357) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.MessageReceiver.onOpenComplete(MessageReceiver.java:347) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.MessageReceiver.onError(MessageReceiver.java:414) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:736) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.processOnClose(BaseLinkHandler.java:74) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.handleRemoteLinkClosed(BaseLinkHandler.java:113) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:48) ~[azure-eventhubs-2.2.0.jar:?]
        at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176) ~[proton-j-0.33.0.jar:?]
        at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) ~[proton-j-0.33.0.jar:?]
        at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) ~[proton-j-0.33.0.jar:?]
        at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291) ~[proton-j-0.33.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:507) ~[azure-eventhubs-2.2.0.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: com.microsoft.azure.eventhubs.IllegalEntityException: The messaging entity 'elk-event:eventhub:elk-log-event~32766|default' could not be found. TrackingId:c380d842-e946-4f12-ba12-daecdf60df2d_B29, SystemTracker:elk-event:eventhub:elk-log-event~32766|default, Timestamp:2019-10-04T09:43:45 TrackingId:778a3beb42734bc098c7419f5f4b8c21_G5, SystemTracker:gateway5, Timestamp:2019-10-04T09:43:45, errorContext[NS: elk-event.servicebus.windows.net, PATH: elk-log-event/ConsumerGroups/default/Partitions/1, REFERENCE_ID: e393e0_8c21_G5_1570182225737]
        at com.microsoft.azure.eventhubs.impl.ExceptionUtil.distinguishNotFound(ExceptionUtil.java:86) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:31) ~[azure-eventhubs-2.2.0.jar:?]
        at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:735) ~[azure-eventhubs-2.2.0.jar:?]
        ... 15 more

与消费者群体或其他什么有关？如果是这样，我不知道如何创建自己的消费者组。

有任何反馈吗？提前致谢。

Answer 1

消费者组的名称必须是$Default

//编辑:我应该注意，通常不建议使用 $Default 消费者组。最佳实践是为每个从事件中心读取数据的应用程序创建一个新的使用者组 - 即使只有一个应用程序。您永远不知道什么时候有人可能有另一个想法并且也想读取数据...此外，指定消费者组可以让您轻松概览应用程序正在从您的中心读取哪些内容。