elasticsearch - logstash 5.0.1 : setup elasticsearch multiple indexes ouput for multiple kafka input topics

Question

我有一个 logstash 输入设置为

input {
  kafka {
  bootstrap_servers => "zookeper_address"
  topics => ["topic1","topic2"]
  }
}

我需要将主题提供给 elasticsearch 中的两个不同索引。任何人都可以帮助我如何为这样的任务设置输出。目前我只能设置

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "my_index"
    codec => "json"
    document_id => "%{id}"
  }
}

我需要在同一个 elasticsearch 实例上使用两个索引，比如 index1 和 index2，它们将由来自 topic1 和 的消息提供>主题2

Answer 1

首先，您需要将 decorate_events 添加到您的 kafka 输入中，以便了解消息来自哪个主题

input {
  kafka {
    bootstrap_servers => "zookeper_address"
    topics => ["topic1","topic2"]
    decorate_events => true
  }
}

然后，您有两个选项，都涉及条件逻辑。第一种是引入过滤器，根据主题名称添加正确的索引名称。为此你需要添加

filter {
   if [kafka][topic] == "topic1" {
      mutate {
         add_field => {"[@metadata][index]" => "index1"}
      }
   } else {
      mutate {
         add_field => {"[@metadata][index]" => "index2"}
      }
   }
   # remove the field containing the decorations, unless you want them to land into ES
   mutate {
      remove_field => ["kafka"]
   }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{[@metadata][index]}"
    codec => "json"
    document_id => "%{id}"
  }
}

然后第二个选项是直接在输出部分执行 if/else，就像这样(但是额外的 kafka 字段将进入 ES):

output {
   if [@metadata][kafka][topic] == "topic1" {
     elasticsearch {
       hosts => ["localhost:9200"]
       index => "index1"
       codec => "json"
       document_id => "%{id}"
     }
   } else {
     elasticsearch {
       hosts => ["localhost:9200"]
       index => "index2"
       codec => "json"
       document_id => "%{id}"
     }
   }
}