elasticsearch - 在 Elasticsearch 输出 ILM 翻转别名中使用变量

Question

我无法获得 ILM 翻转别名来接受变量。在这个特定的例子中，我们将有一个 ELK 集群托管多个环境的日志。日志条目将在进入 logstash 管道之前用其环境标记。我希望条目转到正确的别名，但是在启动 logstash 时出现以下错误(截断了堆栈跟踪):

An unexpected error occurred! {:error=>java.net.URISyntaxException: Malformed escape pair at index 0: %{[fields][Environment]}-logs

这是我的 logstash 管道:

input { 
  rabbitmq {
    host => "rabbitmq"
    port => 5672
    user => "guest"
    password => "guest"
    subscription_retry_interval_seconds => 5
    queue => "logstash-queue"
    exchange => "logs"
    exchange_type => "direct"
    durable => true
    key => "logstash"
  }
}

filter {
  mutate {
    rename => {"Properties" => "fields"}
  }
  mutate {
    lowercase => ["[fields][Environment]"]
  }
}

output {
  elasticsearch {
      hosts => ["http://elasticsearch:9200"]
      template_name=>"app-logs"
      ilm_enabled => true
      ilm_rollover_alias => "%{[fields][Environment]}-logs"
      ilm_pattern => "{now/d}-000001"
      ilm_policy => "30_day_retention_logs_policy"
    }
}

Answer 1

根据 https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_writing_to_different_indices_best_practices

You cannot use dynamic variable substitution when ilm_enabled is true and when using ilm_rollover_alias.

建议的解决方案(在撰写本文时)是使用多个输出。我建议自动化 logstash.conf 、salt、ansible 等生成，这样您就不必继续手动编辑配置文件。

output {
        if <condition> {
            elasticsearch {
                ...
                index => "logstash-<env>-logs"
                ilm...
            }
<etc>

在弹性 github 上存在与此相关的问题 - 这种灵活性被删除确实很奇怪，但是你去了。