elasticsearch - Logstash输入->某些属性或可参数化文件中的JDBC？

Question

我正在使用logstash来摄取elasticsearch。我正在使用输入jdbc，需要对输入的jdbc设置进行参数化，例如连接字符串，传递等，这对我有很大的帮助，因为我有10个.conf文件，其中每个文件都有30个jdbc和30个输出。

因此，由于每个文件都具有相同的设置，您是否想知道是否可以做一些常规操作或从某个地方引用该信息？

我有30次:...

input {
  # Number 1
  jdbc {
        jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/ifxjdbc-4.50.3.jar"
        jdbc_driver_class => "com.informix.jdbc.IfxDriver"
        jdbc_connection_string => "jdbc:informix-sqli://xxxxxxx/schema:informixserver=server"
        jdbc_user => "xxx"
        jdbc_password => "xxx"
        schedule => "*/1 * * * *"                    
        statement => "SELECT * FROM public.test ORDER BY id ASC"
        tags => "001"
  }

  # Number 2
  jdbc {
        jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/ifxjdbc-4.50.3.jar"
        jdbc_driver_class => "com.informix.jdbc.IfxDriver"
        jdbc_connection_string => "jdbc:informix-sqli://xxxxxxx/schema:informixserver=server"
        jdbc_user => "xxx"
        jdbc_password => "xxx"
        schedule => "*/1 * * * *"                    
        statement => "SELECT * FROM public.test2 ORDER BY id ASC"
        tags => "002"
  }


  [.........]

  # Number X
  jdbc {
        jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/ifxjdbc-4.50.3.jar"
        jdbc_driver_class => "com.informix.jdbc.IfxDriver"
        jdbc_connection_string => "jdbc:informix-sqli://xxxxxxx/schema:informixserver=server"
        jdbc_user => "xxx"
        jdbc_password => "xxx"
        schedule => "*/1 * * * *"                    
        statement => "SELECT * FROM public.testx ORDER BY id ASC"
        tags => "00x"
  }

}

filter { 

  mutate { 
    add_field => { "[@metadata][mitags]" => "%{tags}" }
  }

  # Number 1
  if "001" in [@metadata][mitags] {


        mutate { 
                  rename => [ "codigo", "[properties][codigo]" ] 
            }
  }

  # Number 2
  if "002" in [@metadata][mitags] {


        mutate { 
                  rename => [ "codigo", "[properties][codigo]" ] 
            }
  }

  [......]

  # Number x
  if "002" in [@metadata][mitags] {


        mutate { 
                  rename => [ "codigo", "[properties][codigo]" ] 
            }
  }


  mutate {
    remove_field => [ "@version","@timestamp","tags" ]
  }



} 

output {

  # Number 1
  if "001" in [@metadata][mitags] {        
        # Para ELK
        elasticsearch {
              hosts => "localhost:9200"
              index => "001"
              document_type => "001"
              document_id => "%{id}"

              manage_template => true
              template => "/home/user/logstash/templates/001.json"
              template_name => "001"
              template_overwrite => true
        }
  } 

   # Number 2
  if "002" in [@metadata][mitags] {        
        # Para ELK
        elasticsearch {
              hosts => "localhost:9200"
              index => "002"
              document_type => "002"
              document_id => "%{id}"

              manage_template => true
              template => "/home/user/logstash/templates/002.json"
              template_name => "002"
              template_overwrite => true
        }
  }

  [....]

   # Number x
  if "00x" in [@metadata][mitags] {        
        # Para ELK
        elasticsearch {
              hosts => "localhost:9200"
              index => "002"
              document_type => "00x"
              document_id => "%{id}"

              manage_template => true
              template => "/home/user/logstash/templates/00x.json"
              template_name => "00x"
              template_overwrite => true
        }
  }

}

Answer 1

您仍然需要为每个查询输入一个jdbc输入，但是您可以改善filter和output块。

在filter块中，您使用场[@metadata][mitags]过滤输入，但是对每个输入应用相同的mutate过滤器，如果您不需要条件，则可以将相同的mutate过滤器应用于如果您不过滤所有输入内容。

您的filter块可以恢复为这一块。

filter {
    mutate { 
        add_field => { "[@metadata][mitags]" => "%{tags}" }
    }
    mutate { 
        rename => [ "codigo", "[properties][codigo]" ] 
    }
    mutate {
        remove_field => [ "@version","@timestamp","tags" ]
    }
}

在 output块中，您可以使用标记仅更改 index， document_type和 template，而无需为此使用条件，可以将字段的值用作参数。

output {
    elasticsearch {
        hosts => "localhost:9200"
        index => "%{[@metadata][mitags]}"
        document_type => "%{[@metadata][mitags]}"
        document_id => "%{id}"
        manage_template => true
        template => "/home/unitech/logstash/templates/%{[@metadata][mitags]}.json"
        template_name => "iol-fue"
        template_overwrite => true
    }
}

但这仅在 [@metadata][mitags]字段中只有一个值的情况下才有效，似乎是这种情况。

编辑:
仅出于历史原因进行编辑(如注释中所述)， template配置不允许使用动态参数，因为仅在启动logstash时才加载动态参数，其他配置也可以正常工作。