gpt4 book ai didi

elasticsearch - 无法删除 grok 过滤器不匹配的事件,logstash, Elasticsearch

转载 作者:行者123 更新时间:2023-11-29 02:53:58 25 4
gpt4 key购买 nike

我正在尝试解析 tomcat 日志并将输出传递给 Elasticsearch 。或多或少它运作良好。当我看到 Elasticsearch 索引数据时,它包含大量匹配数据,标签字段为 _grokparsefailure。这会导致大量重复的匹配数据。为避免这种情况,我尝试在标签包含 _grokparsefailure 时删除事件。此配置写在 grok 过滤器下面的 logstash.conf 文件中。仍然输出到 Elasticsearch 包含索引文档,其中包含带有 _grokparsefailure 的标签。如果 grok 失败,我不希望该匹配项进入 Elasticsearch ,因为它会在 Elasticsearch 中导致重复数据。

logstash.conf 文件是:

input {

file {

path => "/opt/elasticSearch/logstash-1.4.2/input.log"
codec => multiline {
pattern => "^\["
negate => true
what => previous
}
start_position => "end"

}

}

filter {

grok {

match => [
"message", "^\[%{GREEDYDATA}\] %{GREEDYDATA} Searching hotels for country %{GREEDYDATA:country}, city %{GREEDYDATA:city}, checkin %{GREEDYDATA:checkin}, checkout %{GREEDYDATA:checkout}, roomstay %{GREEDYDATA:roomstay}, No. of hotels returned is %{NUMBER:hotelcount} ."
]

}

if "_grokparsefailure" in [tags]{

drop { }

}

}

output {

file {
path => "/opt/elasticSearch/logstash-1.4.2/output.log"
}

elasticsearch {
cluster => "elasticsearchdev"
}

}

Elasticsearch 响应 http://172.16.37.97:9200/logstash-2015.12.23/_search?pretty=true

下面给出的输出包含三个文档,其中第一个在 _source -> tags 字段中包含 _grokparsefailure。

我不希望它出现在这个输出中。所以可能需要从 logstash 中限制它,这样它就不会进入 Elasticsearch 。

{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.0,
"hits" : [

{

"_index" : "logstash-2015.12.23",
"_type" : "logs",
"_id" : "J6CoEhKaSE68llz5nEbQSQ",
"_score" : 1.0,
"_source":{"message":"[2015-12-23 12:08:40,124] ERROR http-80-5_@{AF3AF784EC08D112D5D6FC92C78B5161,127.0.0.1,1450852688060} com.mmt.hotels.web.controllers.search.HotelsSearchController - Searching hotels for country IN, city DEL, checkin 28-03-2016, checkout 29-03-2016, roomstay 1e0e, No. of hotels returned is 6677 .","@version":"1","@timestamp":"2015-12-23T14:17:03.436Z","host":"ggn-37-97","path":"/opt/elasticSearch/logstash-1.4.2/input.log","tags":["_grokparsefailure"]}

},

{

"_index" : "logstash-2015.12.23",
"_type" : "logs",
"_id" : "2XMc6nmnQJ-Bi8vxigyG8Q",
"_score" : 1.0,
"_source":{"@timestamp":"2015-12-23T14:17:02.894Z","message":"[2015-12-23 12:08:40,124] ERROR http-80-5_@{AF3AF784EC08D112D5D6FC92C78B5161,127.0.0.1,1450852688060} com.mmt.hotels.web.controllers.search.HotelsSearchController - Searching hotels for country IN, city DEL, checkin 28-03-2016, checkout 29-03-2016, roomstay 1e0e, No. of hotels returned is 6677 .","@version":"1","host":"ggn-37-97","path":"/opt/elasticSearch/logstash-1.4.2/input.log","country":"IN","city":"DEL","checkin":"28-03-2016","checkout":"29-03-2016","roomstay":"1e0e","hotelcount":"6677"}

},

{

"_index" : "logstash-2015.12.23",
"_type" : "logs",
"_id" : "fKLqw1LJR1q9YDG2yudRDw",
"_score" : 1.0,
"_source":{"@timestamp":"2015-12-23T14:16:12.684Z","message":"[2015-12-23 12:08:40,124] ERROR http-80-5_@{AF3AF784EC08D112D5D6FC92C78B5161,127.0.0.1,1450852688060} com.mmt.hotels.web.controllers.search.HotelsSearchController - Searching hotels for country IN, city DEL, checkin 28-03-2016, checkout 29-03-2016, roomstay 1e0e, No. of hotels returned is 6677 .","@version":"1","host":"ggn-37-97","path":"/opt/elasticSearch/logstash-1.4.2/input.log","country":"IN","city":"DEL","checkin":"28-03-2016","checkout":"29-03-2016","roomstay":"1e0e","hotelcount":"6677"}

} ]
}
}

]

最佳答案

您可以尝试在 output 部分测试 _grokparsefailure,如下所示:

output {
if "_grokparsefailure" not in [tags] {
file {
path => "/opt/elasticSearch/logstash-1.4.2/output.log"
}

elasticsearch {
cluster => "elasticsearchdev"
}
}
}

关于elasticsearch - 无法删除 grok 过滤器不匹配的事件,logstash, Elasticsearch ,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/34437902/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com