csv - 无法使用Logstash解析CSV文件

Question

我在使用logstash将CSV文件导入ElasticSearch中时遇到问题，以便在Kibana中进行进一步处理。

这是我的logstash配置文件:

input {
    file {
        path => ["/absolute_path_to_file/export.csv"]
        start_position => beginning 
        ignore_older => 0 
        sincedb_path => "/dev/null"
    }
}
#filter {
#   csv {
#       columns => [
#           "id",
#           "cislo_smlouvy",
#           "zdroj",
#           "produkt",
#           "sjednani",
#           "datum_odeslani",
#           "cas_odeslani",
#           "pojistovna",
#           "tarif",
#           "pojistnik",
#           "telefon",
#           "predmet_pojisteni",
#           "rz",
#           "pocatek_pojisteni",
#           "rocni_pojistne",
#           "urgence",
#           "stav"
#       ]
#       separator => ";"
#       remove_field => ["message"]
#   }
#}
output {
#   elasticsearch {
#       hosts => "localhost:9200"
#       index => "smlouvy"
#   }
    stdout {
        codec => rubydebug
    }
}

摘录自我的CSV文件:

"id";"číslo smlouvy";"zdroj";"produkt";"sjednání";"datum odeslaní";"čas odeslání";"pojišťovna";"tarif";"pojistník";"pojistnik telefon";"předmět pojištění";"rz";"počátek";"roční pojistné";"urgence";"stav"
"114951";"6132681255";"SRO";"POV";;"1.6.2016";"12:28";"csob";"csob-2";"BB TEST";"721666333";"Škoda Favorit";"NENÍ";"2.6.2016 00:00";"4657,00";;"TEST"
"114950";;"POV";"POV";"VO Bukvicova";"1.6.2016";"12:16";"csob";"csob-2";"BB BB";"721000111";"BMW X3";"NENÍ";"3.6.2016 00:00";"5550,00";;"TEST"

我正在调用此命令: sudo logstash -f /absolute_path_to_file/logstash.conf --vebrose
具有以下输出:

starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 2
Registering file input {:path=>["/absolute_path_to_file/export.csv"], :level=>:info}
Starting pipeline {:id=>"main", :pipeline_workers=>2, :batch_size=>125, :batch_delay=>5, :max_inflight=>250, :level=>:info}
Pipeline main started

一会儿什么都不做之后，我关闭了它:

^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>"main"}
Closing inputs {:level=>:info}
Closed inputs {:level=>:info}
Input plugins stopped! Will shutdown filter/output workers. {:level=>:info}
Pipeline main has been shutdown

可能相关的版本信息:

logstash 2.3.2
logstash-input-file (2.2.5)
logstash-filter-csv (2.1.3)
logstash-output-elasticsearch (2.6.2)
logstash-output-stdout (2.0.6)
logstash-codec-rubydebug (2.0.7)

我已经阅读了所有可以找到的文档，并尝试从GitHub复制很多logstash.conf示例，但是没有运气。有什么我想念的帮助吗？

Answer 1

所以我终于找到了问题所在。它与输入的CSV文件相同。

CSV文件的新行包含\r，logstash默认值为\n。

顺便说一句:您不能在logstash文件输入过滤器配置中将\r设置为定界符，因此我不得不使用\n将CSV文件转换为一个