csv - 尝试索引CSV文件时Logstash没有响应

Question

我有一个具有以下结构的CSV文件

col1, col2, col3 
1|E|D
2|A|F
3|E|F
... 

我试图使用logstash在ElasticSearch上为其编制索引，所以我创建了以下logstash配置文件:

input {
  file {
    path => "/path/to/data"
    start_position => "beginning"
  }
}
filter {
  csv {
      separator => "|"
     columns => ["col1","col2","col3"]
}
}
output {
   elasticsearch {
     hosts => ["localhost:9200"]
     index => "myindex"
     document_type => "mydoctype"
  }
stdout {}
}

但是logstash暂停，除了以下情况之外，没有其他消息:

$ /opt/logstash/bin/logstash -f logstash.conf
Settings: Default pipeline workers: 8
Pipeline main started

增加详细程度会显示以下消息(不包含任何特定错误)

$ /opt/logstash/bin/logstash -v -f logstash.conf
starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 8
Registering file input {:path=>["/path/to/data"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/home/username/.sincedb_55b24c6ff18079626c5977ba5741584a", :path=>["/path/to/data"], :level=>:info}
Using mapping template from {:path=>nil, :level=>:info}
Attempting to install template {:manage_template=>{"template"=>"logstash-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"string", "index"=>"not_analyzed"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"float"}, "longitude"=>{"type"=>"float"}}}}}}}, :level=>:info}
New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["localhost:9200"], :level=>:info}
Starting pipeline {:id=>"main", :pipeline_workers=>8, :batch_size=>125, :batch_delay=>5, :max_inflight=>1000, :level=>:info}
Pipeline main started

关于如何为csv文件建立索引的任何建议？

Answer 1

如果在测试期间您之前已经处理过该文件，logstash将在输出所引用的sincedb文件中保留该记录(inode和字节偏移)。您可以删除文件(如果不需要)，或在文件{}输入中设置sincedb_path。