mysql - Logstash 未从 MySQL 填充 ElasticSearch

Question

我目前正处于使用 ELK 堆栈和 MySQL 数据库设置一些 Kibana 仪表板的早期阶段。根据logstash配置检查实用程序，我有一个有效的.conf文件，但弹性中没有显示任何内容。

首先，我的数据库已填充:

mysql> SELECT COUNT(session_id) AS session_id FROM scans;
+------------+
| session_id |
+------------+
|          6 |
+------------+
1 row in set (0.00 sec)

这是我的logstash conf 文件:

input {
        jdbc {
           jdbc_connection_string => "jdbc:mysql://localhost:3306/dashboarddb"
           jdbc_user => "user"
           jdbc_password => "password"
           jdbc_driver_library => "/home/ubuntu/mysql-connector-java-8.0.16/mysql-connector-java-8.0.16.jar"
           jdbc_driver_class => "com.mysql.jdbc.Driver"
           statement => "SELECT * FROM scans;"
         }
}

output {
         elasticsearch {
             hosts => "localhost:9200"
             index => "scans"
             document_id => "%{session_id}"
       }

}

当我启动logstash时:

[2019-06-07T19:30:45,740][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.1.1"}
[2019-06-07T19:30:51,727][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-06-07T19:30:51,924][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-06-07T19:30:51,973][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-06-07T19:30:51,976][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2019-06-07T19:30:52,002][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-06-07T19:30:52,022][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-06-07T19:30:52,029][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x3cc19c34 run>"}
[2019-06-07T19:30:52,148][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-06-07T19:30:52,237][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"main"}
[2019-06-07T19:30:52,344][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-06-07T19:30:52,808][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-06-07T19:30:53,742][INFO ][logstash.inputs.jdbc     ] (0.018699s) SELECT * FROM scans;
[2019-06-07T19:30:55,542][INFO ][logstash.runner          ] Logstash shut down.

当我检查弹性时:

/var/log/logstash$ curl -H "Content-Type: application/json" -XGET '127.0.0.1:9200/scans/_search?q=Something&pretty'
{
  "took" : 826,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

所以真的不知道从这里去哪里 - 任何帮助将不胜感激!

Answer 1

从提供的日志输出中可以看到，logstash 在管道尝试执行 sql 查询后立即关闭。这解释了为什么数据没有发送到elasticsearch。正如 ibexit 建议的那样，请在日志记录中启用调试级别并发布相同执行步骤的输出。