mysql - 使用 JDBC 抓取 mysql 查询并将其发送到 elasticsearch

Question

早上好，我正在尝试从 MySQL 数据库获取数据并将其发送到 elasticsearch，这样我就可以在 Kibana 中创建图表并显示它。

我正在使用带有此配置的elasticsearch-jdbc插件:

input {
  jdbc {
    jdbc_driver_library => "/home/USERNAME/mysql-connector-java-5.1.40/mysql-connector-java-5.1.40-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/DB_NAME"
    jdbc_user => "USER"
    jdbc_password => "PASSWORD"
    schedule => "* * * * *"
    statement => "SELECT * from eps"
  }
}

output {
  elasticsearch {
       hosts => ["localhost:9200"]
       sniffing => false
       index => "eps"
       document_type => "eps"
       document_id => "%{uid}"
       }
}

但没有成功。当我查看/var/log/logstash/logstash-plain.log 时我收到此错误:

[2016-11-17T11:54:17,592][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n  beats {\n    port => 5044\n   y\n }\n}\n\noutput {\n  elasticsearch {\n    hosts => [\"localhost:9200\"]\n    manage_template => false\n    index => \"%{[@metadata][beat]}-%{+YYYY.MM.dd}\"\n    document_type => \"%{[@metadata][type]}\"\n  }\n}\n\ninput {\n  jdbc {\n    jdbc_driver_library => \"/home/USERNAME/mysql-connector-java-5.1.40/mysql-connector-java-5.1.40-bin.jar\"\n    jdbc_driver_class => \"com.mysql.jdbc.Driver\"\n    jdbc_connection_string => \"jdbc:mysql://localhost:3306/DB_NAME\"\n    jdbc_user => \"USER\"\n    jdbc_password => \"PASSWORD\"\n    schedule => \"* * * * *\"\n    statement => \"SELECT * from eps\"\n  }\n}\n\noutput {\n  elasticsearch {\n       hosts => [\"localhost:9200\"]\n       sniffing => false\n       index => \"eps\"\n       document_type => \"eps\"\n       document_id => \"%{uid}\"\n       }\n}\n\n", :reason=>"Expected one of #, => at line 5, column 2 (byte 42) after input {\n  beats {\n    port => 5044\n   y\n "}

我使用以下方法创建了新的 ID 名称:

curl -XPUT 'localhost:9200/river/my_jdbc_river/_meta' -d '{
>     "type" : "jdbc",
>     "jdbc" : {
>         "url" : "jdbc:mysql://localhost:3306/MY_DB",
>         "user" : "USER",
>         "password" : "PASSWORD",
>         "sql" : "select * from eps"
>     }
> }'

我已经为这篇文章更改了用户名和密码，但我确定 - 并且它们有效。不是那样的。

知道我做错了什么吗？谢谢伊丹

Answer 1

错误消息表明您的配置无效。

从错误消息中，我们可以看到:

input {
  beats {
    port => 5044
    y
  }
}

y 在这里没有任何关系，问题就出在它身上。它也在错误消息的末尾。