elasticsearch - logstash始终只在我的索引中放置1个条目

Question

我有以下logstash conf文件:

input {
  jdbc {
    jdbc_driver_library => "C:\Program Files\Microsoft JDBC DRIVER 6.2 for SQL Server\sqljdbc_6.2\enu\mssql-jdbc-6.2.1.jre8"
    jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
    jdbc_connection_string => "jdbc:sqlserver://TST-DBS-20;user=Elasticsearch;password=elastic123;"
    jdbc_user => "Elasticsearch"
    statement => "SELECT NewsID, HeadLine, BodyText, DateSubmitted, Approved, NULLIF(UserName,'') as UserName, NULLIF(Type,'') as NewsType, NULLIF(Caption,'') as caption, NULLIF(Author,'') as Author, NULLIF(Contact,'') as Contact, NULLIF(StaffID,'') as StaffID, SocialClubRegionID, DateCreated, CreatedBy, LastModifiedDate, ModifiedBy
FROM [News].[dbo].[News]"
  }
}
filter {
}
output {
  elasticsearch {
    hosts => ["tst-sch-20:9200"]
    index => "newsindex"
    document_id => "%{id}"
    user => "elastic"
    password => elastic123
  }
  stdout { codec => json }
}

并且我创建了以下索引:

put newsindex
{
  "settings" : {
    "number_of_shards":3,
    "number_of_replicas":2
  },
  "mappings" : {
    "news": {
      "properties": {
        "NewsId": {
          "type": "integer"
        },
        "newstype": {
          "type": "text"
        },
        "bodytext": {
          "type": "text"
        }
      }
    }
  }
}

运行上述脚本后，logstash日志文件中没有任何条目可指示发生任何错误。如果我直接在SQL中运行SQL命令，那么奇怪的是，索引中的单个条目是我的select语句的最后一个条目，因此几乎就像脚本正在插入然后覆盖一样，最终我得到了一条记录。

Answer 1

如果查看加载到Elasticsearch中的记录的_id字段，您会看到它是%{id}，因为您的查询没有id字段。您将希望更改为document_id => "%{newsid}"或根据您的查询有意义的任何内容。