jdbc - 使用 logstash 将数据从 MySQL 索引到 ElasticSearch 时，它显示 Java heapspace 错误

Question

MySQL 表有 191000 行。当我尝试下面的操作时，它会运行一段时间并显示此错误:

logstash/bin/logstash -f /home/vatsa/logs/conts-out.conf  

java.lang.OutOfMemoryError: Java heap space Dumping heap to /home/vatsa/logstash/heapdump.hprof Exception flushing buffer at interval! {:error=>"Java heap space", :class=>"Java::JavaLang::OutOfMemoryError", :level=>:warn} Error: Your application used more memory than the safety cap of 1G. Specify -J-Xmx####m to increase it (#### = cap size in MB). Specify -w for full OutOfMemoryError stack trace

我不明白这个错误。如何解决这个问题。提前致谢。

Answer 1

您可以尝试启用查询分页。这将限制从 MySQL 检索的元素数量。对应设置如下:

input {
  jdbc {
    ...
    jdbc_paging_enabled => true
    jdbc_page_size => 100000
    ...
  }
}

来自 Logstash documentation jdbc_paging_enabled:

This will cause a sql statement to be broken up into multiple queries. Each query will use limits and offsets to collectively retrieve the full result-set.