linux - Logstash 保持运行而不会崩溃或输出(windows 以及 linux)

Question

我正在使用一个简单的 conf 文件接收 log4j 文件并输出为标准输出(用于调试)

input { 
    file {
    type => "log4j"
    path => "C:\Stash\log4stash.log"
    start_position => "beginning"  } 
} 
output {# only for debug purpose
    stdout {
    codec => json 
    } 
}

当我在 Windows 中运行它时，有时它会在控制台上输出，但在以下输出之后几乎什么都没有显示:

←[33mUsing milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}←[0m

在 Linux 上也有类似的行为。任何指针？非常感谢。

更新:来自 linux 的 conf 文件和 (logstash) 调试日志。

input {
 file {
    type => "log4j"
    path => "/root/Downloads/log4stash.log"
    start_position => "beginning"
  }
}

output {
    # only for debug purposes   
    stdout {
        codec => rubydebug 
    }
    elasticsearch_http {
        host => "<elastic.server>"
        port => 9200    
    }
}

从日志来看，logstash 似乎读取文件并处理它，在注册和处理后进入等待状态。但是标准输出在哪里？和/或 Elasticsearch 中没有任何内容(使用 kibana UI 查询)。再次感谢您的投入。

    [root@cdh-cc3 logstash-1.4.2]# bin/logstash -f /root/Downloads/ll4j-2.conf --debug

Reading config file {:file=>"logstash/agent.rb", :level=>:debug, :line=>"301"}
Compiled pipeline code:
@inputs = []
@filters = []
@outputs = []
@input_file_1 = plugin("input", "file", LogStash::Util.hash_merge_many({ "type" => ("log4j".force_encoding("UTF-8")) }, { "path" => ("/root/Downloads/log4stash.log".force_encoding("UTF-8")) }, { "start_position" => ("beginning".force_encoding("UTF-8")) }))

@inputs << @input_file_1
@output_stdout_2 = plugin("output", "stdout", LogStash::Util.hash_merge_many({ "codec" => ("json".force_encoding("UTF-8")) }))

@outputs << @output_stdout_2
@output_elasticsearch_http_3 = plugin("output", "elasticsearch_http", LogStash::Util.hash_merge_many({ "host" => ("elastic.search".force_encoding("UTF-8")) }, { "port" => 9200 }))

@outputs << @output_elasticsearch_http_3
  @filter_func = lambda do |event, &block|
    extra_events = []
    @logger.debug? && @logger.debug("filter received", :event => event.to_hash)
    extra_events.each(&block)
  end
  @output_func = lambda do |event, &block|
    @logger.debug? && @logger.debug("output received", :event => event.to_hash)
    @output_stdout_2.handle(event)
    @output_elasticsearch_http_3.handle(event)

  end {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"26"}
Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn, :file=>"logstash/config/mixin.rb", :line=>"209"}
config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::File/@type = "log4j" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::File/@path = ["/root/Downloads/log4stash.log"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::File/@start_position = "beginning" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::File/@debug = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::File/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::File/@stat_interval = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::File/@discover_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Inputs::File/@sincedb_write_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Codecs::JSON/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::Stdout/@codec = <LogStash::Codecs::JSON charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::Stdout/@type = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::Stdout/@tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::Stdout/@exclude_tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::Stdout/@workers = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
Using milestone 2 output plugin 'elasticsearch_http'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn, :file=>"logstash/config/mixin.rb", :line=>"209"}
config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@host = "elastic.search" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@port = 9200 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@type = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@exclude_tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@workers = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@index = "logstash-%{+YYYY.MM.dd}" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@manage_template = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@template_name = "logstash" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@template_overwrite = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@user = nil {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@password = <password> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@flush_size = 100 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@idle_flush_time = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@document_id = nil {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
config LogStash::Outputs::ElasticSearchHTTP/@replication = "sync" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"105"}
Registering file input {:path=>["/root/Downloads/log4stash.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"74"}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/root/.sincedb_624f220c3c6db2a30932d4fb6e93769d", :path=>["/root/Downloads/log4stash.log"], :level=>:info, :file=>"logstash/inputs/file.rb", :line=>"115"}
Pipeline started {:level=>:info, :file=>"logstash/pipeline.rb", :line=>"78"}
_sincedb_open: reading from /root/.sincedb_624f220c3c6db2a30932d4fb6e93769d {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"199"}
_sincedb_open: setting [1451032, 0, 64768] to 1206 {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"203"}
_discover_file_glob: /root/Downloads/log4stash.log: glob is: ["/root/Downloads/log4stash.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"117"}
_discover_file: /root/Downloads/log4stash.log: new: /root/Downloads/log4stash.log (exclude is []) {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"126"}
_open_file: /root/Downloads/log4stash.log: opening {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"98"}
/root/Downloads/log4stash.log: sincedb last value 1206, cur size 1206 {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"122"}
/root/Downloads/log4stash.log: sincedb: seeking to 1206 {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"124"}
writing sincedb (delta since last write = 1423673806) {:level=>:debug, :file=>"filewatch/tail.rb", :line=>"177"}
/root/Downloads/log4stash.log: file grew, old size 0, new size 1206 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"81"}
Automatic template management enabled {:manage_template=>"true", :level=>:info, :file=>"logstash/outputs/elasticsearch_http.rb", :line=>"104"}
Template Search URL: {:template_search_url=>"http://elastic.search:9200/_template/*", :level=>:debug, :file=>"logstash/outputs/elasticsearch_http.rb", :line=>"112"}
_discover_file_glob: /root/Downloads/log4stash.log: glob is: ["/root/Downloads/log4stash.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"117"}
_discover_file_glob: /root/Downloads/log4stash.log: glob is: ["/root/Downloads/log4stash.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"117"}
_discover_file_glob: /root/Downloads/log4stash.log: glob is: ["/root/Downloads/log4stash.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"117"}

Answer 1

谢谢大家。发现发生了什么事。我发现 logstash“记住”了它解析过的文件。因此，我第一次将日志文件发送到 logstash 时进行了空运行，没有将它们发送到 elastic。自那次运行以来，logstash 不再解析相同的文件。删除了 sincedb 文件并使 logstash 再次解析它们(在 linux 上试过)。经验教训:-)