elasticsearch - logstash 发送数据到国外 Elasticsearch 失败

Question

这是来自 logstash.err 的错误:

Faraday::ConnectionFailed: End of file reached call at /opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/adapter/net_http.rb:44 build_response at /opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/rack_builder.rb:139 run_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/connection.rb:377 perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/http/faraday.rb:24 call at org/jruby/RubyProc.java:271 perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/base.rb:187 perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/http/faraday.rb:20 perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/client.rb:102 perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.1/lib/elasticsearch/api/namespace/common.rb:21 get_template at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.1/lib/elasticsearch/api/actions/indices/get_template.rb:24 template_exists? at /opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:132 template_install at /opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:21 register at /opt/logstash/lib/logstash/outputs/elasticsearch.rb:259 each at org/jruby/RubyArray.java:1613 outputworker at /opt/logstash/lib/logstash/pipeline.rb:220 start_outputs at /opt/logstash/lib/logstash/pipeline.rb:152

这是我的输出配置:

output {
        elasticsearch { 
            host => "X.X.X.X"
            port => "9300"
            protocol => "http"
            cluster => "elasticsearch_david"
        }   
    }

没有连接问题，有什么想法吗？

对 tcpdump 的进一步调查给出:

GET /_template/logstash HTTP/1.1
User-Agent: Faraday v0.9.0
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
Connection: close

Answer 1

问题在于端口和协议(protocol)不匹配:

output {
        elasticsearch { 
            host => "X.X.X.X"
            port => "9300"
            protocol => "http"
            cluster => "elasticsearch_david"
        }   
    }

您将协议(protocol)设置为“http”，这将需要端口 9200(ES 用于 http 请求的默认端口)，但将端口设置为 9300，这是用于集群间通信的端口，通常与“节点”协议(protocol)一起使用。

不幸的是，文档与协议(protocol)的默认值相矛盾:

protocol

Value can be any of: "node", "transport", "http"
There is no default value for this setting.

Choose the protocol used to talk to Elasticsearch.

The ‘node’ protocol will connect to the cluster as a normal Elasticsearch node (but will not store data). This allows you to use things like multicast discovery. If you use the node protocol, you must permit bidirectional communication on the port 9300 (or whichever port you have configured).

The ‘transport’ protocol will connect to the host you specify and will not show up as a ‘node’ in the Elasticsearch cluster. This is useful in situations where you cannot permit connections outbound from the Elasticsearch cluster to this Logstash server.

The ‘http’ protocol will use the Elasticsearch REST/HTTP interface to talk to elasticsearch.

All protocols will use bulk requests when talking to Elasticsearch.

The default protocol setting under java/jruby is “node”. The default protocol on non-java rubies is “http”

您最好的选择是根据您想要执行的操作将协议(protocol)设置设置为“node”、“http”或“transport”之一，并让 logstash 为您设置适当的端口:

output {
    elasticsearch { 
    host => "X.X.X.X"
    protocol => "http"
    cluster => "elasticsearch_david"
} 

见 http://logstash.net/docs/1.4.1/outputs/elasticsearch#protocol