docker - Fluentd Elasticsearch插件未从Raspberry Pi上的Kubernetes连接到Elasticsearch

Question

编辑:我已根据efrat-levitan's的建议在评论中添加了新信息。当我按照原始注释的建议将Elasticsearch版本升级到7.6.0时，列出的日志输出略有不同。为了帮助调试，我也没有立即启动Elasticsearch。可以在日志的ECONNREFUSED消息中看到其效果。我在下面的摘要中指出了日志文件的更改。其余大部分消息文本(即不是日志片段)与以前相同。

我一直在努力通过docker中运行的fluent-plugin-elasticsearch插件来获得fluentd的ARM版本(针对Raspberry Pi 3和4)。我一直无法找到合适的docker镜像，所以我自己构建了一个(如果有人知道在哪里可以找到我，我将不胜感激)。我从fluentd-docker-image存储库(不包括Elasticsearch插件)开始，并根据需要修改了它，使用fluentd-kubernetes-daemonset存储库(包括Elasticsearch插件)。好消息是，它可以在Raspberry Pi上正常启动。坏消息是它似乎甚至没有尝试连接到ElasticSearch(Raspberry Pi网络的外部)。日志文件如下所示:

2020-03-02 18:13:15 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.12'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.4'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.3.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.6.1'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.1'
2020-03-02 18:13:16 +0000 [info]: gem 'fluentd' version '1.9.2'
2020-03-02 18:13:16 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-03-02 18:13:16 +0000 [info]: using configuration file: <ROOT>
  <filter **>
    @type stdout
  </filter>
  <source>
    @type forward
    @label @ES
  </source>
  <label @ES>
    <match out.elasticsearch.**>
      @type elasticsearch
      @log_level "info"
      include_tag_key true
      host "10.0.0.223"
      port 9200
      path ""
      scheme http
      index_name "logstash"
      include_timestamp true
      log_es_400_reason false
      logstash_prefix "logstash"
      logstash_dateformat "%Y.%m.%d"
      logstash_format true
      ssl_verify true
      ssl_version TLSv1_2
      user
      password xxxxxx
      reload_connections false
      reconnect_on_error true
      reload_on_failure true
      request_timeout 5s
      sniffer_class_name "Fluent::Plugin::ElasticsearchSimpleSniffer"
      type_name "doc"
      template_name
      template_file
      template_overwrite false
      time_key "@timestamp"
      <buffer>
        flush_thread_count 8
        flush_interval 5s
        chunk_limit_size 2M
        queue_limit_length 32
        retry_max_interval 30
        retry_forever true
      </buffer>
    </match>
  </label>
  <label @ERROR>
    <match **>
      @type stdout
    </match>
  </label>
</ROOT>
2020-03-02 18:13:16 +0000 [info]: starting fluentd-1.9.2 pid=7 ruby="2.6.5"
2020-03-02 18:13:16 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "-r", "/usr/local/bundle/gems/fluent-plugin-elasticsearch-4.0.4/lib/fluent/plugin/elasticsearch_simple_sniffer.rb", "--under-supervisor"]
2020-03-02 18:13:20 +0000 [info]: adding match in @ES pattern="out.elasticsearch.**" type="elasticsearch"
2020-03-02 18:13:23 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:23 +0000 [warn]: #0 Remaining retry: 14. Retry to communicate after 2 second(s).
2020-03-02 18:13:27 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:27 +0000 [warn]: #0 Remaining retry: 13. Retry to communicate after 4 second(s).
2020-03-02 18:13:35 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:35 +0000 [warn]: #0 Remaining retry: 12. Retry to communicate after 8 second(s).
2020-03-02 18:13:51 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:51 +0000 [warn]: #0 Remaining retry: 11. Retry to communicate after 16 second(s).
2020-03-02 18:13:51 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2020-03-02 18:13:51 +0000 [info]: adding match in @ERROR pattern="**" type="stdout"
2020-03-02 18:13:51 +0000 [info]: adding filter pattern="**" type="stdout"
2020-03-02 18:13:51 +0000 [info]: adding source type="forward"
2020-03-02 18:13:51 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-03-02 18:13:51 +0000 [info]: #0 starting fluentd worker pid=22 ppid=7 worker=0
2020-03-02 18:13:51 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
2020-03-02 18:13:51 +0000 [info]: #0 fluentd worker is now running worker=0
2020-03-02 18:13:51.581170450 +0000 fluent.info: {"pid":22,"ppid":7,"worker":0,"message":"starting fluentd worker pid=22 ppid=7 worker=0"}
2020-03-02 18:13:51 +0000 [warn]: #0 no patterns matched tag="fluent.info"
2020-03-02 18:13:51.585716902 +0000 fluent.info: {"port":24224,"bind":"0.0.0.0","message":"listening port port=24224 bind=\"0.0.0.0\""}
2020-03-02 18:13:51 +0000 [warn]: #0 no patterns matched tag="fluent.info"
2020-03-02 18:13:51.593737828 +0000 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}

为了进一步缩小日志范围，fluentd似乎确实在配置和连接方面都了解Elasticsearch:

2020-03-02 18:13:20 +0000 [info]: adding match in @ES pattern="out.elasticsearch.**" type="elasticsearch"
2020-03-02 18:13:23 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)

“ECONNREFUSED”错误是因为我停止了Elasticsearch来进行故障排除。因此Fluentd正在尝试建立连接。如下一个代码片段所示，启动Elasticsearch之后，它会连接并继续处理:

2020-03-02 18:13:51 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.

问题在于Fluentd似乎并没有真正通过Elasticsearch完成“签名”过程。我希望成功时会看到类似这样的消息，或者某种错误消息。

2020-02-28 21:56:26 +0000 [info]: #0 [out_es] Connection opened to Elasticsearch cluster => {:host=>"10.0.0.223", :port=>9200, :scheme=>"http", :path=>""}

我还希望在Elasticsearch中看到一些证据，表明确实确实完成了“登录”过程。例如，我没有在Elasticsearch中看到“logstash”索引，也没有在Elasticsearch日志中看到任何证据表明除Kibana之外的任何其他客户端都已连接。

该配置已记录在上面，对我来说似乎是正确的。日志中记录的命令行对我来说也很好。

/usr/local/bin/ruby -Eascii-8bit:ascii-8bit /usr/local/bundle/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins -r /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.0.4/lib/fluent/plugin/elasticsearch_simple_sniffer.rb --under-supervisor

登录到pod并连接到Elasticsearch也可以正常工作:

$ ks exec -it fluentd-h2qzn sh
$ curl http://10.0.0.223:9200
{
  "name" : "Richs-MacBook.local",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "OkZ2-Lj2RjW-pVyVl0C7og",
  "version" : {
    "number" : "7.6.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "7f634e9f44834fbc12724506cc1da681b0c3b1e3",
    "build_date" : "2020-02-06T00:09:00.449973Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

有人可以说明缺乏与Elasticsearch主机的连接性还是提供我可以执行的其他一些故障排除步骤？

谢谢，
丰富

Answer 1

如Efrat Levitan所述，问题确实是版本不匹配。我安装了Elasticsearch 7.6.0和相应的Kibana版本7.6.0，并且可以正常工作。 Fluentd已启动并正在运行，并且日志在Kibana中显示。