elasticsearch - 如何使用 FluentD 和 ELK ElasticStack 配置证书授权

Question

我在连接 Amazon EKS 集群中的 FluentD 安装时遇到问题，该集群会将数据直接发送到 Azure 中的 ElasticSearch 堆栈。我想像使用 Filebeat 一样使用证书(ca.pem、cert.pem 和 cert.key)而不是用户/密码身份验证来配置它。

我已经设法让 FluentD pod 启动并运行，并且 RBAC 正在正常工作，并且由于证书授权的文档似乎不存在，我试图进行一些试验和错误，但没有成功。

我的证书配置如下:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: elasticsearch-azure
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd-role
  namespace: elastisearch-azure
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
      - pods/logs
    verbs: ["get", "list", "watch"]
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentd-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluentd-role
subjects:
  - kind: ServiceAccount
    name: fluentd
    namespace: elasticsearch-azure
---

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: elasticsearch-azure
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
        envFrom:
        - secretRef:
            name: fluent-tls
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "{{server_namne}}"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "{port}"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "https"
          # Option to configure elasticsearch plugin with self signed certs
          # ================================================================
          - name: FLUENT_ELASTICSEARCH_SSL_VERIFY
            value: "true"
          # Option to configure elasticsearch plugin with tls
          # ================================================================
          - name: FLUENT_ELASTICSEARCH_SSL_VERSION
            value: "TLSv1_2"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: ssl
          mountPath: /fluent-tls/ssl
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      # certificates folder for filebeat
      - name: ssl
        secret:
          secretName: fluent-tls

我已经使用以下命令创建了 secret :

kubectl create secret generic fluent-tls \
--from-file=ca_file=./chain.pem \
--from-file=cert_pem=./cert.pem \
--from-file=cert_key=./cert.key

运行 pod 时出现的错误如下:

 <match **>
    @type elasticsearch
    @id out_es
    @log_level "info"
    include_tag_key true
    host "super-sercret-host.com"
    port even-more-secret-portnumber
    path ""
    scheme https
    ssl_verify false
    ssl_version TLSv1_2
    reload_connections false
    reconnect_on_error true
    reload_on_failure true
    log_es_400_reason false
    logstash_prefix "logstash"
    logstash_format true
    index_name "logstash"
    type_name "fluentd"
    <buffer>
      flush_thread_count 8
      flush_interval 5s
      chunk_limit_size 2M
      queue_limit_length 32
      retry_max_interval 30
      retry_forever true
    </buffer>
  </match>
</ROOT>
2019-12-16 14:51:30 +0000 [info]: starting fluentd-1.7.4 pid=6 ruby="2.6.5"
2019-12-16 14:51:30 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.6.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "--under-supervisor"]
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.13'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '3.7.1'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.1'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.3.0'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.6.1'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.1'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2'
2019-12-16 14:51:31 +0000 [info]: gem 'fluentd' version '1.7.4'
2019-12-16 14:51:31 +0000 [info]: adding match pattern="fluent.**" type="null"
2019-12-16 14:51:31 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2019-12-16 14:51:31 +0000 [info]: adding match pattern="**" type="elasticsearch"
2019-12-16 14:51:34 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. SSL_connect returned=1 errno=0 state=error: sslv3 alert handshake failure (OpenSSL::SSL::SSLError)
2019-12-16 14:51:34 +0000 [warn]: #0 [out_es] Remaining retry: 14. Retry to communicate after 2 second(s).
2019-12-16 14:51:38 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. SSL_connect returned=1 errno=0 state=error: sslv3 alert handshake failure (OpenSSL::SSL::SSLError)
2019-12-16 14:51:38 +0000 [warn]: #0 [out_es] Remaining retry: 13. Retry to communicate after 4 second(s).

我知道，如果我只能将配置属性:ca_file、client_pem 和 client_key 参数放入标签中，它可能会起作用，但到目前为止我还没有做到这一点。任何帮助表示赞赏。

Answer 1

谢谢你!太感谢了;您的配置示例帮助解决了我在 FluentD 和 ES 之间的 SSL 身份验证问题，现在也许我可以帮助您。

我正在使用 Open Distro for Elasticsearch 和 Bitnami 的 FluentD，我遇到了类似的错误。我的决定是使用您的部分配置，但我必须将主机更改为主机，就像这样；

hosts               https://admin:admin@odfe-node1:9200

似乎您必须在一行中指定用户名、协议(protocol)和端口，就像您必须在集群中指定多个主机一样。它对我有用。看 FluentD docs以供进一步引用。