docker - filebeat + kubernetes + elasticsearch不保存特定字段

Question

我创建了一个 namespace 来获取带有文件拍的日志并保存到elasticsearch。
为什么不在Elasticsearch上保存有关Kubernetes的字段，如何举例如例？

Kubernetes fields

        "kubernetes" : {
            "labels" : {
              "app" : "MY-APP",
              "pod-template-hash" : "959f54cd",
              "serving" : "true",
              "version" : "1.0",
              "visualize" : "true"
            },
            "pod" : {
              "uid" : "e20173cb-3c5f-11ea-836e-02c1ee65b375",
              "name" : "MY-APP-959f54cd-lhd5p"
            },
            "node" : {
              "name" : "ip-xxx-xx-xx-xxx.ec2.internal"
            },
            "container" : {
              "name" : "istio"
            },
            "namespace" : "production",
            "replicaset" : {
              "name" : "MY-APP-959f54cd"
            }
          }

当前正在这样保存:

      "_source" : {
          "@timestamp" : "2020-01-23T12:33:14.235Z",
          "ecs" : {
            "version" : "1.0.0"
          },
          "host" : {
            "name" : "worker-node1"
          },
          "agent" : {
            "hostname" : "worker-node1",
            "id" : "xxxxx-xxxx-xxx-xxxx-xxxxxxxxxxxxxx",
            "version" : "7.1.1",
            "type" : "filebeat",
            "ephemeral_id" : "xxxx-xxxx-xxxx-xxxxxxxxxxxxx"
          },
          "log" : {
            "offset" : xxxxxxxx,
            "file" : {
              "path" : "/var/lib/docker/containers/xxxx96ec2bfd9a3e4f4ac83581ad90/7fd55e1249aa009df3f8e3250c967bbe541c9596xxxxxac83581ad90-json.log"
            }
          },
          "stream" : "stdout",
          "message" : "xxxxxxxx",
          "input" : {
            "type" : "docker"
          }
        }

要遵循我的filebeat.config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.config:
      inputs:
        # Mounted `filebeat-inputs` configmap:
        path: ${path.config}/inputs.d/*.yml
        # Reload inputs configs as they change:
        reload.enabled: false
        multiline.pattern: '^[[:space:]]'
        multiline.negate: false
        multiline.match: after
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false

    # To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      hints.enabled: true

    processors:
      - add_cloud_metadata:
      - add_kubernetes_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      protocol: "http"
    setup.ilm.enabled: false
    ilm.enabled: false
    xpack.monitoring:
      enabled: true

DamemonSet如下所示:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      hostNetwork: true
      terminationGracePeriodSeconds: 30
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat-oss:7.1.1
          args: [
            "-c", "/etc/filebeat.yml",
            "-e",
          ]
          env:
            - name: ELASTICSEARCH_HOST
              value: xxxxxxxxxxxxx
            - name: ELASTICSEARCH_PORT
              value: "9200"
          securityContext:
            runAsUser: 0
            # If using Red Hat OpenShift uncomment this:
            #privileged: true
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat.yml
              readOnly: true
              subPath: filebeat.yml
            - name: inputs
              mountPath: /usr/share/filebeat/inputs.d
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config
          configMap:
            defaultMode: 0600
            name: filebeat-config
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: inputs
          configMap:
            defaultMode: 0600
            name: filebeat-inputs
        # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate

在将配置应用到kubernetes之前，我确实删除过Elasticsearch的注册表文件。

Answer 1

正如我在评论中所述。看来您的ConfigMap缺少容器日志中的paths:。应该是这样的:

       type: container
       paths:
         - /var/log/containers/*${data.kubernetes.container.id}.log

将您的配置文件与 this one进行比较。

希望对您有所帮助。