elasticsearch - 如何在 journalbeat 中排除日志/事件

Question

我们正在使用 journalbeat 将 kubernetes 集群的日志推送到 Elasticsearch 。它工作正常并插入日志。然而，它也插入了我们不想要的“200 OK”和“INFO”等事件。 journalbeat.yaml如下
日记节拍.yaml

  journalbeat.yml: |
    name: "${NODENAME}"
    journalbeat.inputs:
    - paths: []
      seek: cursor
      cursor_seek_fallback: tail

    processors:
    - add_kubernetes_metadata:
        host: "${NODENAME}"
        in_cluster: true
        default_indexers.enabled: false
        default_matchers.enabled: false
        indexers:
          - container:
        matchers:
          - fields:
              lookup_fields: ["container.id"]
    - decode_json_fields:
        fields: ["message"]
        process_array: false
        max_depth: 1
        target: ""
        overwrite_keys: true
    - drop_event.when:
        or:
        - regexp.kubernetes.pod.name: "filebeat-.*"
        - regexp.kubernetes.pod.name: "journalbeat-.*"
        - regexp.kubernetes.pod.name: "nginx-ingress-controller-.*"
        - regexp.kubernetes.pod.name: "prometheus-operator-.*"

    setup.template.enabled: false
    setup.template.name: "journal-${ENVIRONMENT}-%{[agent.version]}"
    setup.template.pattern: "journal-${ENVIRONMENT}-%{[agent.version]}-*"
    setup.template.settings:
      index.number_of_shards: 10
      index.refresh_interval: 10s

    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
      username: '${ELASTICSEARCH_USERNAME}'
      password: '${ELASTICSEARCH_PASSWORD}'
      index: "journal-${ENVIRONMENT}-system-%{[agent.version]}-%{+YYYY.MM.dd}"
      indices:
      - index: "journal-${ENVIRONMENT}-k8s-%{[agent.version]}-%{+YYYY.MM.dd}"
        when.has_fields:
        - 'kubernetes.namespace'

如何排除“INFO”和“200 OK”事件等日志？

Answer 1

据我所知，无法排除 Journalbeat 中的日志.它以其他方式工作，这意味着你告诉它要寻找什么输入。
你应该阅读 Configuration input :

By default, Journalbeat reads log events from the default systemd journals. To specify other journal files, set the paths option in the journalbeat.inputs section of the journalbeat.yml file. Each path can be a directory path (to collect events from all journals in a directory), or a file path.

journalbeat.inputs:
- paths:
  - "/dev/log"
  - "/var/log/messages/my-journal-file.journal"

Within the configuration file, you can also specify options that control how Journalbeat reads the journal files and which fields are sent to the configured output. See Configuration options for a list of available options.

熟悉 Configuration options并使用 translated fields以您想要的确切输入为目标。

{beatname_lc}.inputs:
- id: consul.service
  paths: []
  include_matches:
    - _SYSTEMD_UNIT=consul.service

- id: vault.service
  paths: []
  include_matches:
    - _SYSTEMD_UNIT=vault.service

您应该使用它来定位您想要推送到弹性的输入。
作为 Journalbeat 的替代品你可以使用 Filebeat并且排除可能如下所示:

type: log
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$"]
exclude_lines: ['.*INFO.*']

希望这对您有所帮助。