json - 将节点Filebeat接收到Elasticsearch

Question

我们将直接从Filebeats发送日志到没有Logstash的Elasticsearch。
日志可以在不同的字段中包含JSON，这些字段也需要解析。我创建了一个管道来解析日志，在开发人员控制台中对其进行了测试，并且输出符合预期。我通过将'pipeline:application_pipeline'添加到filebeat.yml中，将Filebeat设置为向该管道发送日志。但是在索引管理中，我只能看到我的文档。
如何检查Filebeat是否将这些日志发送到管道？
日志示例:

{"level":"info","message":"Webhook DeletePrice-{\"_headers\":{\"x-forwarded-proto\":[\"https\"],\"x-requested-with\":[\"\"],\"x-client-ip\":[\"93.84.120.32\"],\"user-agent\":[\"1C+Enterprise\\/8.3\"],\"accept\":[\"application\\/json\"],\"host\":[\"host.com\"],\"content-length\":[\"\"],\"content-type\":[\"\"]},\"company_id\":\"10248103\",\"service_id\":\"102.01.02S\",\"service_type\":\"clientApi\"}","service":"servicename","project":"someproject.com","event_id":"255A854BED569B8D4C21B5DE6D8E109C","payload":[],"date_server":"2020-07-24T11:45:48+00:00","date_unix":1595591148.966919}

{"level":"error","message":"NO service integration","service":"servicename","project":"someproject.com","event_id":"D3986456E5A42AF8574230C29D1D474D","payload":{"exception":{"class":"\\Ship\\Exceptions\\IntegrationException","message":"NO service integration","code":0,"file":"/var/www/builds/someproject.com/build.lab.service-public-api.2020_07_22_12_17_45/app/Containers/Price/UI/API/Controllers/Controller.php:406"}},"date_server":"2020-07-24T08:40:34+00:00","date_unix":1595580034.975073}

{"level":"info","message":"No photo in priceId-3696930","service":"service-private-api","project":"someproject.com","event_id":"FBEDA2C9600BFE11523592114B32BAEB","payload":[],"date_server":"2020-07-24T12:16:40+00:00","date_unix":1595593000.97212}

{"level":"error","message":"C404HttpException: 404 \u0421\u0442\u0440\u0430\u043d\u0438\u0446\u0430 \u043d\u0435 \u043d\u0430\u0439\u0434\u0435\u043d\u0430 in \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/modules\/personal\/controllers\/RobotsController.php:65\nStack trace:\n#0 \/var\/www\/builds\/build.artox-lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(4226): RobotsController->actionIndex()\n#1 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(3739): CInlineAction->runWithParams(Array)\n#2 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(3724): CController->runAction(Object(CInlineAction))\n#3 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(3714): CController->runActionWithFilters(Object(CInlineAction), Array)\n#4 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(1799): CController->run('index')\n#5 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(1719): CWebApplication->runController('personal\/robots...')\n#6 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/protected\/vendor\/yiisoft\/yii\/framework\/yiilite.php(1236): CWebApplication->processRequest()\n#7 \/var\/www\/builds\/build.lab.classified-platform.2020_07_29_12_13_54\/htdocs\/index.php(22): CApplication->run()\n#8 {main}\nREQUEST_URI=\/robots.txt\n---","service":"artox-lab\/classified-platform","project":"someproject.com","event_id":"91a10782a3566a74d5abefa9589c926c","payload":"exception.C404HttpException.404","date_server":"2020-07-29T14:25:34+03:00","date_unix":1596021934.218448}

管道示例:

PUT _ingest/pipeline/application_pipeline
{
      "description" : "Pipeline for parsing application.log for services",
      "processors" : [
        {
          "grok" : {
            "field" : "message",
            "patterns" : [
              "%{JSON:json_message_payload}"
            ],
            "pattern_definitions" : {
              "JSON" : "{.*$"
            },
            "ignore_failure" : true,
            "ignore_missing" : true
          }
        },
        {
          "remove" : {
            "field" : "json_message_payload",
            "ignore_failure" : true
          }
        }
      ]
    }
  }

输出:

{
  "_index" : "application_index",
  "_type" : "_doc",
  "_id" : "6",
  "_version" : 1,
  "_seq_no" : 3,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "date_server" : "2020-07-29T15:16:17+03:00",
    "level" : "error",
    "project" : "103by",
    "message" : """
C404HttpException: 404 Страница не найдена in /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/modules/personal/components/PersonalController.php:140
Stack trace:
#0 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3737): PersonalController->beforeAction(Object(ShowGalleryPhotoAction))
#1 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3724): CController->runAction(Object(ShowGalleryPhotoAction))
#2 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(3714): CController->runActionWithFilters(Object(ShowGalleryPhotoAction), Array)
#3 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1799): CController->run('showGalleryPhot...')
#4 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1719): CWebApplication->runController('personal/galler...')
#5 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/protected/vendor/yiisoft/yii/framework/yiilite.php(1236): CWebApplication->processRequest()
#6 /var/www/builds/build.artox-lab.classified-platform.2020_07_29_12_13_54/htdocs/index.php(22): CApplication->run()
#7 {main}
REQUEST_URI=/gallery/23609/1439643/
HTTP_REFERER=http://rnpcomr.103.by/gallery/23609/1439643/
---
""",
    "date_unix" : 1.596024977817727E9,
    "event_id" : "b75c7a1ef2f8780986931b038d2f8599",
    "payload" : "exception.C404HttpException.404",
    "service" : "artox-lab/classified-platform"
  }
}

Filebeat配置:

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elk.artoxlab.com:9200"]
  pipeline: application_pipeline

Answer 1

如果您运行GET _nodes/stats/ingest，您将在nodes.xyz.ingest.pipelines.application_pipeline中查看管道的使用情况统计信息
另一点值得注意的是，您也可以在Filebeat本身中执行相同的操作，而不必仅通过定义 decode_json_fields processor来使用摄取管道，如下所示:

processors:
  - decode_json_fields:
      fields: ["message"]
      process_array: true
      max_depth: 2
      target: ""
      overwrite_keys: true
      add_error_key: false

更新:如果您仍然看不到数据被索引，我建议您在管道中构建一些 failure handling。将其更改为此，以防万一由于某种原因索引编制失败，您可以在 failed-xyz索引中看到包含错误原因的文档。

PUT _ingest/pipeline/application_pipeline
{
  "description": "Pipeline for parsing application.log for services",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{JSON:json_message_payload}"
        ],
        "pattern_definitions": {
          "JSON": "{.*$"
        },
        "ignore_failure": true,
        "ignore_missing": true
      }
    },
    {
      "remove": {
        "field": "json_message_payload",
        "ignore_failure": true
      }
    }
  ],
  "on_failure": [
    {
      "append": {
        "field": "meta.errors",
        "value": "{{ _ingest.on_failure_message }}, {{ _ingest.on_failure_processor_type }}, {{ _ingest.on_failure_processor_tag }}"
      }
    },
    {
      "set": {
        "field": "_index",
        "value": "failed-{{ _index }}"
      }
    }
  ]
}