I am running fluentd as a DaemonSet in a Kubernetes cluster. fluentd writes the log entries to OpenSearch. Take a look at https://github.com/fluent/fluentd-kubernetes-daemonset

我在Kubernetes集群中以守护集的身份运行fluentd。Fluentd将日志条目写入OpenSearch。看一看https://github.com/fluent/fluentd-kubernetes-daemonset

I must lay some background before my question: The way it works is that Kubernetes pods write to stdout, and the container runtime writes this to a certain location, namely /var/logs/pods/<pod_specific_location> . The format of these log files is as such:

在我的问题之前，我必须先介绍一些背景知识：它的工作方式是Kubernetes pod写入stdout，容器运行时将其写入某个位置，即/var/Logs/pods/ 。这些日志文件的格式如下：

31-12-23T12:00:00.123456Z    stdout    F     my great log message

Now, fluentd is configured to pick it from there, and using the cri parser plugin transforms it to:

现在，fluentd被配置为从那里获取它，并使用CRI解析器插件将其转换为：

{
"time": "31-12-23T12:00:00.123456Z",
"stream": "stdout",
"logtag": "F",
"message": "my great log message"
}

Now, say I run a pod in my cluster that writes the following log message:

现在，假设我在群集中运行一个Pod，该Pod写入以下日志消息：

hello

Further down the road, using the kubernetes metadata plugin, fluentd enriches this record with kubernetes metadata, such as namespace name, pod name, etc. etc., so it'll look something like:

接下来，通过使用Kubernetes元数据插件，fluentd使用Kubernetes元数据(如名称空间名称、Pod名称等)丰富了该记录，因此它将如下所示：

{
"stream":"stdout",
"logtag":"F",
"time":"31-12-23T12:00:00.123456Z",
"message": "my great log message"
"docker":
{"container_id":"9077644273956d3f3e9d171240f412b3b6e959984a5fd99adfcb77f9b998a370"},
"kubernetes":
{"container_name":"demo-app",
"namespace_name":"foo",
"pod_name":"foo-ns-app",
"container_image":"docker.io/yoavklein3/net-tools:latest",
"container_image_id":"docker.io/yoavklein3/net-tools@sha256:3fd9646a14d97ecc2d236a5bebd88faf617bc6045f1e4f32c49409f1c930879a",
"pod_id":"a69fb942-c0ab-457d-b752-ffa3fa27e574",
"pod_ip":"10.0.2.224",
"host":"ip-10-0-2-5.ec2.internal",
"master_url":"https://172.20.0.1:443/api",
"namespace_id":"6bdf5fe9-9a5a-4501-ab6c-deddd241e071",
"namespace_labels":{"kubernetes.io/metadata.name":"foo"}}}

Now, using the opensearch plugin it is sent to Opensearch.

现在，使用OpenSearch插件，它被发送到OpenSearch。

Now, when I open Opensearch Dashboards, I can see a field called @timestamp, and I just can't figure out where this field comes from:

现在，当我打开OpenSearch仪表板时，我可以看到一个名为@Timestamp的字段，但我就是想不出这个字段是从哪里来的：

This is a document in OpenSearch (apologies for not sticking to the example above exactly, but the concept remains the same):

这是OpenSearch中的一个文档(很抱歉没有完全遵循上面的示例，但概念保持不变)：

{
  "_index": "logstash-2023.06.06",
  "_type": "_doc",
  "_id": "sVHjj4gByMQm1Wc45hv2",
  "_version": 1,
  "_score": null,
  "_source": {
    "stream": "stdout",
    "logtag": "F",
    "time": "2023-06-06T08:47:35.874884092Z",
    "docker": {
      "container_id": "9077644273956d3f3e9d171240f412b3b6e959984a5fd99adfcb77f9b998a370"
    },
    "kubernetes": {
      "container_name": "demo-app",
      "namespace_name": "foo",
      "pod_name": "foo-ns-app",
      "container_image": "docker.io/yoavklein3/net-tools:latest",
      "container_image_id": "docker.io/yoavklein3/net-tools@sha256:3fd9646a14d97ecc2d236a5bebd88faf617bc6045f1e4f32c49409f1c930879a",
      "pod_id": "a69fb942-c0ab-457d-b752-ffa3fa27e574",
      "pod_ip": "10.0.2.224",
      "host": "ip-10-0-2-5.ec2.internal",
      "master_url": "https://172.20.0.1:443/api",
      "namespace_id": "6bdf5fe9-9a5a-4501-ab6c-deddd241e071",
      "namespace_labels": {
        "kubernetes.io/metadata.name": "foo"
      }
    },
    "data": "This is from FOO namespace",
    "@timestamp": "2023-06-06T08:47:35.882677347+00:00",
    "tag": "kubernetes.var.log.containers.foo-ns-app_foo_demo-app-9077644273956d3f3e9d171240f412b3b6e959984a5fd99adfcb77f9b998a370.log"
  },
  "fields": {
    "@timestamp": [
      "2023-06-06T08:47:35.882Z"
    ],
    "time": [
      "2023-06-06T08:47:35.874Z"
    ]
  },
  "sort": [
    1686041255882
  ]
}

NOTE: the message field is missing, and there's the data field instead. This is due to parsing the message field as JSON. You can ignore this, it's completely irrelevant, just noting if you're confused.

注意：消息字段缺失，取而代之的是数据字段。这是因为将消息字段解析为JSON。你可以忽略这一点，这是完全无关紧要的，只要你感到困惑就行了。

EDIT

I don't think that the source of this @timestamp field is the Opensearch plugin. Why? because when I run fluentd with opensearch NOT in a kubernetes cluster, but rather using other input plugins, I can't see this field.

我不认为这个@Timestamp字段的来源是OpenSearch插件。为什么？因为当我不是在Kubernetes集群中，而是使用其他输入插件使用OpenSearch运行fluentd时，我看不到这个字段。

I can see a field called @timestamp, and I just can't figure out where this field comes from...

This field is added by the opensearch plugin, the value is the point in time when the message is ingested.

此字段由OpenSearch插件添加，值为接收邮件的时间点。