google-cloud-stackdriver - 安装在 VM 上的 GCP stackdriver-agent 每分钟发送一次奇怪的日志

Question

请你能帮我解决以下问题吗？

我在 node.js 上有一个后端服务
我将它部署在 GCE VM 上。
它工作正常，但在安装日志记录和监控代理后，我在日志查看器中看到非常奇怪的日志。我查看了生成该日志的付费内容。它是堆栈驱动程序代理。

以下是它们:

A 2020-05-15T22:45:26Z write_gcm: can not take infinite value
A 2020-05-15T22:45:26Z write_gcm: wg_typed_value_create_from_value_t_inline failed for swap/percent/value! Continuing. 
A 2020-05-15T22:45:26Z write_gcm: can not take infinite value 
A 2020-05-15T22:45:26Z write_gcm: wg_typed_value_create_from_value_t_inline failed for swap/percent/value! Continuing. 
A 2020-05-15T22:45:26Z write_gcm: can not take infinite value 
A 2020-05-15T22:45:26Z write_gcm: wg_typed_value_create_from_value_t_inline failed for swap/percent/value! Continuing. 
A 2020-05-15T22:45:28Z write_gcm: Server response (CollectdTimeseriesRequest) contains errors:#012{#012  "payloadErrors": [#012    {#012      "error": {#012        "code": 3,#012        "message": "Unsupported collectd plugin/type combination: plugin: \"processes\" type: \"io_octets\""#012      }#012    },#012    {#012      "index": 5,#012      "error": {#012        "code": 3,#012        "message": "Unsupported collectd plugin/type combination: plugin: \"processes\" type: \"io_octets\""#012      }#012    },#012    {#012      "index": 10,#012      "error": {#012        "code": 3,#012        "message": "Unsupported collectd plugin/type combination: plugin: \"processes\" type: \"io_octets\""#012      }#012    },#012    {#012      "index": 15,#012      "error": {#012        "code": 3,#012        "message": "Unsupported collectd plugin/type combination: plugin: \"processes\" type: \"io_octets\""#012      }#012    },#012    {#012      "index": 20,#012      "error": {#012        "code": 3,#012        "message": "Unsupported collectd plugin/type combination: plugin: \"processes\" type: \"io_octets\""#012      }#012    },#012    {#012      "index": 25 
A 2020-05-15T22:45:29Z write_gcm: Server response (CollectdTimeseriesRequest) contains errors:#012{#012  "payloadErrors": [#012    {#012      "error": {#012        "code": 3,#012        "message": "Unsupported collectd plugin/type combination: plugin: \"processes\" type: \"io_octets\""#012      }#012    }#012  ]#012} 
A 2020-05-15T22:45:29Z write_gcm: Unsuccessful HTTP request 400: {#012  "error": {#012    "code": 400,#012    "message": "Field timeSeries[3].points[0].interval.start_time had an invalid value of \"2020-05-15T15:45:27.348251-07:00\": The start time must be before the end time (2020-05-15T15:45:27.348251-07:00) for the non-gauge metric 'agent.googleapis.com/agent/api_request_count'.",#012    "status": "INVALID_ARGUMENT"#012  }#012} 
A 2020-05-15T22:45:29Z write_gcm: Error talking to the endpoint. 
A 2020-05-15T22:45:29Z write_gcm: wg_transmit_unique_segment failed. 
A 2020-05-15T22:45:29Z write_gcm: wg_transmit_unique_segments failed. Flushing. 

所以，每分钟我都会看到这样的日志出现。
当我停止 stackdriver-agent 服务时，它们消失了。
我的项目中有 4 个虚拟机。只有其中两个出现这样的问题
在 Cent OS7 VM 和 Ubuntu 18 VM 上

Answer 1

到目前为止有2个PIT:

https://issuetracker.google.com/issues/160340568

https://issuetracker.google.com/issues/161054680

最后一篇有谷歌工程师对错误的解释 400 :

These messages are annoying but harmless. You are not losing any metrics. you can safely ignore these logs.

The root cause is a server-side config change and affects all agents. That change only affected the verbosity of the responses, not the processing of the requests. some of the incoming metrics were silently dropped before that change, and are now dropped noisily.

The metrics are sent by default by the upstream collectd plugin, and there are no controls for us to completely prevent those metrics from being sent. The log spam messages result from collectd's internal processing of those metrics.

If you'd like to filter out all the noisy logs you're seeing, you can create a Log Exclusion[1][2] or Log Sink[3][4]. A Log Exclusion will match logs up with a specified filter and drop them from the logs viewer before they come in, and a Log Sink would take logs and direct them to a Storage bucket, Big Query Table, or PubSub topic.

• [1] https://cloud.google.com/logging/docs/exclusions#overview
• [2] https://cloud.google.com/logging/docs/exclusions#create-filter
• [3] https://cloud.google.com/logging/docs/export
• [4] https://cloud.google.com/logging/docs/export/configure_export_v2

关于交换，有一篇博文:

https://myshittycode.com/2020/06/13/gcp-stackdriver-agent-write_gcm-can-not-take-infinite-value-error/

This error occurs because the VM instance does not have swap memory, hence this metric plugin tries to divide by 0.

To fix this, remove this configuration and restart stackdriver-agent.