logging - 在 Google Cloud Kubernetes pod 中消耗磁盘空间的日志上的日志轮换

Question

我们在 Google Cloud Platform Kubernetes 集群中有一个 pod，将 JsonFormatted 写入 StdOut。这是 Stackdriver 开箱即用的。但是，我们看到 pod 的磁盘使用量一直在增长，我们无法理解如何在 Deployment 上设置最大大小以进行日志轮换。

Google Cloud 和 Kubernetes 上的文档对此不清楚。

这只是最后一个小时:

Answer 1

您确定 pod 的磁盘使用率很高是因为日志吗？
如果应用程序将日志写入标准输出，它不会使用 pod 内的任何磁盘空间。
所有日志通常都存储在节点文件系统上的日志文件中，并且可以由节点 logrotate 进程管理。
也许应用程序将 pod 的磁盘空间用于其他用途，例如临时文件或调试信息？
这是documentation的部分内容与日志轮换相关:

Logging at the node level:

Everything a containerized application writes to stdout and stderr ishandled and redirected somewhere by a container engine. For example,the Docker container engine redirects those two streams to a loggingdriver, which is configured in Kubernetes to write to a file injson format.

An important consideration in node-level logging is implementing logrotation, so that logs don’t consume all available storage on thenode.

Kubernetes currently is not responsible for rotating logs, but rathera deployment tool should set up a solution to address that. Forexample, in Kubernetes clusters, deployed by the kube-up.sh script,there is a logrotate tool configured to run each hour.

You can also set up a container runtime to rotate application’s logsautomatically, e.g. by using Docker’s log-opt.

In the kube-up.sh script, the latter approach is used for COS image onGCP, and the former approach is used in any other environment. Inboth cases, by default rotation is configured to take place when logfile exceeds 10MB.

As an example, you can find detailed information about how kube-up.shsets up logging for COS image on GCP in the corresponding script.

这是与 logrotate 相关的脚本的一部分:

# Installs logrotate configuration files
function setup-logrotate() {
  mkdir -p /etc/logrotate.d/
  # Configure log rotation for all logs in /var/log, which is where k8s services
  # are configured to write their log files. Whenever logrotate is ran, this
  # config will:
  # * rotate the log file if its size is > 100Mb OR if one day has elapsed
  # * save rotated logs into a gzipped timestamped backup
  # * log file timestamp (controlled by 'dateformat') includes seconds too. This
  #   ensures that logrotate can generate unique logfiles during each rotation
  #   (otherwise it skips rotation if 'maxsize' is reached multiple times in a
  #   day).
  # * keep only 5 old (rotated) logs, and will discard older logs.
  cat > /etc/logrotate.d/allvarlogs <<EOF
/var/log/*.log {
    rotate ${LOGROTATE_FILES_MAX_COUNT:-5}
    copytruncate
    missingok
    notifempty
    compress
    maxsize ${LOGROTATE_MAX_SIZE:-100M}
    daily
    dateext
    dateformat -%Y%m%d-%s
    create 0644 root root
}
EOF

}