java - 在 Docker 容器中运行的 JVM 的驻留集大小 (RSS) 和 Java 总提交内存 (NMT) 之间的差异

Question

场景:

我有一个在 docker 容器中运行的 JVM。我使用两个工具做了一些内存分析:1) top 2) Java Native Memory Tracking。这些数字看起来令人困惑，我试图找出导致差异的原因。

问题:

Java 进程的 RSS 报告为 1272MB，Java 总内存报告为 790.55 MB。我如何解释剩余的内存 1272 - 790.55 = 481.44 MB 去了哪里？

为什么我在查看 this question 之后仍想保持这个问题开放？关于 SO:

我确实看到了答案，并且解释很有意义。但是，在从 Java NMT 和 pmap -x 获得输出后，我仍然无法具体映射哪些 Java 内存地址实际上是常驻和物理映射的。我需要一些具体的解释(带有详细的步骤)来找出导致 RSS 和 Java Total commited memory 之间差异的原因。

热门输出

Java NMT

Docker 内存统计数据

图表

我的 docker 容器运行时间超过 48 小时。现在，当我看到一个包含以下内容的图表时:

1. 分配给 docker 容器的总内存 = 2 GB
2. Java 最大堆 = 1 GB
3. 总提交 (JVM) = 始终小于 800 MB
4. 使用的堆 (JVM) = 始终小于 200 MB
5. 未使用堆 (JVM) = 始终小于 100 MB。
6. RSS = 大约 1.1 GB。

那么，1.1 GB (RSS) 到 800 MB(Java 提交的总内存)之间的内存是什么？

Answer 1

你在“Analyzing java memory usage in a Docker container”中有一些线索，来自 Mikhail Krestjaninoff :

(需要明确的是，三年后的 2019 年 5 月，the situation does improves with openJDK 8u212)

Resident Set Size is the amount of physical memory currently allocated and used by a process (without swapped out pages). It includes the code, data and shared libraries (which are counted in every process which uses them)

Why does docker stats info differ from the ps data?

Answer for the first question is very simple - Docker has a bug (or a feature - depends on your mood): it includes file caches into the total memory usage info. So, we can just avoid this metric and use ps info about RSS.

Well, ok - but why is RSS higher than Xmx?

Theoretically, in case of a java application

RSS = Heap size + MetaSpace + OffHeap size

where OffHeap consists of thread stacks, direct buffers, mapped files (libraries and jars) and JVM code itse

Since JDK 1.8.40 we have Native Memory Tracker!

As you can see, I’ve already added -XX:NativeMemoryTracking=summary property to the JVM, so we can just invoke it from the command line:

docker exec my-app jcmd 1 VM.native_memory summary

(这就是OP所做的)

Don’t worry about the “Unknown” section - seems that NMT is an immature tool and can’t deal with CMS GC (this section disappears when you use an another GC).

Keep in mind, that NMT displays “committed” memory, not "resident" (which you get through the ps command). In other words, a memory page can be committed without considering as a resident (until it directly accessed).

That means that NMT results for non-heap areas (heap is always preinitialized) might be bigger than RSS values.

(这就是“Why does a JVM report more committed memory than the linux process resident set size?”出现的地方)

As a result, despite the fact that we set the jvm heap limit to 256m, our application consumes 367M. The “other” 164M are mostly used for storing class metadata, compiled code, threads and GC data.

First three points are often constants for an application, so the only thing which increases with the heap size is GC data.
This dependency is linear, but the “k” coefficient (y = kx + b) is much less then 1.

---

更一般地说，这似乎跟在 issue 15020 后面。自 docker 1.7 以来报告了类似的问题

I'm running a simple Scala (JVM) application which loads a lot of data into and out of memory.
I set the JVM to 8G heap (-Xmx8G). I have a machine with 132G memory, and it can't handle more than 7-8 containers because they grow well past the 8G limit I imposed on the JVM.

(docker stat 是 reported as misleading before，因为它显然将文件缓存包含在总内存使用信息中)

docker stat shows that each container itself is using much more memory than the JVM is supposed to be using. For instance:

CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
dave-1 3.55% 10.61 GB/135.3 GB 7.85% 7.132 MB/959.9 MB
perf-1 3.63% 16.51 GB/135.3 GB 12.21% 30.71 MB/5.115 GB

It almost seems that the JVM is asking the OS for memory, which is allocated within the container, and the JVM is freeing memory as its GC runs, but the container doesn't release the memory back to the main OS. So... memory leak.