hadoop - Yarn 上保留的内存是什么

Question

我成功地在 Yarn 上启动了一个 spark 应用程序。然而，内存使用情况有点奇怪，如下所示:

http://imgur.com/1k6VvSI

保留内存是什么意思？我怎样才能有效地使用所有可用内存？

提前致谢。

Answer 1

查看 this blog来自 Cloudera，解释了 YARN 中的新内存管理。

这里是相关的部分:

... An implementation detail of this change that prevents applications from starving under this new flexibility is the notion of reserved containers. Imagine two jobs are running that each have enough tasks to saturate more than the entire cluster. One job wants each of its mappers to get 1GB, and another job wants its mappers to get 2GB. Suppose the first job starts and fills up the entire cluster. Whenever one of its task finishes, it will leave open a 1GB slot. Even though the second job deserves the space, a naive policy will give it to the first one because it’s the only job with tasks that fit. This could cause the second job to be starved indefinitely. To prevent this unfortunate situation, when space on a node is offered to an application, if the application cannot immediately use it, it reserves it, and no other application can be allocated a container on that node until the reservation is fulfilled. Each node may have only one reserved container. The total reserved memory amount is reported in the ResourceManager UI. A high number means that it may take longer for new jobs to get space. ,,,