gpt4 book ai didi

hadoop - NameNode堆使用率与ResourceManager堆使用率之间的区别(试图查找NameNode堆使用率的原因)?

转载 作者:行者123 更新时间:2023-12-02 20:21:44 24 4
gpt4 key购买 nike

NameNode堆用法和ResourceManager堆用法之间有什么区别?我正在尝试查找大量的NameNode堆使用原因。

在ambari仪表板中,我看到...
enter image description here

在运行一些sqoop作业时。不知道是什么原因导致此处的NN使用率很高(没有使用hadoop管理员的丰富经验)?这是不寻常的金额(仅在最近才发现)吗?

此外,在100%完成mapreduce任务后,sqoop作业似乎被冻结了比平常更长的时间。看到...

[2020-01-31 14:00:55,193]  INFO mapreduce.JobSubmitter: number of splits:12
[2020-01-31 14:00:55,402] INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1579648183118_1085
[2020-01-31 14:00:55,402] INFO mapreduce.JobSubmitter: Executing with tokens: []
[2020-01-31 14:00:55,687] INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.0.0-78/0/resource-types.xml
[2020-01-31 14:00:55,784] INFO impl.YarnClientImpl: Submitted application application_1579648183118_1085
[2020-01-31 14:00:55,837] mapreduce.Job: The url to track the job: http://hw001.ucera.local:8088/proxy/application_1579648183118_1085/
[2020-01-31 14:00:55,837] mapreduce.Job: Running job: job_1579648183118_1085
[2020-01-31 14:01:02,964] mapreduce.Job: Job job_1579648183118_1085 running in uber mode : false
[2020-01-31 14:01:02,965] mapreduce.Job: map 0% reduce 0%
[2020-01-31 14:01:18,178] mapreduce.Job: map 8% reduce 0%
[2020-01-31 14:02:21,552] mapreduce.Job: map 17% reduce 0%
[2020-01-31 14:04:55,239] mapreduce.Job: map 25% reduce 0%
[2020-01-31 14:05:36,417] mapreduce.Job: map 33% reduce 0%
[2020-01-31 14:05:37,424] mapreduce.Job: map 42% reduce 0%
[2020-01-31 14:05:40,440] mapreduce.Job: map 50% reduce 0%
[2020-01-31 14:05:41,444] mapreduce.Job: map 58% reduce 0%
[2020-01-31 14:05:44,455] mapreduce.Job: map 67% reduce 0%
[2020-01-31 14:05:52,484] mapreduce.Job: map 75% reduce 0%
[2020-01-31 14:05:56,499] mapreduce.Job: map 83% reduce 0%
[2020-01-31 14:05:59,528] mapreduce.Job: map 92% reduce 0%
[2020-01-31 14:06:00,534] INFO mapreduce.Job: map 100% reduce 0%

<...after some time longer than usual...>

[2020-01-31 14:10:05,446] INFO mapreduce.Job: Job job_1579648183118_1085 completed successfully

我的Hadoop版本
[airflow@airflowetl root]$ hadoop version
Hadoop 3.1.1.3.1.0.0-78
Source code repository git@github.com:hortonworks/hadoop.git -r e4f82af51faec922b4804d0232a637422ec29e64
Compiled by jenkins on 2018-12-06T12:26Z
Compiled with protoc 2.5.0
From source with checksum eab9fa2a6aa38c6362c66d8df75774
This command was run using /usr/hdp/3.1.0.0-78/hadoop/hadoop-common-3.1.1.3.1.0.0-78.jar

任何有更多Hadoop经验的人都知道这里会发生什么吗?有任何调试建议吗?

最佳答案

Namenode堆主要由HDFS中存储的文件块数确定。特别是,许多小文件或一次写入的许多文件会导致大堆。

ResourceManager与名称节点不相关。它的堆将取决于正在主动跟踪的YARN作业的数量

在我维护的集群中,namenode堆为32G,我认为ResourceManager只有8GB

关于hadoop - NameNode堆使用率与ResourceManager堆使用率之间的区别(试图查找NameNode堆使用率的原因)?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/60012928/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com