hadoop - 容量计划程序队列问题-6ren

hadoop - 容量计划程序队列问题

转载作者：行者123 更新时间：2023-12-02 21:50:00

我正在尝试在我的独立节点中设置容量调度程序队列，但与此有关。

我能够通过UI在Cloudera Manager中创建能够在队列中看到的队列，但是当我登录并尝试执行mapred queue -list时，它不会显示我已设置的队列。它仅显示默认队列，我已从配置中省略了该队列。

容量调度程序配置:

<?xml version="1.0"?>
<configuration>

<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>card,bank,digital</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.capacity</name>
<value>100</value>
</property>


<property>
<name>yarn.scheduler.capacity.root.digital.capacity</name>
<value>41</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.card.capacity</name>
<value>29</value>
</property>


<property>
<name>yarn.scheduler.capacity.root.bank.capacity</name>
<value>30</value>
</property>

<property>
<name>yarn.scheduler.capacity.root.digital.acl_submit_applications</name>
<value>cloudera</value>
 </property>

下面的日志

    2014-02-18 08:09:54,940 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing card
capacity = 0.29 [= (float) configuredCapacity / 100 ]
asboluteCapacity = 0.29 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 2900 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 2900 [= (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit / 100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory) / maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS:  [= configuredAcls ]

2014-02-18 08:09:54,941 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: card: capacity=0.29, absoluteCapacity=0.29, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,941 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing bank
capacity = 0.3 [= (float) configuredCapacity / 100 ]
asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit / 100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory) / maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS:  [= configuredAcls ]

2014-02-18 08:09:54,941 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: bank: capacity=0.3, absoluteCapacity=0.3, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing digital
capacity = 0.41 [= (float) configuredCapacity / 100 ]
asboluteCapacity = 0.41 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 4100 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 4100 [= (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit / 100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory) / maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS:cloudera  [= configuredAcls ]

2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: digital: capacity=0.41, absoluteCapacity=0.41, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: root: numChildQueue= 3, capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>usedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0, asboluteMaxCapacity=1.0, state=RUNNING, acls=ADMINISTER_QUEUE:*SUBMIT_APPLICATIONS:*
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing card
capacity = 0.29 [= (float) configuredCapacity / 100 ]
asboluteCapacity = 0.29 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 2900 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 2900 [= (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit / 100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory) / maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS:  [= configuredAcls ]

2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root: re-configured queue: card: capacity=0.29, absoluteCapacity=0.29, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing bank
capacity = 0.3 [= (float) configuredCapacity / 100 ]
asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit / 100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory) / maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS:  [= configuredAcls ]

2014-02-18 08:09:54,943 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root: re-configured queue: bank: capacity=0.3, absoluteCapacity=0.3, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,943 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing digital
capacity = 0.41 [= (float) configuredCapacity / 100 ]
asboluteCapacity = 0.41 [= parentAbsoluteCapacity * capacity ]
maxCapacity = 1.0 [= configuredMaxCapacity ]
absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]
userLimit = 100 [= configuredUserLimit ]
userLimitFactor = 1.0 [= configuredUserLimitFactor ]
maxApplications = 4100 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)]
maxApplicationsPerUser = 4100 [= (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor) ]
maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ]
maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]
maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit / 100.0f) * userLimitFactor),1) ]
usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * absoluteCapacity)]
absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]
maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]
minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory) / maximumAllocationMemory ]
numContainers = 0 [= currentNumContainers ]
state = RUNNING [= configuredState ]
acls = ADMINISTER_QUEUE: SUBMIT_APPLICATIONS:cloudera  [= configuredAcls ]

2014-02-18 08:09:54,943 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root: re-configured queue: digital: capacity=0.41, absoluteCapacity=0.41, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0
2014-02-18 08:09:54,943 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=cloudera IP=127.0.0.1    OPERATION=refreshQueues TARGET=AdminService RESULT=SUCCESS
2014-02-18 08:10:19,772 INFO org.apache.hadoop.yarn.server.resourcemanager.AdminService: RM Admin: refreshAdminAcls invoked by user cloudera
2014-02-18 08:10:19,792 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=cloudera IP=127.0.0.1    OPERATION=refreshAdminAcls  TARGET=AdminService RESULT=SUCCESS

映射命令输出

    Queue Name : default
Queue State : running
Scheduling Info : Queue configuration
Capacity Percentage: 100.0%
User Limit: 100%
Priority Supported: NO
-------------
Map tasks
Capacity: 2 slots
Used capacity: 0 (0.0% of Capacity)
Running tasks: 0
-------------
Reduce tasks
Capacity: 2 slots
Used capacity: 0 (0.0% of Capacity)
Running tasks: 0
-------------
Job info
Number of Waiting Jobs: 0
Number of Initializing Jobs: 0
Number of users who have submitted jobs: 0

最佳答案

更改capacity-scheduler.xml中的以下属性，以在用户队列中使用100％的资源
yarn.scheduler.capacity.root.it.user-limit-factor=2 yarn.scheduler.capacity.root.price.user-limit-factor=1

关于hadoop - 容量计划程序队列问题，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/21861616/

文章推荐： c# - 工作单元测试的单元测试失败

文章推荐： php - 无法发送电子邮件

文章推荐： apache - 什么是Amazon S3的开源替代品？

计划 - eq？比较2个字符串？
我的程序有问题。我有一个比较两个字符串的条件: (if (eq? (exp1) (exp2))) 当 exp1 给我一个字符串，exp2 给我一个字符串。可以肯定的是，当我更改“eq？”时到“=”，
GWT future 计划
我们有多种主要使用 GWT 开发的产品，目前由我们的最终客户使用。想知道 GWT 的路线图。我得到了一些非官方的更新，谷歌正在将 GWT 中开发的产品转移到其他一些新技术。这是真的吗？ GWT 的长
Jenkins 定期构建 - 计划
我希望每 15 分钟定期构建一次。我在网上看过，我正在使用这个时间表:*/15 * * * * Jenkins 告诉我使用 H/15 * * * * 来平均分配负载而不是 */15 * * * * 有
计划，电话/抄送
所以我正试图在 Scheme 中找出整个 call/cc 的东西。下面是我正在使用的代码: (+ 1 (call/cc (lambda (k) (if (number? k)
计划，电话/抄送
所以我正试图在 Scheme 中找出整个 call/cc 的东西。下面是我正在使用的代码: (+ 1 (call/cc (lambda (k) (if (number? k)
Azure 计划 Web 作业有时会触发两次
我们有一个 Azure WebJob，计划在 UTC 每天上午 8:00 运行(CRON - 0 00 08 * * *)。大多数时候它都会正确触发，但有时会触发两次(第二次运行)第一次运行后约 10
terraform - 命令行参数过多 Terraform 计划
我是 Terraform 的新手。我正在尝试通过 azure 管道创建一个简单的存储帐户，但是当我运行我的管道时，我收到错误“太多命令行参数”。我很震惊，我不知道自己做错了什么。有人可以帮忙吗。这是
ballerina - 如何终止芭蕾舞 Actor 计划
我想在某些逻辑中间停止芭蕾舞 Actor 程序。如何使用代码停止 ballerina 中正在运行的程序？我正在寻找相当于 java 中的 System.exit(0) 的东西。最佳答案我相信您正在
agile - 与多层团队一起进行 Scrum 计划
关闭。这个问题不符合Stack Overflow guidelines .它目前不接受答案。想改进这个问题？将问题更新为 on-topic对于堆栈溢出。 8年前关闭。 Improve this qu
Azure 计划 Web 作业有时会触发两次
我们有一个 Azure WebJob，计划在 UTC 每天上午 8:00 运行(CRON - 0 00 08 * * *)。大多数时候它都会正确触发，但有时会触发两次(第二次运行)第一次运行后约 10
terraform - 命令行参数过多 Terraform 计划
我是 Terraform 的新手。我正在尝试通过 azure 管道创建一个简单的存储帐户，但是当我运行我的管道时，我收到错误“太多命令行参数”。我很震惊，我不知道自己做错了什么。有人可以帮忙吗。这是
syntax - 为什么这是合法的( Racket )计划？
我正在浏览 htdp 并在一开始的某个地方发现了这个:- Explain why the following sentences are illegal definitions: 1. (define
php - 处理续订月份的成员(member)计划
我正在使用 Laravel 开发成员(member)门户。成员(member)资格有不同的类别，例如1) 单人2) 成人3) 家庭以及不同价格的所有类型。我有一个 plans 表和 plans_s
php - 正在执行 MySQL 计划？
我使用 DreamHost 作为我的网站的服务器，并且我尝试每天、每周和每月执行某个 MySQL 查询来更改我的网站的数据库。我开始在本地主机上使用事件调度程序，然后我发现我无法在 DreamHost
bash - 每两周一次的 Crontab 计划
这周我的 crontab 作业发生了一个问题。设置如下，每两周正常运行一次，直到现在。 10 06 * * 1 test $(($(date +\%W)\%2)) -eq 0 && echo 'te
linux - 计划 cron 作业错误
编写了一个简单的脚本，它将在日志文件中写入日期时间戳，并且每次运行该脚本时，它都会附加到该日志文件中。 #!/bin/sh echo $(date) >> log.txt 当我尝试每 1 分钟安排一次
c - 不了解 PIPE 计划
我对 PIPE 的了解是它用于单向通信，它有助于在两个相关进程之间进行通信。我从一本书中得到了下面的 PIPE 编程代码示例。我正在尝试使用 printf 理解代码并在代码的每一行之后打印出所有点。但
计划， Racket 帮助。想要返回符合条件的数字
代码如下: (define make-simple-sv-num (lambda (delare) (let ((tal (random-from-to 100000 1000000)))
math - ( Racket /计划)减法产生的结果非常小
我目前正在使用“How To Design Programs”——使用 Scheme/Racket；我在 Scheme 的 R5RS 版本中遇到了一个非常奇特的功能。在进行简单的减法时，尽管使用的是
ruby - 确定 ice_cube 计划
我想确定时间表的详细信息。例如: 我有一个事件的时间表:event.schedule "Every 3 months on the 10th day of the month" 由哈希表示: {

行者123

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

hadoop - 容量计划程序队列问题