Hadoop 的 Capacity Scheduler - 设置多个队列

Question

我尝试设置 2 个队列 - 队列 1、队列 2。我将这些队列的名称添加到 mapred-site.xml

    <property>
        <name>mapred.queue.names</name>
        <value>queue1,queue2</value>
    </property> 

我配置了 CapacityScheduler.xml，如下所示。

    <?xml version="1.0"?>
    <configuration>
         <property>
            <name>mapred.capacity-scheduler.maximum-system-jobs</name>
            <value>3000</value>   
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue1.capacity</name>
            <value>100</value>   
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.capacity</name>
            <value>100</value>   
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.maximum-capacity</name>
            <value>-1</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.maximum-capacity</name>
            <value>-1</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.supports-priority</name>
            <value>false</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.supports-priority</name>
            <value>false</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.minimum-user-limit-percent</name>
            <value>100</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.minimum-user-limit-percent</name>
            <value>100</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.user-limit-factor</name>
            <value>1</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.user-limit-factor</name>
            <value>1</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.maximum-initialized-active-tasks</name>
            <value>200000</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.maximum-initialized-active-tasks</name>
            <value>200000</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.maximum-initialized-active-tasks-per-user</name>
            <value>100000</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.maximum-initialized-active-tasks-per-user</name>
            <value>100000</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.init-accept-jobs-factor</name>
            <value>10</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.init-accept-jobs-factor</name>
            <value>10</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.default-supports-priority</name>
            <value>false</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.default-minimum-user-limit-percent</name>
            <value>100</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.default-user-limit-factor</name>
            <value>1</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.default-maximum-active-tasks-per-queue</name>
            <value>200000</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.default-maximum-active-tasks-per-user</name>
            <value>100000</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.default-init-accept-jobs-factor</name>
            <value>10</value>
         </property>

         <!-- Capacity scheduler Job Initialization configuration parameters -->
         <property>
            <name>mapred.capacity-scheduler.init-poll-interval</name>
            <value>5000</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.init-worker-threads</name>
            <value>5</value>
         </property>
    </configuration>

bin/start-all.sh 启动以下服务。

    17083 DataNode
    17557 TaskTracker
    17373 JobTracker
    16902 NameNode
    17279 SecondaryNameNode
    17703 Jps

我可以在

中查看 Jobtracker 的 WEB UI

    http://localhost:50030/ 

Tasktracker 的 WEB 用户界面

    http://localhost:50060/

显示“无法连接”。但几秒钟后，jobtracker 和 tasktracker 关闭。终端上的 jps 命令只显示

    17083 DataNode
    16902 NameNode
    17279 SecondaryNameNode
    17703 Jps

可能的解决方案是什么。

Answer 1

你的两个队列的容量都是 100 ，这使得容量调度器认为有几个队列每个都有 100% 的容量。我建议您将设置更改为:

<?xml version="1.0"?>
<configuration>
     <property>
        <name>mapred.capacity-scheduler.maximum-system-jobs</name>
        <value>3000</value>   
     </property>
     <property>
        <name>mapred.capacity-scheduler.queue.queue1.capacity</name>
        <value>80</value> <!-- change here -->
     </property>
     <property>
        <name>mapred.capacity-scheduler.queue.queue2.capacity</name>
        <value>20</value> <!-- change here -->   
     </property>

     <property>
        <name>mapred.capacity-scheduler.queue.queue1.maximum-capacity</name>
        <value>-1</value>
     </property>
     <property>
        <name>mapred.capacity-scheduler.queue.queue2.maximum-capacity</name>
        <value>-1</value>
     </property>