目前使用一個User(hdp0)，五個pool(default,client1,client2,client3,client4)
User(hdp0)提交四個job都只在default pool，其他的四個pool都沒收到job，而且default pool使用全部群集的資源
那如何將各個job分送到各個pool，是需要下參數指定每個job到某個pool嗎?還是?
使用hadoop1.2.1版本，fair scheduler的配置如下:
mared-site.xml:
代碼:
<property>
<name>mapred.jobtracker.taskScheduler</name>
<value>org.apache.hadoop.mapred.FairScheduler</value>
</property>
<property>
<name>mapred.fairscheduler.allocation.file</name>
<value>/home/hdp0/hadoop-1.2.1/conf/fair-scheduler.xml</value>
</property>
<property>
<name>mapred.fairscheduler.preemption</name>
<value>true</value>
</property>
<property>
<name>mapred.fairscheduler.assignmultiple</name>
<value>true</value>
</property>
<property>
<name>mapred.fairscheduler.poolnameproperty</name>
<value>(default,low),(client1,high),(client2,high),(client3,low),(client4,low)</value>
description>job.set("mapred.queue.name",pool);//pool is set to either 'high' or 'low'</description>
</property>
<property>
<name>mapred.fairscheduler.preemption.only.log</name>
<value>true</value>
</property>
<property>
<name>mapred.fairscheduler.preemption.interval</name>
<value>15000</value>
</property>
<property>
<name>mapred.queue.names</name>
<value>default,client1,client2,client3,client4</value>
</property>
<property>
<name>mapred.acls.enabled</name>
<value>true</value>
</property>
fair-scheduler.xml:
代碼:
<allocations>
<pool name="client1">
<minMaps>6</minMaps>
<minReduces>0</minReduces>
<maxRunningJobs>3</maxRunningJobs>
<weight>2.0</weight>
<minSharePreemptionTimeout>30</minSharePreemptionTimeout>
</pool>
<pool name="client2">
<minMaps>6</minMaps>
<minReduces>0</minReduces>
<maxRunningJobs>3</maxRunningJobs>
<weight>2.0</weight>
<minSharePreemptionTimeout>30</minSharePreemptionTimeout>
</pool>
<pool name="client3">
<minMaps>2</minMaps>
<minReduces>0</minReduces>
<maxRunningJobs>2</maxRunningJobs>
<weight>1.0</weight>
<minSharePreemptionTimeout>30</minSharePreemptionTimeout>
</pool>
<pool name="client4">
<minMaps>2</minMaps>
<minReduces>0</minReduces>
<maxRunningJobs>2</maxRunningJobs>
<weight>1.0</weight>
<minSharePreemptionTimeout>30</minSharePreemptionTimeout>
</pool>
<user name="hdp0">
<maxRunningJobs>20</maxRunningJobs>
</user>
<poolMaxJobsDefault>20</poolMaxJobsDefault>
<userMaxJobsDefault>20</userMaxJobsDefault>
<defaultMinSharePreemptionTimeout>600</defaultMinSharePreemptionTimeout>
<fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
</allocations>
mapred-queue-acls.xml:
代碼:
<property>
<name>mapred.queue.default.acl-submit-job</name>
<value> </value>
</property>
<property>
<name>mapred.queue.default.acl-administer-jobs</name>
<value> </value>
</property>
<property>
<name>mapred.queue.client1.acl-submit-job</name>
<value>hdp0</value>
</property>
<property>
<name>mapred.queue.client2.acl-submit-job</name>
<value>hdp0</value>
</property>
<property>
<name>mapred.queue.client3.acl-submit-job</name>
<value>hdp0</value>
</property>
<property>
<name>mapred.queue.client4.acl-submit-job</name>
<value>hdp0</value>
</property>
<property>
<name>mapred.queue.client1.acl-administer-jobs</name>
<value>hdp0</value>
</property>
<property>
<name>mapred.queue.client2.acl-administer-jobs</name>
<value>hdp0</value>
</property>
<property>
<name>mapred.queue.client3.acl-administer-jobs</name>
<value>hdp0</value>
</property>
<property>
<name>mapred.queue.client4.acl-administer-jobs</name>
<value>hdp0</value>
</property>
格式編排有點跑掉，抱歉!
希望有人能幫我解答，謝謝!
登入
註冊
問答集
搜尋
