FairScheduler (Apache Hadoop Main 2.2.0 API)

java.lang.Object
- org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

All Implemented Interfaces:

EventHandler<SchedulerEvent>, Recoverable, ResourceScheduler, YarnScheduler
```
@InterfaceAudience.LimitedPrivate(value="yarn")
@InterfaceStability.Unstable
public class FairScheduler
extends Object
implements ResourceScheduler
```
A scheduler that schedules resources between a set of queues. The scheduler keeps track of the resources used by each queue, and attempts to maintain fairness by scheduling tasks at queues whose allocations are farthest below an ideal fair distribution. The fair scheduler supports hierarchical queues. All queues descend from a queue named "root". Available resources are distributed among the children of the root queue in the typical fair scheduling fashion. Then, the children distribute the resources assigned to them to their children in the same fashion. Applications may only be scheduled on leaf queues. Queues can be specified as children of other queues by placing them as sub-elements of their parents in the fair scheduler configuration file. A queue's name starts with the names of its parents, with periods as separators. So a queue named "queue1" under the root named, would be referred to as "root.queue1", and a queue named "queue2" under a queue named "parent1" would be referred to as "root.parent1.queue2".

Field Summary

Fields
Modifier and Type	Field and Description
`protected Map<ApplicationAttemptId,FSSchedulerApp>`	`applications`
`protected boolean`	`assignMultiple`
`static Resource`	`CONTAINER_RESERVED`
`protected long`	`lastPreemptionUpdateTime`
`protected int`	`maxAssign`
`protected double`	`nodeLocalityThreshold`
`protected boolean`	`preemptionEnabled`
`protected long`	`preemptionInterval`
`protected double`	`rackLocalityThreshold`
`protected boolean`	`sizeBasedWeight`
`protected long`	`UPDATE_INTERVAL`
`protected long`	`waitTimeBeforeKill`
`protected WeightAdjuster`	`weightAdjuster`

Constructor Summary

Constructors
Constructor and Description

FairScheduler()

Constructors
Constructor and Description
`FairScheduler()`

Method Summary

Methods
Modifier and Type	Method and Description
`protected void`	`addApplication(ApplicationAttemptId applicationAttemptId, String queueName, String user)` Add a new application to the scheduler, with a given id, queue name, and user.
`Allocation`	`allocate(ApplicationAttemptId appAttemptId, List<ResourceRequest> ask, List<ContainerId> release, List<String> blacklistAdditions, List<String> blacklistRemovals)` The main api between the ApplicationMaster and the Scheduler.
`boolean`	`checkAccess(UserGroupInformation callerUGI, QueueACL acl, String queueName)` Check if the user has permission to perform the operation.
`ResourceWeights`	`getAppWeight(AppSchedulable app)`
`Clock`	`getClock()`
`Resource`	`getClusterCapacity()`
`FairSchedulerConfiguration`	`getConf()`
`RMContainerTokenSecretManager`	`getContainerTokenSecretManager()`
`org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog`	`getEventLog()`
`Resource`	`getIncrementResourceCapability()`
`Resource`	`getMaximumResourceCapability()` Get maximum allocatable `Resource`.
`Resource`	`getMinimumResourceCapability()` Get minimum allocatable `Resource`.
`double`	`getNodeLocalityThreshold()`
`SchedulerNodeReport`	`getNodeReport(NodeId nodeId)` Get node resource usage report.
`int`	`getNumClusterNodes()` Get the number of nodes available in the cluster.
`QueueInfo`	`getQueueInfo(String queueName, boolean includeChildQueues, boolean recursive)` Get queue information
`QueueManager`	`getQueueManager()`
`List<QueueUserACLInfo>`	`getQueueUserAclInfo()` Get acls for queues for current user.
`double`	`getRackLocalityThreshold()`
`QueueMetrics`	`getRootQueueMetrics()` Get the root queue for the scheduler.
`FSSchedulerApp`	`getSchedulerApp(ApplicationAttemptId appAttemptId)`
`SchedulerAppReport`	`getSchedulerAppInfo(ApplicationAttemptId appAttemptId)` Get the Scheduler app for a given app attempt Id.
`void`	`handle(SchedulerEvent event)`
`protected void`	`preemptResources(Collection<FSLeafQueue> scheds, Resource toPreempt)` Preempt a quantity of resources from a list of QueueSchedulables.
`protected void`	`preemptTasksIfNecessary()` Check for queues that need tasks preempted, either because they have been below their guaranteed share for minSharePreemptionTimeout or they have been below half their fair share for the fairSharePreemptionTimeout.
`void`	`recover(RMStateStore.RMState state)`
`void`	`reinitialize(Configuration conf, RMContext rmContext)` Re-initialize the `ResourceScheduler`.
`protected Resource`	`resToPreempt(FSLeafQueue sched, long curTime)` Return the resource amount that this queue is allowed to preempt, if any.
`protected void`	`setClock(Clock clock)`
`protected void`	`update()` Recompute the internal variables used by the scheduler - per-job weights, fair shares, deficits, minimum slot allocations, and amount of used and required resources per job.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - CONTAINER_RESERVED
```
public static final Resource CONTAINER_RESERVED
```
  - UPDATE_INTERVAL
```
protected long UPDATE_INTERVAL
```
  - lastPreemptionUpdateTime
```
protected long lastPreemptionUpdateTime
```
  - applications
```
protected Map<ApplicationAttemptId,FSSchedulerApp> applications
```
  - preemptionInterval
```
protected long preemptionInterval
```
  - waitTimeBeforeKill
```
protected long waitTimeBeforeKill
```
  - preemptionEnabled
```
protected boolean preemptionEnabled
```
  - sizeBasedWeight
```
protected boolean sizeBasedWeight
```
  - weightAdjuster
```
protected WeightAdjuster weightAdjuster
```
  - nodeLocalityThreshold
```
protected double nodeLocalityThreshold
```
  - rackLocalityThreshold
```
protected double rackLocalityThreshold
```
  - assignMultiple
```
protected boolean assignMultiple
```
  - maxAssign
```
protected int maxAssign
```
- Constructor Detail
  - FairScheduler
```
public FairScheduler()
```
- Method Detail
  - getConf
```
public FairSchedulerConfiguration getConf()
```
  - getQueueManager
```
public QueueManager getQueueManager()
```
  - update
```
protected void update()
```
    Recompute the internal variables used by the scheduler - per-job weights, fair shares, deficits, minimum slot allocations, and amount of used and required resources per job.
  - preemptTasksIfNecessary
```
protected void preemptTasksIfNecessary()
```
    Check for queues that need tasks preempted, either because they have been below their guaranteed share for minSharePreemptionTimeout or they have been below half their fair share for the fairSharePreemptionTimeout. If such queues exist, compute how many tasks of each type need to be preempted and then select the right ones using preemptTasks.
  - preemptResources
```
protected void preemptResources(Collection<FSLeafQueue> scheds,
                    Resource toPreempt)
```
    Preempt a quantity of resources from a list of QueueSchedulables. The policy for this is to pick apps from queues that are over their fair share, but make sure that no queue is placed below its fair share in the process. We further prioritize preemption by choosing containers with lowest priority to preempt.
  - resToPreempt
```
protected Resource resToPreempt(FSLeafQueue sched,
                    long curTime)
```
    Return the resource amount that this queue is allowed to preempt, if any. If the queue has been below its min share for at least its preemption timeout, it should preempt the difference between its current share and this min share. If it has been below half its fair share for at least the fairSharePreemptionTimeout, it should preempt enough tasks to get up to its full fair share. If both conditions hold, we preempt the max of the two amounts (this shouldn't happen unless someone sets the timeouts to be identical for some reason).
  - getContainerTokenSecretManager
```
public RMContainerTokenSecretManager getContainerTokenSecretManager()
```
  - getAppWeight
```
public ResourceWeights getAppWeight(AppSchedulable app)
```
  - getMinimumResourceCapability
```
public Resource getMinimumResourceCapability()
```
    Description copied from interface: YarnScheduler
    
    Get minimum allocatable Resource.
    
    Specified by:
    
    getMinimumResourceCapability in interface YarnScheduler
    
    Returns:
    minimum allocatable resource
  - getIncrementResourceCapability
```
public Resource getIncrementResourceCapability()
```
  - getMaximumResourceCapability
```
public Resource getMaximumResourceCapability()
```
    Description copied from interface: YarnScheduler
    
    Get maximum allocatable Resource.
    
    Specified by:
    
    getMaximumResourceCapability in interface YarnScheduler
    
    Returns:
    maximum allocatable resource
  - getNodeLocalityThreshold
```
public double getNodeLocalityThreshold()
```
  - getRackLocalityThreshold
```
public double getRackLocalityThreshold()
```
  - getClusterCapacity
```
public Resource getClusterCapacity()
```
  - getClock
```
public Clock getClock()
```
  - setClock
```
protected void setClock(Clock clock)
```
  - getEventLog
```
public org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog getEventLog()
```
  - addApplication
```
protected void addApplication(ApplicationAttemptId applicationAttemptId,
                  String queueName,
                  String user)
```
    Add a new application to the scheduler, with a given id, queue name, and user. This will accept a new app even if the user or queue is above configured limits, but the app will not be marked as runnable.
  - allocate
```
public Allocation allocate(ApplicationAttemptId appAttemptId,
                  List<ResourceRequest> ask,
                  List<ContainerId> release,
                  List<String> blacklistAdditions,
                  List<String> blacklistRemovals)
```
    Description copied from interface: YarnScheduler
    
    The main api between the ApplicationMaster and the Scheduler. The ApplicationMaster is updating his future resource requirements and may release containers he doens't need.
    
    Specified by:
    
    allocate in interface YarnScheduler
    
    Returns:
    the Allocation for the application
  - getNodeReport
```
public SchedulerNodeReport getNodeReport(NodeId nodeId)
```
    Description copied from interface: YarnScheduler
    
    Get node resource usage report.
    
    Specified by:
    
    getNodeReport in interface YarnScheduler
    
    Returns:
    the SchedulerNodeReport for the node or null if nodeId does not point to a defined node.
  - getSchedulerApp
```
public FSSchedulerApp getSchedulerApp(ApplicationAttemptId appAttemptId)
```
  - getSchedulerAppInfo
```
public SchedulerAppReport getSchedulerAppInfo(ApplicationAttemptId appAttemptId)
```
    Description copied from interface: YarnScheduler
    
    Get the Scheduler app for a given app attempt Id.
    
    Specified by:
    
    getSchedulerAppInfo in interface YarnScheduler
    
    Parameters:
    appAttemptId - the id of the application attempt
    
    Returns:
    SchedulerApp for this given attempt.
  - getRootQueueMetrics
```
public QueueMetrics getRootQueueMetrics()
```
    Description copied from interface: YarnScheduler
    
    Get the root queue for the scheduler.
    
    Specified by:
    
    getRootQueueMetrics in interface YarnScheduler
    
    Returns:
    the root queue for the scheduler.
  - handle
```
public void handle(SchedulerEvent event)
```
    Specified by:
    
    handle in interface EventHandler<SchedulerEvent>
  - recover
```
public void recover(RMStateStore.RMState state)
             throws Exception
```
    Specified by:
    
    recover in interface Recoverable
    
    Throws:
    
    Exception
  - reinitialize
```
public void reinitialize(Configuration conf,
                RMContext rmContext)
                  throws IOException
```
    Description copied from interface: ResourceScheduler
    
    Re-initialize the ResourceScheduler.
    
    Specified by:
    
    reinitialize in interface ResourceScheduler
    
    Parameters:
    conf - configuration
    
    Throws:
    
    IOException
  - getQueueInfo
```
public QueueInfo getQueueInfo(String queueName,
                     boolean includeChildQueues,
                     boolean recursive)
                       throws IOException
```
    Description copied from interface: YarnScheduler
    
    Get queue information
    
    Specified by:
    
    getQueueInfo in interface YarnScheduler
    
    Parameters:
    queueName - queue name
    includeChildQueues - include child queues?
    recursive - get children queues?
    
    Returns:
    queue information
    
    Throws:
    
    IOException
  - getQueueUserAclInfo
```
public List<QueueUserACLInfo> getQueueUserAclInfo()
```
    Description copied from interface: YarnScheduler
    
    Get acls for queues for current user.
    
    Specified by:
    
    getQueueUserAclInfo in interface YarnScheduler
    
    Returns:
    acls for queues for current user
  - getNumClusterNodes
```
public int getNumClusterNodes()
```
    Description copied from interface: YarnScheduler
    
    Get the number of nodes available in the cluster.
    
    Specified by:
    
    getNumClusterNodes in interface YarnScheduler
    
    Returns:
    the number of available nodes.
  - checkAccess
```
public boolean checkAccess(UserGroupInformation callerUGI,
                  QueueACL acl,
                  String queueName)
```
    Description copied from interface: YarnScheduler
    
    Check if the user has permission to perform the operation. If the user has QueueACL.ADMINISTER_QUEUE permission, this user can view/modify the applications in this queue
    
    Specified by:
    
    checkAccess in interface YarnScheduler
    
    Returns:
    true if the user has the permission, false otherwise

Class FairScheduler

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

CONTAINER_RESERVED

UPDATE_INTERVAL

lastPreemptionUpdateTime

applications

preemptionInterval

waitTimeBeforeKill

preemptionEnabled

sizeBasedWeight

weightAdjuster

nodeLocalityThreshold

rackLocalityThreshold

assignMultiple

maxAssign

Constructor Detail

FairScheduler

Method Detail

getConf

getQueueManager

update

preemptTasksIfNecessary

preemptResources

resToPreempt

getContainerTokenSecretManager

getAppWeight

getMinimumResourceCapability

getIncrementResourceCapability

getMaximumResourceCapability

getNodeLocalityThreshold

getRackLocalityThreshold

getClusterCapacity

getClock

setClock

getEventLog

addApplication

allocate

getNodeReport

getSchedulerApp

getSchedulerAppInfo

getRootQueueMetrics

handle

recover

reinitialize

getQueueInfo

getQueueUserAclInfo

getNumClusterNodes

checkAccess