org.apache.hadoop.mapred

Class ResourceMgrDelegate

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.client.api.YarnClient

org.apache.hadoop.mapred.ResourceMgrDelegate

All Implemented Interfaces:

Closeable, AutoCloseable, Service

public class ResourceMgrDelegate
extends YarnClient

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected YarnClient	client

Constructor Summary

Constructors

Constructor and Description
ResourceMgrDelegate(YarnConfiguration conf) Delegate responsible for communicating with the Resource Manager's ApplicationClientProtocol.

Method Summary

Methods

Modifier and Type	Method and Description
YarnClientApplication	createApplication() Obtain a YarnClientApplication for a new application, which in turn contains the ApplicationSubmissionContext and GetNewApplicationResponse objects.
TaskTrackerInfo[]	getActiveTrackers()
JobStatus[]	getAllJobs()
List<QueueInfo>	getAllQueues() Get information (QueueInfo) about all queues, recursively if there is a hierarchy
Token<AMRMTokenIdentifier>	getAMRMToken(ApplicationId appId) Get the AMRM token of the application.
ApplicationId	getApplicationId()
ApplicationReport	getApplicationReport(ApplicationId appId) Get a report of the given Application.
List<ApplicationReport>	getApplications() Get a report (ApplicationReport) of all Applications in the cluster.
List<ApplicationReport>	getApplications(EnumSet<YarnApplicationState> applicationStates) Get a report (ApplicationReport) of Applications matching the given application states in the cluster.
List<ApplicationReport>	getApplications(Set<String> applicationTypes) Get a report (ApplicationReport) of Applications matching the given application types in the cluster.
List<ApplicationReport>	getApplications(Set<String> applicationTypes, EnumSet<YarnApplicationState> applicationStates) Get a report (ApplicationReport) of Applications matching the given application types and application states in the cluster.
TaskTrackerInfo[]	getBlacklistedTrackers()
List<QueueInfo>	getChildQueueInfos(String parent) Get information (QueueInfo) about all the immediate children queues of the given queue
QueueInfo[]	getChildQueues(String parent)
ClusterMetrics	getClusterMetrics()
Token	getDelegationToken(Text renewer)
String	getFilesystemName()
JobID	getNewJobID()
List<NodeReport>	getNodeReports(NodeState... states) Get a report of nodes (NodeReport) in the cluster.
long	getProtocolVersion(String arg0, long arg1)
QueueInfo	getQueue(String queueName)
QueueAclsInfo[]	getQueueAclsForCurrentUser()
List<QueueUserACLInfo>	getQueueAclsInfo() Get information about acls for current user on all the existing queues.
QueueInfo	getQueueInfo(String queueName) Get information (QueueInfo) about a given queue.
QueueInfo[]	getQueues()
Token	getRMDelegationToken(Text renewer) Get a delegation token so as to be able to talk to YARN using those tokens.
List<QueueInfo>	getRootQueueInfos() Get information (QueueInfo) about top level queues.
QueueInfo[]	getRootQueues()
String	getStagingAreaDir()
String	getSystemDir()
long	getTaskTrackerExpiryInterval()
YarnClusterMetrics	getYarnClusterMetrics() Get metrics (YarnClusterMetrics) about the cluster.
void	killApplication(ApplicationId applicationId) Kill an application identified by given ID.
protected void	serviceInit(Configuration conf) All initialization code needed by a service.
protected void	serviceStart() Actions called during the INITED to STARTED transition.
protected void	serviceStop() Actions called during the transition to the STOPPED state.
void	setJobPriority(JobID arg0, String arg1)
ApplicationId	submitApplication(ApplicationSubmissionContext appContext) Submit a new application to YARN. It is a blocking call, such that it will not return ApplicationId until the submitted application has been submitted and accepted by the ResourceManager.

Methods inherited from class org.apache.hadoop.yarn.client.api.YarnClient

createYarnClient

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

client

@InterfaceAudience.Private
protected YarnClient client

Constructor Detail

ResourceMgrDelegate

public ResourceMgrDelegate(YarnConfiguration conf)

Delegate responsible for communicating with the Resource Manager's ApplicationClientProtocol.

Parameters:

conf - the configuration object.

Method Detail

serviceInit

protected void serviceInit(Configuration conf)
                    throws Exception

Description copied from class: AbstractService

All initialization code needed by a service. This method will only ever be called once during the lifecycle of a specific service instance. Implementations do not need to be synchronized as the logic in AbstractService.init(Configuration) prevents re-entrancy. The base implementation checks to see if the subclass has created a new configuration instance, and if so, updates the base class value

Overrides:

serviceInit in class AbstractService

Parameters:

conf - configuration

Throws:

Exception - on a failure -these will be caught, possibly wrapped, and wil; trigger a service stop

serviceStart

protected void serviceStart()
                     throws Exception

Description copied from class: AbstractService

Actions called during the INITED to STARTED transition. This method will only ever be called once during the lifecycle of a specific service instance. Implementations do not need to be synchronized as the logic in AbstractService.start() prevents re-entrancy.

Overrides:

serviceStart in class AbstractService

Throws:

Exception - if needed -these will be caught, wrapped, and trigger a service stop

serviceStop

protected void serviceStop()
                    throws Exception

Description copied from class: AbstractService

Actions called during the transition to the STOPPED state. This method will only ever be called once during the lifecycle of a specific service instance. Implementations do not need to be synchronized as the logic in AbstractService.stop() prevents re-entrancy. Implementations MUST write this to be robust against failures, including checks for null references -and for the first failure to not stop other attempts to shut down parts of the service.

Overrides:

serviceStop in class AbstractService

Throws:

Exception - if needed -these will be caught and logged.

getActiveTrackers

public TaskTrackerInfo[] getActiveTrackers()
                                    throws IOException,
                                           InterruptedException

Throws:

IOException

InterruptedException

getAllJobs

public JobStatus[] getAllJobs()
                       throws IOException,
                              InterruptedException

Throws:

IOException

InterruptedException

getBlacklistedTrackers

public TaskTrackerInfo[] getBlacklistedTrackers()
                                         throws IOException,
                                                InterruptedException

Throws:

IOException

InterruptedException

getClusterMetrics

public ClusterMetrics getClusterMetrics()
                                 throws IOException,
                                        InterruptedException

Throws:

IOException

InterruptedException

getDelegationToken

public Token getDelegationToken(Text renewer)
                         throws IOException,
                                InterruptedException

Throws:

IOException

InterruptedException

getFilesystemName

public String getFilesystemName()
                         throws IOException,
                                InterruptedException

Throws:

IOException

InterruptedException

getNewJobID

public JobID getNewJobID()
                  throws IOException,
                         InterruptedException

Throws:

IOException

InterruptedException

getQueue

public QueueInfo getQueue(String queueName)
                   throws IOException,
                          InterruptedException

Throws:

IOException

InterruptedException

getQueueAclsForCurrentUser

public QueueAclsInfo[] getQueueAclsForCurrentUser()
                                           throws IOException,
                                                  InterruptedException

Throws:

IOException

InterruptedException

getQueues

public QueueInfo[] getQueues()
                      throws IOException,
                             InterruptedException

Throws:

IOException

InterruptedException

getRootQueues

public QueueInfo[] getRootQueues()
                          throws IOException,
                                 InterruptedException

Throws:

IOException

InterruptedException

getChildQueues

public QueueInfo[] getChildQueues(String parent)
                           throws IOException,
                                  InterruptedException

Throws:

IOException

InterruptedException

getStagingAreaDir

public String getStagingAreaDir()
                         throws IOException,
                                InterruptedException

Throws:

IOException

InterruptedException

getSystemDir

public String getSystemDir()
                    throws IOException,
                           InterruptedException

Throws:

IOException

InterruptedException

getTaskTrackerExpiryInterval

public long getTaskTrackerExpiryInterval()
                                  throws IOException,
                                         InterruptedException

Throws:

IOException

InterruptedException

setJobPriority

public void setJobPriority(JobID arg0,
                  String arg1)
                    throws IOException,
                           InterruptedException

Throws:

IOException

InterruptedException

getProtocolVersion

public long getProtocolVersion(String arg0,
                      long arg1)
                        throws IOException

Throws:

IOException

getApplicationId

public ApplicationId getApplicationId()

createApplication

public YarnClientApplication createApplication()
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Obtain a YarnClientApplication for a new application, which in turn contains the ApplicationSubmissionContext and GetNewApplicationResponse objects.

Specified by:

createApplication in class YarnClient

Returns:

YarnClientApplication built for a new application

Throws:

YarnException

IOException

submitApplication

public ApplicationId submitApplication(ApplicationSubmissionContext appContext)
                                throws YarnException,
                                       IOException

Description copied from class: YarnClient

Submit a new application to YARN. It is a blocking call, such that it will not return ApplicationId until the submitted application has been submitted and accepted by the ResourceManager.

Specified by:

submitApplication in class YarnClient

Parameters:

appContext - ApplicationSubmissionContext containing all the details needed to submit a new application

Returns:

ApplicationId of the accepted application

Throws:

YarnException

IOException

See Also:

YarnClient.createApplication()

killApplication

public void killApplication(ApplicationId applicationId)
                     throws YarnException,
                            IOException

Description copied from class: YarnClient

Kill an application identified by given ID.

Specified by:

killApplication in class YarnClient

Parameters:

applicationId - ApplicationId of the application that needs to be killed

Throws:

YarnException - in case of errors or if YARN rejects the request due to access-control restrictions.

IOException

See Also:

YarnClient.getQueueAclsInfo()

getApplicationReport

public ApplicationReport getApplicationReport(ApplicationId appId)
                                       throws YarnException,
                                              IOException

Description copied from class: YarnClient

Get a report of the given Application.

In secure mode, YARN verifies access to the application, queue etc. before accepting the request.

If the user does not have VIEW_APP access then the following fields in the report will be set to stubbed values:

• host - set to "N/A"
• RPC port - set to -1
• client token - set to "N/A"
• diagnostics - set to "N/A"
• tracking URL - set to "N/A"
• original tracking URL - set to "N/A"
• resource usage report - all values are -1

Specified by:

getApplicationReport in class YarnClient

Parameters:

appId - ApplicationId of the application that needs a report

Returns:

application report

Throws:

YarnException

IOException

getAMRMToken

public Token<AMRMTokenIdentifier> getAMRMToken(ApplicationId appId)
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Get the AMRM token of the application.

The AMRM token is required for AM to RM scheduling operations. For managed Application Masters Yarn takes care of injecting it. For unmanaged Applications Masters, the token must be obtained via this method and set in the UserGroupInformation of the current user.

The AMRM token will be returned only if all the following conditions are met:

the requester is the owner of the ApplicationMaster

the application master is an unmanaged ApplicationMaster

the application master is in ACCEPTED state

Else this method returns NULL.

Specified by:

getAMRMToken in class YarnClient

Parameters:

appId - ApplicationId of the application to get the AMRM token

Returns:

the AMRM token if available

Throws:

YarnException

IOException

getApplications

public List<ApplicationReport> getApplications()
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Get a report (ApplicationReport) of all Applications in the cluster.

If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in YarnClient.getApplicationReport(ApplicationId).

Specified by:

getApplications in class YarnClient

Returns:

a list of reports of all running applications

Throws:

YarnException

IOException

getApplications

public List<ApplicationReport> getApplications(Set<String> applicationTypes)
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Get a report (ApplicationReport) of Applications matching the given application types in the cluster.

If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in YarnClient.getApplicationReport(ApplicationId).

Specified by:

getApplications in class YarnClient

Returns:

a list of reports of applications

Throws:

YarnException

IOException

getApplications

public List<ApplicationReport> getApplications(EnumSet<YarnApplicationState> applicationStates)
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Get a report (ApplicationReport) of Applications matching the given application states in the cluster.

If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in YarnClient.getApplicationReport(ApplicationId).

Specified by:

getApplications in class YarnClient

Returns:

a list of reports of applications

Throws:

YarnException

IOException

getApplications

public List<ApplicationReport> getApplications(Set<String> applicationTypes,
                                      EnumSet<YarnApplicationState> applicationStates)
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Get a report (ApplicationReport) of Applications matching the given application types and application states in the cluster.

If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in YarnClient.getApplicationReport(ApplicationId).

Specified by:

getApplications in class YarnClient

Returns:

a list of reports of applications

Throws:

YarnException

IOException

getYarnClusterMetrics

public YarnClusterMetrics getYarnClusterMetrics()
                                         throws YarnException,
                                                IOException

Description copied from class: YarnClient

Get metrics (YarnClusterMetrics) about the cluster.

Specified by:

getYarnClusterMetrics in class YarnClient

Returns:

cluster metrics

Throws:

YarnException

IOException

getNodeReports

public List<NodeReport> getNodeReports(NodeState... states)
                                throws YarnException,
                                       IOException

Description copied from class: YarnClient

Get a report of nodes (NodeReport) in the cluster.

Specified by:

getNodeReports in class YarnClient

Parameters:

states - The NodeStates to filter on. If no filter states are given, nodes in all states will be returned.

Returns:

A list of node reports

Throws:

YarnException

IOException

getRMDelegationToken

public Token getRMDelegationToken(Text renewer)
                           throws YarnException,
                                  IOException

Description copied from class: YarnClient

Get a delegation token so as to be able to talk to YARN using those tokens.

Specified by:

getRMDelegationToken in class YarnClient

Parameters:

renewer - Address of the renewer who can renew these tokens when needed by securely talking to YARN.

Returns:

a delegation token (Token) that can be used to talk to YARN

Throws:

YarnException

IOException

getQueueInfo

public QueueInfo getQueueInfo(String queueName)
                       throws YarnException,
                              IOException

Description copied from class: YarnClient

Get information (QueueInfo) about a given queue.

Specified by:

getQueueInfo in class YarnClient

Parameters:

queueName - Name of the queue whose information is needed

Returns:

queue information

Throws:

YarnException - in case of errors or if YARN rejects the request due to access-control restrictions.

IOException

getAllQueues

public List<QueueInfo> getAllQueues()
                             throws YarnException,
                                    IOException

Description copied from class: YarnClient

Get information (QueueInfo) about all queues, recursively if there is a hierarchy

Specified by:

getAllQueues in class YarnClient

Returns:

a list of queue-information for all queues

Throws:

YarnException

IOException

getRootQueueInfos

public List<QueueInfo> getRootQueueInfos()
                                  throws YarnException,
                                         IOException

Description copied from class: YarnClient

Get information (QueueInfo) about top level queues.

Specified by:

getRootQueueInfos in class YarnClient

Returns:

a list of queue-information for all the top-level queues

Throws:

YarnException

IOException

getChildQueueInfos

public List<QueueInfo> getChildQueueInfos(String parent)
                                   throws YarnException,
                                          IOException

Description copied from class: YarnClient

Get information (QueueInfo) about all the immediate children queues of the given queue

Specified by:

getChildQueueInfos in class YarnClient

Parameters:

parent - Name of the queue whose child-queues' information is needed

Returns:

a list of queue-information for all queues who are direct children of the given parent queue.

Throws:

YarnException

IOException

getQueueAclsInfo

public List<QueueUserACLInfo> getQueueAclsInfo()
                                        throws YarnException,
                                               IOException

Description copied from class: YarnClient

Get information about acls for current user on all the existing queues.

Specified by:

getQueueAclsInfo in class YarnClient

Returns:

a list of queue acls (QueueUserACLInfo) for current user

Throws:

YarnException

IOException