org.apache.hadoop.mapred

Class YARNRunner

java.lang.Object

org.apache.hadoop.mapred.YARNRunner

All Implemented Interfaces:

VersionedProtocol, ClientProtocol

public class YARNRunner
extends Object
implements ClientProtocol

This class enables the current JobClient (0.22 hadoop) to run on YARN.

Field Summary

Fields inherited from interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

versionID

Constructor Summary

Constructors

Constructor and Description
YARNRunner(Configuration conf) Yarn runner incapsulates the client interface of yarn
YARNRunner(Configuration conf, ResourceMgrDelegate resMgrDelegate) Similar to YARNRunner(Configuration) but allowing injecting ResourceMgrDelegate.
YARNRunner(Configuration conf, ResourceMgrDelegate resMgrDelegate, ClientCache clientCache) Similar to YARNRunner(Configuration, ResourceMgrDelegate) but allowing injecting ClientCache.

Method Summary

Methods

Modifier and Type	Method and Description
void	cancelDelegationToken(Token<DelegationTokenIdentifier> arg0) Cancel a delegation token.
ApplicationSubmissionContext	createApplicationSubmissionContext(Configuration jobConf, String jobSubmitDir, Credentials ts)
TaskTrackerInfo[]	getActiveTrackers() Get all active trackers in cluster.
JobStatus[]	getAllJobs() Get all the jobs submitted.
TaskTrackerInfo[]	getBlacklistedTrackers() Get all blacklisted trackers in cluster.
QueueInfo[]	getChildQueues(String parent) Returns immediate children of queueName.
ClusterMetrics	getClusterMetrics() Get the current status of the cluster
Token<DelegationTokenIdentifier>	getDelegationToken(Text renewer) Get a new delegation token.
String	getFilesystemName() A MapReduce system always operates on a single filesystem.
Counters	getJobCounters(JobID arg0) Grab the current job counters
String	getJobHistoryDir() Gets the directory location of the completed job history files.
JobStatus	getJobStatus(JobID jobID) Grab a handle to a job that is already known to the JobTracker.
Cluster.JobTrackerStatus	getJobTrackerStatus() Get the JobTracker's status.
LogParams	getLogFileParams(JobID jobID, TaskAttemptID taskAttemptID) Gets the location of the log file for a job if no taskAttemptId is specified, otherwise gets the log location for the taskAttemptId.
JobID	getNewJobID() Allocate a name for the job.
ProtocolSignature	getProtocolSignature(String protocol, long clientVersion, int clientMethodsHash) Return protocol version corresponding to protocol interface.
long	getProtocolVersion(String arg0, long arg1) Return protocol version corresponding to protocol interface.
QueueInfo	getQueue(String queueName) Gets scheduling information associated with the particular Job queue
QueueAclsInfo[]	getQueueAclsForCurrentUser() Gets the Queue ACLs for current user
AccessControlList	getQueueAdmins(String arg0) Get the administrators of the given job-queue.
QueueInfo[]	getQueues() Gets set of Queues associated with the Job Tracker
QueueInfo[]	getRootQueues() Gets the root level queues.
String	getStagingAreaDir() Get a hint from the JobTracker where job-specific files are to be placed.
String	getSystemDir() Grab the jobtracker system directory path where job-specific files are to be placed.
TaskCompletionEvent[]	getTaskCompletionEvents(JobID arg0, int arg1, int arg2) Get task completion events for the jobid, starting from fromEventId.
String[]	getTaskDiagnostics(TaskAttemptID arg0) Get the diagnostics for a given task in a given job
TaskReport[]	getTaskReports(JobID jobID, TaskType taskType) Grab a bunch of info on the tasks that make up the job
long	getTaskTrackerExpiryInterval()
void	killJob(JobID arg0) Kill the indicated job
boolean	killTask(TaskAttemptID arg0, boolean arg1) Kill indicated task attempt.
long	renewDelegationToken(Token<DelegationTokenIdentifier> arg0) Renew an existing delegation token
void	setJobPriority(JobID arg0, String arg1) Set the priority of the specified job
void	setResourceMgrDelegate(ResourceMgrDelegate resMgrDelegate)
JobStatus	submitJob(JobID jobId, String jobSubmitDir, Credentials ts) Submit a Job for execution.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

YARNRunner

public YARNRunner(Configuration conf)

Yarn runner incapsulates the client interface of yarn

Parameters:

conf - the configuration object for the client

YARNRunner

public YARNRunner(Configuration conf,
          ResourceMgrDelegate resMgrDelegate)

Similar to YARNRunner(Configuration) but allowing injecting ResourceMgrDelegate. Enables mocking and testing.

Parameters:

conf - the configuration object for the client

resMgrDelegate - the resourcemanager client handle.

YARNRunner

public YARNRunner(Configuration conf,
          ResourceMgrDelegate resMgrDelegate,
          ClientCache clientCache)

Similar to YARNRunner(Configuration, ResourceMgrDelegate) but allowing injecting ClientCache. Enable mocking and testing.

Parameters:

conf - the configuration object

resMgrDelegate - the resource manager delegate

clientCache - the client cache object.

Method Detail

setResourceMgrDelegate

@InterfaceAudience.Private
public void setResourceMgrDelegate(ResourceMgrDelegate resMgrDelegate)

cancelDelegationToken

public void cancelDelegationToken(Token<DelegationTokenIdentifier> arg0)
                           throws IOException,
                                  InterruptedException

Description copied from interface: ClientProtocol

Cancel a delegation token.

Specified by:

cancelDelegationToken in interface ClientProtocol

Parameters:

arg0 - the token to cancel

Throws:

IOException

InterruptedException

getActiveTrackers

public TaskTrackerInfo[] getActiveTrackers()
                                    throws IOException,
                                           InterruptedException

Description copied from interface: ClientProtocol

Get all active trackers in cluster.

Specified by:

getActiveTrackers in interface ClientProtocol

Returns:

array of TaskTrackerInfo

Throws:

IOException

InterruptedException

getAllJobs

public JobStatus[] getAllJobs()
                       throws IOException,
                              InterruptedException

Description copied from interface: ClientProtocol

Get all the jobs submitted.

Specified by:

getAllJobs in interface ClientProtocol

Returns:

array of JobStatus for the submitted jobs

Throws:

IOException

InterruptedException

getBlacklistedTrackers

public TaskTrackerInfo[] getBlacklistedTrackers()
                                         throws IOException,
                                                InterruptedException

Description copied from interface: ClientProtocol

Get all blacklisted trackers in cluster.

Specified by:

getBlacklistedTrackers in interface ClientProtocol

Returns:

array of TaskTrackerInfo

Throws:

IOException

InterruptedException

getClusterMetrics

public ClusterMetrics getClusterMetrics()
                                 throws IOException,
                                        InterruptedException

Description copied from interface: ClientProtocol

Get the current status of the cluster

Specified by:

getClusterMetrics in interface ClientProtocol

Returns:

summary of the state of the cluster

Throws:

IOException

InterruptedException

getDelegationToken

public Token<DelegationTokenIdentifier> getDelegationToken(Text renewer)
                                                    throws IOException,
                                                           InterruptedException

Description copied from interface: ClientProtocol

Get a new delegation token.

Specified by:

getDelegationToken in interface ClientProtocol

Parameters:

renewer - the user other than the creator (if any) that can renew the token

Returns:

the new delegation token

Throws:

IOException

InterruptedException

getFilesystemName

public String getFilesystemName()
                         throws IOException,
                                InterruptedException

Description copied from interface: ClientProtocol

A MapReduce system always operates on a single filesystem. This function returns the fs name. ('local' if the localfs; 'addr:port' if dfs). The client can then copy files into the right locations prior to submitting the job.

Specified by:

getFilesystemName in interface ClientProtocol

Throws:

IOException

InterruptedException

getNewJobID

public JobID getNewJobID()
                  throws IOException,
                         InterruptedException

Description copied from interface: ClientProtocol

Allocate a name for the job.

Specified by:

getNewJobID in interface ClientProtocol

Returns:

a unique job name for submitting jobs.

Throws:

IOException

InterruptedException

getQueue

public QueueInfo getQueue(String queueName)
                   throws IOException,
                          InterruptedException

Description copied from interface: ClientProtocol

Gets scheduling information associated with the particular Job queue

Specified by:

getQueue in interface ClientProtocol

Parameters:

queueName - Queue Name

Returns:

Scheduling Information of the Queue

Throws:

IOException

InterruptedException

getQueueAclsForCurrentUser

public QueueAclsInfo[] getQueueAclsForCurrentUser()
                                           throws IOException,
                                                  InterruptedException

Description copied from interface: ClientProtocol

Gets the Queue ACLs for current user

Specified by:

getQueueAclsForCurrentUser in interface ClientProtocol

Returns:

array of QueueAclsInfo object for current user.

Throws:

IOException

InterruptedException

getQueues

public QueueInfo[] getQueues()
                      throws IOException,
                             InterruptedException

Description copied from interface: ClientProtocol

Gets set of Queues associated with the Job Tracker

Specified by:

getQueues in interface ClientProtocol

Returns:

Array of the Queue Information Object

Throws:

IOException

InterruptedException

getRootQueues

public QueueInfo[] getRootQueues()
                          throws IOException,
                                 InterruptedException

Description copied from interface: ClientProtocol

Gets the root level queues.

Specified by:

getRootQueues in interface ClientProtocol

Returns:

array of JobQueueInfo object.

Throws:

IOException

InterruptedException

getChildQueues

public QueueInfo[] getChildQueues(String parent)
                           throws IOException,
                                  InterruptedException

Description copied from interface: ClientProtocol

Returns immediate children of queueName.

Specified by:

getChildQueues in interface ClientProtocol

Returns:

array of JobQueueInfo which are children of queueName

Throws:

IOException

InterruptedException

getStagingAreaDir

public String getStagingAreaDir()
                         throws IOException,
                                InterruptedException

Description copied from interface: ClientProtocol

Get a hint from the JobTracker where job-specific files are to be placed.

Specified by:

getStagingAreaDir in interface ClientProtocol

Returns:

the directory where job-specific files are to be placed.

Throws:

IOException

InterruptedException

getSystemDir

public String getSystemDir()
                    throws IOException,
                           InterruptedException

Description copied from interface: ClientProtocol

Grab the jobtracker system directory path where job-specific files are to be placed.

Specified by:

getSystemDir in interface ClientProtocol

Returns:

the system directory where job-specific files are to be placed.

Throws:

IOException

InterruptedException

getTaskTrackerExpiryInterval

public long getTaskTrackerExpiryInterval()
                                  throws IOException,
                                         InterruptedException

Specified by:

getTaskTrackerExpiryInterval in interface ClientProtocol

Throws:

IOException

InterruptedException

submitJob

public JobStatus submitJob(JobID jobId,
                  String jobSubmitDir,
                  Credentials ts)
                    throws IOException,
                           InterruptedException

Description copied from interface: ClientProtocol

Submit a Job for execution. Returns the latest profile for that job.

Specified by:

submitJob in interface ClientProtocol

Throws:

IOException

InterruptedException

createApplicationSubmissionContext

public ApplicationSubmissionContext createApplicationSubmissionContext(Configuration jobConf,
                                                              String jobSubmitDir,
                                                              Credentials ts)
                                                                throws IOException

Throws:

IOException

setJobPriority

public void setJobPriority(JobID arg0,
                  String arg1)
                    throws IOException,
                           InterruptedException

Description copied from interface: ClientProtocol

Set the priority of the specified job

Specified by:

setJobPriority in interface ClientProtocol

Parameters:

arg0 - ID of the job

arg1 - Priority to be set for the job

Throws:

IOException

InterruptedException

getProtocolVersion

public long getProtocolVersion(String arg0,
                      long arg1)
                        throws IOException

Description copied from interface: VersionedProtocol

Return protocol version corresponding to protocol interface.

Specified by:

getProtocolVersion in interface VersionedProtocol

Parameters:

arg0 - The classname of the protocol interface

arg1 - The version of the protocol that the client speaks

Returns:

the version that the server will speak

Throws:

IOException - if any IO error occurs

renewDelegationToken

public long renewDelegationToken(Token<DelegationTokenIdentifier> arg0)
                          throws IOException,
                                 InterruptedException

Description copied from interface: ClientProtocol

Renew an existing delegation token

Specified by:

renewDelegationToken in interface ClientProtocol

Parameters:

arg0 - the token to renew

Returns:

the new expiration time

Throws:

IOException

InterruptedException

getJobCounters

public Counters getJobCounters(JobID arg0)
                        throws IOException,
                               InterruptedException

Description copied from interface: ClientProtocol

Grab the current job counters

Specified by:

getJobCounters in interface ClientProtocol

Throws:

IOException

InterruptedException

getJobHistoryDir

public String getJobHistoryDir()
                        throws IOException,
                               InterruptedException

Description copied from interface: ClientProtocol

Gets the directory location of the completed job history files.

Specified by:

getJobHistoryDir in interface ClientProtocol

Throws:

IOException

InterruptedException

getJobStatus

public JobStatus getJobStatus(JobID jobID)
                       throws IOException,
                              InterruptedException

Description copied from interface: ClientProtocol

Grab a handle to a job that is already known to the JobTracker.

Specified by:

getJobStatus in interface ClientProtocol

Returns:

Status of the job, or null if not found.

Throws:

IOException

InterruptedException

getTaskCompletionEvents

public TaskCompletionEvent[] getTaskCompletionEvents(JobID arg0,
                                            int arg1,
                                            int arg2)
                                              throws IOException,
                                                     InterruptedException

Description copied from interface: ClientProtocol

Get task completion events for the jobid, starting from fromEventId. Returns empty array if no events are available.

Specified by:

getTaskCompletionEvents in interface ClientProtocol

Parameters:

arg0 - job id

arg1 - event id to start from.

arg2 - the max number of events we want to look at

Returns:

array of task completion events.

Throws:

IOException

InterruptedException

getTaskDiagnostics

public String[] getTaskDiagnostics(TaskAttemptID arg0)
                            throws IOException,
                                   InterruptedException

Description copied from interface: ClientProtocol

Get the diagnostics for a given task in a given job

Specified by:

getTaskDiagnostics in interface ClientProtocol

Parameters:

arg0 - the id of the task

Returns:

an array of the diagnostic messages

Throws:

IOException

InterruptedException

getTaskReports

public TaskReport[] getTaskReports(JobID jobID,
                          TaskType taskType)
                            throws IOException,
                                   InterruptedException

Description copied from interface: ClientProtocol

Grab a bunch of info on the tasks that make up the job

Specified by:

getTaskReports in interface ClientProtocol

Throws:

IOException

InterruptedException

killJob

public void killJob(JobID arg0)
             throws IOException,
                    InterruptedException

Description copied from interface: ClientProtocol

Kill the indicated job

Specified by:

killJob in interface ClientProtocol

Throws:

IOException

InterruptedException

killTask

public boolean killTask(TaskAttemptID arg0,
               boolean arg1)
                 throws IOException,
                        InterruptedException

Description copied from interface: ClientProtocol

Kill indicated task attempt.

Specified by:

killTask in interface ClientProtocol

Parameters:

arg0 - the id of the task to kill.

arg1 - if true the task is failed and added to failed tasks list, otherwise it is just killed, w/o affecting job failure status.

Throws:

IOException

InterruptedException

getQueueAdmins

public AccessControlList getQueueAdmins(String arg0)
                                 throws IOException

Description copied from interface: ClientProtocol

Get the administrators of the given job-queue. This method is for hadoop internal use only.

Specified by:

getQueueAdmins in interface ClientProtocol

Returns:

Queue administrators ACL for the queue to which job is submitted to

Throws:

IOException

getJobTrackerStatus

public Cluster.JobTrackerStatus getJobTrackerStatus()
                                             throws IOException,
                                                    InterruptedException

Description copied from interface: ClientProtocol

Get the JobTracker's status.

Specified by:

getJobTrackerStatus in interface ClientProtocol

Returns:

Cluster.JobTrackerStatus of the JobTracker

Throws:

IOException

InterruptedException

getProtocolSignature

public ProtocolSignature getProtocolSignature(String protocol,
                                     long clientVersion,
                                     int clientMethodsHash)
                                       throws IOException

Description copied from interface: VersionedProtocol

Return protocol version corresponding to protocol interface.

Specified by:

getProtocolSignature in interface VersionedProtocol

Parameters:

protocol - The classname of the protocol interface

clientVersion - The version of the protocol that the client speaks

clientMethodsHash - the hashcode of client protocol methods

Returns:

the server protocol signature containing its version and a list of its supported methods

Throws:

IOException

See Also:

for a default implementation

getLogFileParams

public LogParams getLogFileParams(JobID jobID,
                         TaskAttemptID taskAttemptID)
                           throws IOException

Description copied from interface: ClientProtocol

Gets the location of the log file for a job if no taskAttemptId is specified, otherwise gets the log location for the taskAttemptId.

Specified by:

getLogFileParams in interface ClientProtocol

Parameters:

jobID - the jobId.

taskAttemptID - the taskAttemptId.

Returns:

log params.

Throws:

IOException