org.apache.hadoop.mapred

Class LocalJobRunner

java.lang.Object

org.apache.hadoop.mapred.LocalJobRunner

All Implemented Interfaces:

VersionedProtocol, ClientProtocol

@InterfaceAudience.Private
@InterfaceStability.Unstable
public class LocalJobRunner
extends Object
implements ClientProtocol

Implements MapReduce locally, in-process, for debugging.

Field Summary

Fields

Modifier and Type	Field and Description
static String	LOCAL_MAX_MAPS The maximum number of map tasks to run in parallel in LocalJobRunner
static org.apache.commons.logging.Log	LOG

Fields inherited from interface org.apache.hadoop.mapreduce.protocol.ClientProtocol

versionID

Constructor Summary

Constructors

Constructor and Description
LocalJobRunner(Configuration conf)
LocalJobRunner(JobConf conf) Deprecated.

Method Summary

Methods

Modifier and Type	Method and Description
void	cancelDelegationToken(Token<DelegationTokenIdentifier> token) Cancel a delegation token.
TaskTrackerInfo[]	getActiveTrackers() Get all active trackers in cluster.
JobStatus[]	getAllJobs() Get all the jobs submitted.
TaskTrackerInfo[]	getBlacklistedTrackers() Get all blacklisted trackers in cluster.
QueueInfo[]	getChildQueues(String queueName) Returns immediate children of queueName.
ClusterMetrics	getClusterMetrics() Get the current status of the cluster
Token<DelegationTokenIdentifier>	getDelegationToken(Text renewer) Get a new delegation token.
String	getFilesystemName() A MapReduce system always operates on a single filesystem.
Counters	getJobCounters(JobID id) Grab the current job counters
String	getJobHistoryDir() Gets the directory location of the completed job history files.
JobStatus	getJobStatus(JobID id) Grab a handle to a job that is already known to the JobTracker.
Cluster.JobTrackerStatus	getJobTrackerStatus() Get the JobTracker's status.
static int	getLocalMaxRunningMaps(JobContext job)
LogParams	getLogFileParams(JobID jobID, TaskAttemptID taskAttemptID) Gets the location of the log file for a job if no taskAttemptId is specified, otherwise gets the log location for the taskAttemptId.
JobID	getNewJobID() Allocate a name for the job.
ProtocolSignature	getProtocolSignature(String protocol, long clientVersion, int clientMethodsHash) Return protocol version corresponding to protocol interface.
long	getProtocolVersion(String protocol, long clientVersion) Return protocol version corresponding to protocol interface.
QueueInfo	getQueue(String queue) Gets scheduling information associated with the particular Job queue
QueueAclsInfo[]	getQueueAclsForCurrentUser() Gets the Queue ACLs for current user
AccessControlList	getQueueAdmins(String queueName) Get the administrators of the given job-queue.
QueueInfo[]	getQueues() Gets set of Queues associated with the Job Tracker
QueueInfo[]	getRootQueues() Gets the root level queues.
String	getStagingAreaDir() Get a hint from the JobTracker where job-specific files are to be placed.
String	getSystemDir() Grab the jobtracker system directory path where job-specific files are to be placed.
TaskCompletionEvent[]	getTaskCompletionEvents(JobID jobid, int fromEventId, int maxEvents) Get task completion events for the jobid, starting from fromEventId.
String[]	getTaskDiagnostics(TaskAttemptID taskid) Returns the diagnostic information for a particular task in the given job.
TaskReport[]	getTaskReports(JobID id, TaskType type) Grab a bunch of info on the tasks that make up the job
long	getTaskTrackerExpiryInterval()
void	killJob(JobID id) Kill the indicated job
boolean	killTask(TaskAttemptID taskId, boolean shouldFail) Throws UnsupportedOperationException
long	renewDelegationToken(Token<DelegationTokenIdentifier> token) Renew an existing delegation token
void	setJobPriority(JobID id, String jp) Set the priority of the specified job
static void	setLocalMaxRunningMaps(JobContext job, int maxMaps) Set the max number of map tasks to run concurrently in the LocalJobRunner.
JobStatus	submitJob(JobID jobid, String jobSubmitDir, Credentials credentials) Submit a Job for execution.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

LOCAL_MAX_MAPS

public static final String LOCAL_MAX_MAPS

The maximum number of map tasks to run in parallel in LocalJobRunner

See Also:

Constant Field Values

Constructor Detail

LocalJobRunner

public LocalJobRunner(Configuration conf)
               throws IOException

Throws:

IOException

LocalJobRunner

@Deprecated
public LocalJobRunner(JobConf conf)
               throws IOException

Deprecated.

Throws:

IOException

Method Detail

getProtocolVersion

public long getProtocolVersion(String protocol,
                      long clientVersion)

Description copied from interface: VersionedProtocol

Return protocol version corresponding to protocol interface.

Specified by:

getProtocolVersion in interface VersionedProtocol

Parameters:

protocol - The classname of the protocol interface

clientVersion - The version of the protocol that the client speaks

Returns:

the version that the server will speak

getProtocolSignature

public ProtocolSignature getProtocolSignature(String protocol,
                                     long clientVersion,
                                     int clientMethodsHash)
                                       throws IOException

Description copied from interface: VersionedProtocol

Return protocol version corresponding to protocol interface.

Specified by:

getProtocolSignature in interface VersionedProtocol

Parameters:

protocol - The classname of the protocol interface

clientVersion - The version of the protocol that the client speaks

clientMethodsHash - the hashcode of client protocol methods

Returns:

the server protocol signature containing its version and a list of its supported methods

Throws:

IOException

See Also:

for a default implementation

getNewJobID

public JobID getNewJobID()

Description copied from interface: ClientProtocol

Allocate a name for the job.

Specified by:

getNewJobID in interface ClientProtocol

Returns:

a unique job name for submitting jobs.

submitJob

public JobStatus submitJob(JobID jobid,
                  String jobSubmitDir,
                  Credentials credentials)
                    throws IOException

Description copied from interface: ClientProtocol

Submit a Job for execution. Returns the latest profile for that job.

Specified by:

submitJob in interface ClientProtocol

Throws:

IOException

killJob

public void killJob(JobID id)

Description copied from interface: ClientProtocol

Kill the indicated job

Specified by:

killJob in interface ClientProtocol

setJobPriority

public void setJobPriority(JobID id,
                  String jp)
                    throws IOException

Description copied from interface: ClientProtocol

Set the priority of the specified job

Specified by:

setJobPriority in interface ClientProtocol

Parameters:

id - ID of the job

jp - Priority to be set for the job

Throws:

IOException

killTask

public boolean killTask(TaskAttemptID taskId,
               boolean shouldFail)
                 throws IOException

Throws UnsupportedOperationException

Specified by:

killTask in interface ClientProtocol

Parameters:

taskId - the id of the task to kill.

shouldFail - if true the task is failed and added to failed tasks list, otherwise it is just killed, w/o affecting job failure status.

Throws:

IOException

getTaskReports

public TaskReport[] getTaskReports(JobID id,
                          TaskType type)

Description copied from interface: ClientProtocol

Grab a bunch of info on the tasks that make up the job

Specified by:

getTaskReports in interface ClientProtocol

getJobStatus

public JobStatus getJobStatus(JobID id)

Description copied from interface: ClientProtocol

Grab a handle to a job that is already known to the JobTracker.

Specified by:

getJobStatus in interface ClientProtocol

Returns:

Status of the job, or null if not found.

getJobCounters

public Counters getJobCounters(JobID id)

Description copied from interface: ClientProtocol

Grab the current job counters

Specified by:

getJobCounters in interface ClientProtocol

getFilesystemName

public String getFilesystemName()
                         throws IOException

Description copied from interface: ClientProtocol

A MapReduce system always operates on a single filesystem. This function returns the fs name. ('local' if the localfs; 'addr:port' if dfs). The client can then copy files into the right locations prior to submitting the job.

Specified by:

getFilesystemName in interface ClientProtocol

Throws:

IOException

getClusterMetrics

public ClusterMetrics getClusterMetrics()

Description copied from interface: ClientProtocol

Get the current status of the cluster

Specified by:

getClusterMetrics in interface ClientProtocol

Returns:

summary of the state of the cluster

getJobTrackerStatus

public Cluster.JobTrackerStatus getJobTrackerStatus()

Description copied from interface: ClientProtocol

Get the JobTracker's status.

Specified by:

getJobTrackerStatus in interface ClientProtocol

Returns:

Cluster.JobTrackerStatus of the JobTracker

getTaskTrackerExpiryInterval

public long getTaskTrackerExpiryInterval()
                                  throws IOException,
                                         InterruptedException

Specified by:

getTaskTrackerExpiryInterval in interface ClientProtocol

Throws:

IOException

InterruptedException

getActiveTrackers

public TaskTrackerInfo[] getActiveTrackers()
                                    throws IOException,
                                           InterruptedException

Get all active trackers in cluster.

Specified by:

getActiveTrackers in interface ClientProtocol

Returns:

array of TaskTrackerInfo

Throws:

IOException

InterruptedException

getBlacklistedTrackers

public TaskTrackerInfo[] getBlacklistedTrackers()
                                         throws IOException,
                                                InterruptedException

Get all blacklisted trackers in cluster.

Specified by:

getBlacklistedTrackers in interface ClientProtocol

Returns:

array of TaskTrackerInfo

Throws:

IOException

InterruptedException

getTaskCompletionEvents

public TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid,
                                            int fromEventId,
                                            int maxEvents)
                                              throws IOException

Description copied from interface: ClientProtocol

Get task completion events for the jobid, starting from fromEventId. Returns empty array if no events are available.

Specified by:

getTaskCompletionEvents in interface ClientProtocol

Parameters:

jobid - job id

fromEventId - event id to start from.

maxEvents - the max number of events we want to look at

Returns:

array of task completion events.

Throws:

IOException

getAllJobs

public JobStatus[] getAllJobs()

Description copied from interface: ClientProtocol

Get all the jobs submitted.

Specified by:

getAllJobs in interface ClientProtocol

Returns:

array of JobStatus for the submitted jobs

getTaskDiagnostics

public String[] getTaskDiagnostics(TaskAttemptID taskid)
                            throws IOException

Returns the diagnostic information for a particular task in the given job. To be implemented

Specified by:

getTaskDiagnostics in interface ClientProtocol

Parameters:

taskid - the id of the task

Returns:

an array of the diagnostic messages

Throws:

IOException

getSystemDir

public String getSystemDir()

Description copied from interface: ClientProtocol

Grab the jobtracker system directory path where job-specific files are to be placed.

Specified by:

getSystemDir in interface ClientProtocol

Returns:

the system directory where job-specific files are to be placed.

See Also:

ClientProtocol.getSystemDir()

getQueueAdmins

public AccessControlList getQueueAdmins(String queueName)
                                 throws IOException

Description copied from interface: ClientProtocol

Get the administrators of the given job-queue. This method is for hadoop internal use only.

Specified by:

getQueueAdmins in interface ClientProtocol

Returns:

Queue administrators ACL for the queue to which job is submitted to

Throws:

IOException

See Also:

ClientProtocol.getQueueAdmins(String)

getStagingAreaDir

public String getStagingAreaDir()
                         throws IOException

Description copied from interface: ClientProtocol

Get a hint from the JobTracker where job-specific files are to be placed.

Specified by:

getStagingAreaDir in interface ClientProtocol

Returns:

the directory where job-specific files are to be placed.

Throws:

IOException

See Also:

ClientProtocol.getStagingAreaDir()

getJobHistoryDir

public String getJobHistoryDir()

Description copied from interface: ClientProtocol

Gets the directory location of the completed job history files.

Specified by:

getJobHistoryDir in interface ClientProtocol

getChildQueues

public QueueInfo[] getChildQueues(String queueName)
                           throws IOException

Description copied from interface: ClientProtocol

Returns immediate children of queueName.

Specified by:

getChildQueues in interface ClientProtocol

Returns:

array of JobQueueInfo which are children of queueName

Throws:

IOException

getRootQueues

public QueueInfo[] getRootQueues()
                          throws IOException

Description copied from interface: ClientProtocol

Gets the root level queues.

Specified by:

getRootQueues in interface ClientProtocol

Returns:

array of JobQueueInfo object.

Throws:

IOException

getQueues

public QueueInfo[] getQueues()
                      throws IOException

Description copied from interface: ClientProtocol

Gets set of Queues associated with the Job Tracker

Specified by:

getQueues in interface ClientProtocol

Returns:

Array of the Queue Information Object

Throws:

IOException

getQueue

public QueueInfo getQueue(String queue)
                   throws IOException

Description copied from interface: ClientProtocol

Gets scheduling information associated with the particular Job queue

Specified by:

getQueue in interface ClientProtocol

Parameters:

queue - Queue Name

Returns:

Scheduling Information of the Queue

Throws:

IOException

getQueueAclsForCurrentUser

public QueueAclsInfo[] getQueueAclsForCurrentUser()
                                           throws IOException

Description copied from interface: ClientProtocol

Gets the Queue ACLs for current user

Specified by:

getQueueAclsForCurrentUser in interface ClientProtocol

Returns:

array of QueueAclsInfo object for current user.

Throws:

IOException

setLocalMaxRunningMaps

public static void setLocalMaxRunningMaps(JobContext job,
                          int maxMaps)

Set the max number of map tasks to run concurrently in the LocalJobRunner.

Parameters:

job - the job to configure

maxMaps - the maximum number of map tasks to allow.

getLocalMaxRunningMaps

public static int getLocalMaxRunningMaps(JobContext job)

Returns:

the max number of map tasks to run concurrently in the LocalJobRunner.

cancelDelegationToken

public void cancelDelegationToken(Token<DelegationTokenIdentifier> token)
                           throws IOException,
                                  InterruptedException

Description copied from interface: ClientProtocol

Cancel a delegation token.

Specified by:

cancelDelegationToken in interface ClientProtocol

Parameters:

token - the token to cancel

Throws:

IOException

InterruptedException

getDelegationToken

public Token<DelegationTokenIdentifier> getDelegationToken(Text renewer)
                                                    throws IOException,
                                                           InterruptedException

Description copied from interface: ClientProtocol

Get a new delegation token.

Specified by:

getDelegationToken in interface ClientProtocol

Parameters:

renewer - the user other than the creator (if any) that can renew the token

Returns:

the new delegation token

Throws:

IOException

InterruptedException

renewDelegationToken

public long renewDelegationToken(Token<DelegationTokenIdentifier> token)
                          throws IOException,
                                 InterruptedException

Description copied from interface: ClientProtocol

Renew an existing delegation token

Specified by:

renewDelegationToken in interface ClientProtocol

Parameters:

token - the token to renew

Returns:

the new expiration time

Throws:

IOException

InterruptedException

getLogFileParams

public LogParams getLogFileParams(JobID jobID,
                         TaskAttemptID taskAttemptID)
                           throws IOException,
                                  InterruptedException

Description copied from interface: ClientProtocol

Gets the location of the log file for a job if no taskAttemptId is specified, otherwise gets the log location for the taskAttemptId.

Specified by:

getLogFileParams in interface ClientProtocol

Parameters:

jobID - the jobId.

taskAttemptID - the taskAttemptId.

Returns:

log params.

Throws:

IOException

InterruptedException