org.apache.hadoop.mapred

Class TaskAttemptListenerImpl

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.service.CompositeService

org.apache.hadoop.mapred.TaskAttemptListenerImpl

All Implemented Interfaces:

Closeable, AutoCloseable, VersionedProtocol, TaskUmbilicalProtocol, TaskAttemptListener, Service

public class TaskAttemptListenerImpl
extends CompositeService
implements TaskUmbilicalProtocol, TaskAttemptListener

This class is responsible for talking to the task umblical. It also converts all the old data structures to yarn data structures. This class HAS to be in this package to access package private methods/classes.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

CompositeService.CompositeServiceShutdownHook

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected TaskHeartbeatHandler	taskHeartbeatHandler

Fields inherited from class org.apache.hadoop.service.CompositeService

STOP_ONLY_STARTED_SERVICES

Fields inherited from interface org.apache.hadoop.mapred.TaskUmbilicalProtocol

versionID

Constructor Summary

Constructors

Constructor and Description
TaskAttemptListenerImpl(AppContext context, JobTokenSecretManager jobTokenSecretManager, RMHeartbeatHandler rmHeartbeatHandler)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	canCommit(TaskAttemptID taskAttemptID) Child checking whether it can commit.
void	commitPending(TaskAttemptID taskAttemptID, TaskStatus taskStatsu) TaskAttempt is reporting that it is in commit_pending and it is waiting for the commit Response Commit it a two-phased protocol.
void	done(TaskAttemptID taskAttemptID) Report that the task is successfully completed.
void	fatalError(TaskAttemptID taskAttemptID, String msg) Report that the task encounted a fatal error.
void	fsError(TaskAttemptID taskAttemptID, String message) Report that the task encounted a local filesystem error.
InetSocketAddress	getAddress()
MapTaskCompletionEventsUpdate	getMapCompletionEvents(JobID jobIdentifier, int startIndex, int maxEvents, TaskAttemptID taskAttemptID) Called by a reduce task to get the map output locations for finished maps.
ProtocolSignature	getProtocolSignature(String protocol, long clientVersion, int clientMethodsHash) Return protocol version corresponding to protocol interface.
long	getProtocolVersion(String arg0, long arg1) Return protocol version corresponding to protocol interface.
JvmTask	getTask(org.apache.hadoop.mapred.JvmContext context) Called when a child task process starts, to get its task.
boolean	ping(TaskAttemptID taskAttemptID) Periodically called by child to check if parent is still alive.
protected void	registerHeartbeatHandler(Configuration conf)
void	registerLaunchedTask(TaskAttemptId attemptID, WrappedJvmID jvmId) Register task attempt.
void	registerPendingTask(Task task, WrappedJvmID jvmID) Register a JVM with the listener.
void	reportDiagnosticInfo(TaskAttemptID taskAttemptID, String diagnosticInfo) Report error messages back to parent.
void	reportNextRecordRange(TaskAttemptID taskAttemptID, org.apache.hadoop.mapred.SortedRanges.Range range) Report the record range which is going to process next by the Task.
protected void	serviceInit(Configuration conf) All initialization code needed by a service.
protected void	serviceStart() Actions called during the INITED to STARTED transition.
protected void	serviceStop() Actions called during the transition to the STOPPED state.
void	shuffleError(TaskAttemptID taskAttemptID, String message) Report that a reduce-task couldn't shuffle map-outputs.
protected void	startRpcServer()
boolean	statusUpdate(TaskAttemptID taskAttemptID, TaskStatus taskStatus) Report child's progress to parent.
protected void	stopRpcServer()
void	unregister(TaskAttemptId attemptID, WrappedJvmID jvmID) Unregister the JVM and the attempt associated with it.

Methods inherited from class org.apache.hadoop.service.CompositeService

addIfService, addService, getServices, removeService

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

taskHeartbeatHandler

protected TaskHeartbeatHandler taskHeartbeatHandler

Constructor Detail

TaskAttemptListenerImpl

public TaskAttemptListenerImpl(AppContext context,
                       JobTokenSecretManager jobTokenSecretManager,
                       RMHeartbeatHandler rmHeartbeatHandler)

Method Detail

serviceInit

protected void serviceInit(Configuration conf)
                    throws Exception

Description copied from class: AbstractService

All initialization code needed by a service. This method will only ever be called once during the lifecycle of a specific service instance. Implementations do not need to be synchronized as the logic in AbstractService.init(Configuration) prevents re-entrancy. The base implementation checks to see if the subclass has created a new configuration instance, and if so, updates the base class value

Overrides:

serviceInit in class CompositeService

Parameters:

conf - configuration

Throws:

Exception - on a failure -these will be caught, possibly wrapped, and wil; trigger a service stop

serviceStart

protected void serviceStart()
                     throws Exception

Description copied from class: AbstractService

Actions called during the INITED to STARTED transition. This method will only ever be called once during the lifecycle of a specific service instance. Implementations do not need to be synchronized as the logic in AbstractService.start() prevents re-entrancy.

Overrides:

serviceStart in class CompositeService

Throws:

Exception - if needed -these will be caught, wrapped, and trigger a service stop

registerHeartbeatHandler

protected void registerHeartbeatHandler(Configuration conf)

startRpcServer

protected void startRpcServer()

serviceStop

protected void serviceStop()
                    throws Exception

Description copied from class: AbstractService

Actions called during the transition to the STOPPED state. This method will only ever be called once during the lifecycle of a specific service instance. Implementations do not need to be synchronized as the logic in AbstractService.stop() prevents re-entrancy. Implementations MUST write this to be robust against failures, including checks for null references -and for the first failure to not stop other attempts to shut down parts of the service.

Overrides:

serviceStop in class CompositeService

Throws:

Exception - if needed -these will be caught and logged.

stopRpcServer

protected void stopRpcServer()

getAddress

public InetSocketAddress getAddress()

Specified by:

getAddress in interface TaskAttemptListener

canCommit

public boolean canCommit(TaskAttemptID taskAttemptID)
                  throws IOException

Child checking whether it can commit.
Commit is a two-phased protocol. First the attempt informs the ApplicationMaster that it is commitPending(TaskAttemptID, TaskStatus). Then it repeatedly polls the ApplicationMaster whether it canCommit(TaskAttemptID) This is a legacy from the centralized commit protocol handling by the JobTracker.

Specified by:

canCommit in interface TaskUmbilicalProtocol

Returns:

true/false

Throws:

IOException

commitPending

public void commitPending(TaskAttemptID taskAttemptID,
                 TaskStatus taskStatsu)
                   throws IOException,
                          InterruptedException

TaskAttempt is reporting that it is in commit_pending and it is waiting for the commit Response
Commit it a two-phased protocol. First the attempt informs the ApplicationMaster that it is commitPending(TaskAttemptID, TaskStatus). Then it repeatedly polls the ApplicationMaster whether it canCommit(TaskAttemptID) This is a legacy from the centralized commit protocol handling by the JobTracker.

Specified by:

commitPending in interface TaskUmbilicalProtocol

Parameters:

taskAttemptID - task's id

taskStatsu - status of the child

Throws:

IOException

InterruptedException

done

public void done(TaskAttemptID taskAttemptID)
          throws IOException

Description copied from interface: TaskUmbilicalProtocol

Report that the task is successfully completed. Failure is assumed if the task process exits without calling this.

Specified by:

done in interface TaskUmbilicalProtocol

Parameters:

taskAttemptID - task's id

Throws:

IOException

fatalError

public void fatalError(TaskAttemptID taskAttemptID,
              String msg)
                throws IOException

Description copied from interface: TaskUmbilicalProtocol

Report that the task encounted a fatal error.

Specified by:

fatalError in interface TaskUmbilicalProtocol

Throws:

IOException

fsError

public void fsError(TaskAttemptID taskAttemptID,
           String message)
             throws IOException

Description copied from interface: TaskUmbilicalProtocol

Report that the task encounted a local filesystem error.

Specified by:

fsError in interface TaskUmbilicalProtocol

Throws:

IOException

shuffleError

public void shuffleError(TaskAttemptID taskAttemptID,
                String message)
                  throws IOException

Description copied from interface: TaskUmbilicalProtocol

Report that a reduce-task couldn't shuffle map-outputs.

Specified by:

shuffleError in interface TaskUmbilicalProtocol

Throws:

IOException

getMapCompletionEvents

public MapTaskCompletionEventsUpdate getMapCompletionEvents(JobID jobIdentifier,
                                                   int startIndex,
                                                   int maxEvents,
                                                   TaskAttemptID taskAttemptID)
                                                     throws IOException

Description copied from interface: TaskUmbilicalProtocol

Called by a reduce task to get the map output locations for finished maps. Returns an update centered around the map-task-completion-events. The update also piggybacks the information whether the events copy at the task-tracker has changed or not. This will trigger some action at the child-process.

Specified by:

getMapCompletionEvents in interface TaskUmbilicalProtocol

startIndex - the index starting from which the locations should be fetched

maxEvents - the max number of locations to fetch

taskAttemptID - The attempt id of the task that is trying to communicate

Returns:

A MapTaskCompletionEventsUpdate

Throws:

IOException

ping

public boolean ping(TaskAttemptID taskAttemptID)
             throws IOException

Description copied from interface: TaskUmbilicalProtocol

Periodically called by child to check if parent is still alive.

Specified by:

ping in interface TaskUmbilicalProtocol

Returns:

True if the task is known

Throws:

IOException

reportDiagnosticInfo

public void reportDiagnosticInfo(TaskAttemptID taskAttemptID,
                        String diagnosticInfo)
                          throws IOException

Description copied from interface: TaskUmbilicalProtocol

Report error messages back to parent. Calls should be sparing, since all such messages are held in the job tracker.

Specified by:

reportDiagnosticInfo in interface TaskUmbilicalProtocol

Parameters:

taskAttemptID - the id of the task involved

diagnosticInfo - the text to report

Throws:

IOException

statusUpdate

public boolean statusUpdate(TaskAttemptID taskAttemptID,
                   TaskStatus taskStatus)
                     throws IOException,
                            InterruptedException

Description copied from interface: TaskUmbilicalProtocol

Report child's progress to parent.

Specified by:

statusUpdate in interface TaskUmbilicalProtocol

Parameters:

taskAttemptID - task-id of the child

taskStatus - status of the child

Returns:

True if the task is known

Throws:

IOException

InterruptedException

getProtocolVersion

public long getProtocolVersion(String arg0,
                      long arg1)
                        throws IOException

Description copied from interface: VersionedProtocol

Return protocol version corresponding to protocol interface.

Specified by:

getProtocolVersion in interface VersionedProtocol

Parameters:

arg0 - The classname of the protocol interface

arg1 - The version of the protocol that the client speaks

Returns:

the version that the server will speak

Throws:

IOException - if any IO error occurs

reportNextRecordRange

public void reportNextRecordRange(TaskAttemptID taskAttemptID,
                         org.apache.hadoop.mapred.SortedRanges.Range range)
                           throws IOException

Description copied from interface: TaskUmbilicalProtocol

Report the record range which is going to process next by the Task.

Specified by:

reportNextRecordRange in interface TaskUmbilicalProtocol

Parameters:

taskAttemptID - the id of the task involved

range - the range of record sequence nos

Throws:

IOException

getTask

public JvmTask getTask(org.apache.hadoop.mapred.JvmContext context)
                throws IOException

Description copied from interface: TaskUmbilicalProtocol

Called when a child task process starts, to get its task.

Specified by:

getTask in interface TaskUmbilicalProtocol

Parameters:

context - the JvmContext of the JVM w.r.t the TaskTracker that launched it

Returns:

Task object

Throws:

IOException

registerPendingTask

public void registerPendingTask(Task task,
                       WrappedJvmID jvmID)

Description copied from interface: TaskAttemptListener

Register a JVM with the listener. This should be called as soon as a JVM ID is assigned to a task attempt, before it has been launched.

Specified by:

registerPendingTask in interface TaskAttemptListener

Parameters:

task - the task itself for this JVM.

jvmID - The ID of the JVM .

registerLaunchedTask

public void registerLaunchedTask(TaskAttemptId attemptID,
                        WrappedJvmID jvmId)

Description copied from interface: TaskAttemptListener

Register task attempt. This should be called when the JVM has been launched.

Specified by:

registerLaunchedTask in interface TaskAttemptListener

Parameters:

attemptID - the id of the attempt for this JVM.

jvmId - the ID of the JVM.

unregister

public void unregister(TaskAttemptId attemptID,
              WrappedJvmID jvmID)

Description copied from interface: TaskAttemptListener

Unregister the JVM and the attempt associated with it. This should be called when the attempt/JVM has finished executing and is being cleaned up.

Specified by:

unregister in interface TaskAttemptListener

Parameters:

attemptID - the ID of the attempt.

jvmID - the ID of the JVM for that attempt.

getProtocolSignature

public ProtocolSignature getProtocolSignature(String protocol,
                                     long clientVersion,
                                     int clientMethodsHash)
                                       throws IOException

Description copied from interface: VersionedProtocol

Return protocol version corresponding to protocol interface.

Specified by:

getProtocolSignature in interface VersionedProtocol

Parameters:

protocol - The classname of the protocol interface

clientVersion - The version of the protocol that the client speaks

clientMethodsHash - the hashcode of client protocol methods

Returns:

the server protocol signature containing its version and a list of its supported methods

Throws:

IOException

See Also:

for a default implementation