org.apache.hadoop.hive.ql.exec.mr

Class ExecDriver

java.lang.Object

org.apache.hadoop.hive.ql.exec.Task<MapredWork>

org.apache.hadoop.hive.ql.exec.mr.ExecDriver

All Implemented Interfaces:

java.io.Serializable, HadoopJobExecHook, Node

Direct Known Subclasses:

MapRedTask

public class ExecDriver
extends Task<MapredWork>
implements java.io.Serializable, HadoopJobExecHook

ExecDriver is the central class in co-ordinating execution of any map-reduce task. It's main responsabilities are: - Converting the plan (MapredWork) into a MR Job (JobConf) - Submitting a MR job to the cluster via JobClient and ExecHelper - Executing MR job in local execution mode (where applicable)

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Task

Task.FeedType

Field Summary

Fields

Modifier and Type	Field and Description
protected JobConf	job
protected HadoopJobExecHelper	jobExecHelper
protected static org.apache.commons.logging.Log	LOG
static java.lang.management.MemoryMXBean	memoryMXBean

Fields inherited from class org.apache.hadoop.hive.ql.exec.Task

BACKUP_COMMON_JOIN, backupChildrenTasks, backupTask, childTasks, clonedConf, COMMON_JOIN, conf, console, CONVERTED_LOCAL_MAPJOIN, CONVERTED_MAPJOIN, CONVERTED_SORTMERGEJOIN, db, driverContext, feedSubscribers, id, initialized, isdone, jobID, LOCAL_MAPJOIN, MAPJOIN_ONLY_NOBACKUP, NO_TAG, parentTasks, queryPlan, queued, started, taskCounters, taskHandle, taskTag, work

Constructor Summary

Constructors

Constructor and Description
ExecDriver() Constructor when invoked from QL.
ExecDriver(MapredWork plan, JobConf job, boolean isSilent) Constructor/Initialization for invocation as independent utility.

Method Summary

Modifier and Type	Method and Description
boolean	checkFatalErrors(Counters ctrs, java.lang.StringBuilder errMsg) Fatal errors are those errors that cannot be recovered by retries.
int	execute(DriverContext driverContext) Execute a query plan using Hadoop.
static java.lang.String	generateCmdLine(HiveConf hconf, Context ctx) Given a Hive Configuration object - generate a command line fragment for passing such configuration information to ExecDriver.
java.lang.String	getName() Gets the name of the node.
java.util.Collection<Operator<? extends OperatorDesc>>	getTopOperators()
StageType	getType() Should be overridden to return the type of the specific task among the types in StageType.
boolean	hasReduce()
void	initialize(HiveConf conf, QueryPlan queryPlan, DriverContext driverContext) Initialization when invoked from QL.
boolean	isMapRedTask()
void	logPlanProgress(SessionState ss)
static void	main(java.lang.String[] args)
boolean	mapDone()
boolean	mapStarted()
boolean	reduceDone()
boolean	reduceStarted()
boolean	requireLock()
protected void	setInputAttributes(Configuration conf) Set hive input format, and input format file if necessary.
void	shutdown()
void	updateCounters(Counters ctrs, RunningJob rj)

Methods inherited from class org.apache.hadoop.hive.ql.exec.Task

addDependentTask, cloneConf, done, executeTask, fetch, getAndInitBackupTask, getBackupChildrenTasks, getBackupTask, getChildren, getChildTasks, getCounters, getDependentTasks, getFeedSubscribers, getId, getInitialized, getJobID, getParentTasks, getQueryPlan, getQueued, getReducer, getResultSchema, getTaskHandle, getTaskTag, getWork, ifRetryCmdWhenFail, isLocalMode, isMapRedLocalTask, isRunnable, pushFeed, receiveFeed, removeDependentTask, removeFromChildrenTasks, setBackupChildrenTasks, setBackupTask, setChildTasks, setDone, setFeedSubscribers, setId, setInitialized, setLocalMode, setParentTasks, setQueryPlan, setQueued, setRetryCmdWhenFail, setStarted, setTaskTag, setWork, started, subscribeFeed

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

job

protected transient JobConf job

memoryMXBean

public static java.lang.management.MemoryMXBean memoryMXBean

jobExecHelper

protected HadoopJobExecHelper jobExecHelper

LOG

protected static final transient org.apache.commons.logging.Log LOG

Constructor Detail

ExecDriver

public ExecDriver()

Constructor when invoked from QL.

ExecDriver

public ExecDriver(MapredWork plan,
                  JobConf job,
                  boolean isSilent)
           throws HiveException

Constructor/Initialization for invocation as independent utility.

Throws:

HiveException

Method Detail

requireLock

public boolean requireLock()

Overrides:

requireLock in class Task<MapredWork>

initialize

public void initialize(HiveConf conf,
                       QueryPlan queryPlan,
                       DriverContext driverContext)

Initialization when invoked from QL.

Overrides:

initialize in class Task<MapredWork>

checkFatalErrors

public boolean checkFatalErrors(Counters ctrs,
                                java.lang.StringBuilder errMsg)

Fatal errors are those errors that cannot be recovered by retries. These are application dependent. Examples of fatal errors include: - the small table in the map-side joins is too large to be feasible to be handled by one mapper. The job should fail and the user should be warned to use regular joins rather than map-side joins. Fatal errors are indicated by counters that are set at execution time. If the counter is non-zero, a fatal error occurred. The value of the counter indicates the error type.

Specified by:

checkFatalErrors in interface HadoopJobExecHook

Returns:

true if fatal errors happened during job execution, false otherwise.

execute

public int execute(DriverContext driverContext)

Execute a query plan using Hadoop.

Specified by:

execute in class Task<MapredWork>

Returns:

status of executing the task

setInputAttributes

protected void setInputAttributes(Configuration conf)

Set hive input format, and input format file if necessary.

mapStarted

public boolean mapStarted()

reduceStarted

public boolean reduceStarted()

mapDone

public boolean mapDone()

reduceDone

public boolean reduceDone()

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException,
                        HiveException

Throws:

java.io.IOException

HiveException

generateCmdLine

public static java.lang.String generateCmdLine(HiveConf hconf,
                                               Context ctx)
                                        throws java.io.IOException

Given a Hive Configuration object - generate a command line fragment for passing such configuration information to ExecDriver.

Throws:

java.io.IOException

isMapRedTask

public boolean isMapRedTask()

Overrides:

isMapRedTask in class Task<MapredWork>

getTopOperators

public java.util.Collection<Operator<? extends OperatorDesc>> getTopOperators()

Overrides:

getTopOperators in class Task<MapredWork>

hasReduce

public boolean hasReduce()

Overrides:

hasReduce in class Task<MapredWork>

getType

public StageType getType()

Description copied from class: Task

Should be overridden to return the type of the specific task among the types in StageType.

Specified by:

getType in class Task<MapredWork>

Returns:

StageType.* or null if not overridden

getName

public java.lang.String getName()

Description copied from interface: Node

Gets the name of the node. This is used in the rule dispatchers.

Specified by:

getName in interface Node

Returns:

String

updateCounters

public void updateCounters(Counters ctrs,
                           RunningJob rj)
                    throws java.io.IOException

Specified by:

updateCounters in interface HadoopJobExecHook

Throws:

java.io.IOException

logPlanProgress

public void logPlanProgress(SessionState ss)
                     throws java.io.IOException

Specified by:

logPlanProgress in interface HadoopJobExecHook

Throws:

java.io.IOException

shutdown

public void shutdown()

Overrides:

shutdown in class Task<MapredWork>