org.apache.hadoop.hive.ql.optimizer

Class GenMapRedUtils

java.lang.Object

org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils

public final class GenMapRedUtils
extends java.lang.Object

General utility common functions for the Processor to convert operator into map-reduce tasks.

Method Summary

Modifier and Type	Method and Description
static MapredWork	getMapRedWork(ParseContext parseCtx) create a new plan and return.
static MapredWork	getMapRedWorkFromConf(HiveConf conf) create a new plan and return.
static void	initPlan(ReduceSinkOperator op, GenMRProcContext opProcCtx) Initialize the current plan by adding it to root tasks.
static void	initUnionPlan(GenMRProcContext opProcCtx, UnionOperator currUnionOp, Task<? extends java.io.Serializable> currTask, boolean local)
static void	initUnionPlan(ReduceSinkOperator op, UnionOperator currUnionOp, GenMRProcContext opProcCtx, Task<? extends java.io.Serializable> unionTask) Initialize the current union plan.
static void	joinPlan(Task<? extends java.io.Serializable> currTask, Task<? extends java.io.Serializable> oldTask, GenMRProcContext opProcCtx) Merge the current task into the old task for the reducer
static void	joinUnionPlan(GenMRProcContext opProcCtx, UnionOperator currUnionOp, Task<? extends java.io.Serializable> currentUnionTask, Task<? extends java.io.Serializable> existingTask, boolean local)
static Operator<? extends OperatorDesc>	putOpInsertMap(Operator<? extends OperatorDesc> op, RowResolver rr, ParseContext parseCtx) insert in the map for the operator to row resolver.
static void	replaceMapWork(java.lang.String sourceAlias, java.lang.String targetAlias, MapWork source, MapWork target) Replace the Map-side operator tree associated with targetAlias in target with the Map-side operator tree associated with sourceAlias in source.
static void	setKeyAndValueDesc(ReduceWork plan, Operator<? extends OperatorDesc> topOp) set key and value descriptor.
static void	setKeyAndValueDescForTaskTree(Task<? extends java.io.Serializable> task) Set the key and value description for all the tasks rooted at the given task.
static void	setTaskPlan(java.lang.String alias_id, Operator<? extends OperatorDesc> topOp, Task<?> task, boolean local, GenMRProcContext opProcCtx) set the current task in the mapredWork.
static void	setTaskPlan(java.lang.String alias_id, Operator<? extends OperatorDesc> topOp, Task<?> task, boolean local, GenMRProcContext opProcCtx, PrunedPartitionList pList) set the current task in the mapredWork.
static void	setTaskPlan(java.lang.String path, java.lang.String alias, Operator<? extends OperatorDesc> topOp, MapWork plan, boolean local, TableDesc tt_desc) set the current task in the mapredWork.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

initPlan

public static void initPlan(ReduceSinkOperator op,
                            GenMRProcContext opProcCtx)
                     throws SemanticException

Initialize the current plan by adding it to root tasks.

Parameters:

op - the reduce sink operator encountered

opProcCtx - processing context

Throws:

SemanticException

initUnionPlan

public static void initUnionPlan(ReduceSinkOperator op,
                                 UnionOperator currUnionOp,
                                 GenMRProcContext opProcCtx,
                                 Task<? extends java.io.Serializable> unionTask)
                          throws SemanticException

Initialize the current union plan.

Parameters:

op - the reduce sink operator encountered

opProcCtx - processing context

Throws:

SemanticException

initUnionPlan

public static void initUnionPlan(GenMRProcContext opProcCtx,
                                 UnionOperator currUnionOp,
                                 Task<? extends java.io.Serializable> currTask,
                                 boolean local)
                          throws SemanticException

Throws:

SemanticException

joinUnionPlan

public static void joinUnionPlan(GenMRProcContext opProcCtx,
                                 UnionOperator currUnionOp,
                                 Task<? extends java.io.Serializable> currentUnionTask,
                                 Task<? extends java.io.Serializable> existingTask,
                                 boolean local)
                          throws SemanticException

Throws:

SemanticException

joinPlan

public static void joinPlan(Task<? extends java.io.Serializable> currTask,
                            Task<? extends java.io.Serializable> oldTask,
                            GenMRProcContext opProcCtx)
                     throws SemanticException

Merge the current task into the old task for the reducer

Parameters:

currTask - the current task for the current reducer

oldTask - the old task for the current reducer

opProcCtx - processing context

Throws:

SemanticException

setTaskPlan

public static void setTaskPlan(java.lang.String alias_id,
                               Operator<? extends OperatorDesc> topOp,
                               Task<?> task,
                               boolean local,
                               GenMRProcContext opProcCtx)
                        throws SemanticException

set the current task in the mapredWork.

Parameters:

alias_id - current alias

topOp - the top operator of the stack

plan - current plan

local - whether you need to add to map-reduce or local work

opProcCtx - processing context

Throws:

SemanticException

setTaskPlan

public static void setTaskPlan(java.lang.String alias_id,
                               Operator<? extends OperatorDesc> topOp,
                               Task<?> task,
                               boolean local,
                               GenMRProcContext opProcCtx,
                               PrunedPartitionList pList)
                        throws SemanticException

set the current task in the mapredWork.

Parameters:

alias_id - current alias

topOp - the top operator of the stack

plan - current plan

local - whether you need to add to map-reduce or local work

opProcCtx - processing context

pList - pruned partition list. If it is null it will be computed on-the-fly.

Throws:

SemanticException

setTaskPlan

public static void setTaskPlan(java.lang.String path,
                               java.lang.String alias,
                               Operator<? extends OperatorDesc> topOp,
                               MapWork plan,
                               boolean local,
                               TableDesc tt_desc)
                        throws SemanticException

set the current task in the mapredWork.

Parameters:

alias - current alias

topOp - the top operator of the stack

plan - current plan

local - whether you need to add to map-reduce or local work

tt_desc - table descriptor

Throws:

SemanticException

setKeyAndValueDesc

public static void setKeyAndValueDesc(ReduceWork plan,
                                      Operator<? extends OperatorDesc> topOp)

set key and value descriptor.

Parameters:

plan - current plan

topOp - current top operator in the path

setKeyAndValueDescForTaskTree

public static void setKeyAndValueDescForTaskTree(Task<? extends java.io.Serializable> task)

Set the key and value description for all the tasks rooted at the given task. Loops over all the tasks recursively.

Parameters:

task -

getMapRedWork

public static MapredWork getMapRedWork(ParseContext parseCtx)

create a new plan and return.

Returns:

the new plan

getMapRedWorkFromConf

public static MapredWork getMapRedWorkFromConf(HiveConf conf)

create a new plan and return. The pan won't contain the name to split sample information in parse context.

Returns:

the new plan

putOpInsertMap

public static Operator<? extends OperatorDesc> putOpInsertMap(Operator<? extends OperatorDesc> op,
                                                              RowResolver rr,
                                                              ParseContext parseCtx)

insert in the map for the operator to row resolver.

Parameters:

op - operator created

rr - row resolver

parseCtx - parse context

replaceMapWork

public static void replaceMapWork(java.lang.String sourceAlias,
                                  java.lang.String targetAlias,
                                  MapWork source,
                                  MapWork target)

Replace the Map-side operator tree associated with targetAlias in target with the Map-side operator tree associated with sourceAlias in source.

Parameters:

sourceAlias -

targetAlias -

source -

target -