org.apache.hadoop.hive.ql.exec

Class PTFOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<PTFDesc>

org.apache.hadoop.hive.ql.exec.PTFOperator

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable, Node

public class PTFOperator
extends Operator<PTFDesc>
implements java.io.Serializable

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.OperatorFunc, Operator.ProgressCounter, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected KeyWrapper	currentKeys
protected KeyWrapper	newKeys

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

alias, beginTime, childOperators, childOperatorsArray, childOperatorsTag, colExprMap, conf, counterNames, counterNameToEnum, counters, done, fatalErrorCntr, groupKeyObject, id, inputObjInspectors, inputRows, isLogInfoEnabled, LOG, numInputRowsCntr, numOutputRowsCntr, operatorId, out, outputObjInspector, outputRows, parentOperators, reporter, state, statsMap, timeTakenCntr, totalTime

Constructor Summary

Constructors

Constructor and Description
PTFOperator()

Method Summary

Modifier and Type	Method and Description
protected void	closeOp(boolean abort) Operator specific close routine.
static void	connectLeadLagFunctionsToPartition(PTFDesc ptfDesc, PTFPartition.PTFPartitionIterator<java.lang.Object> pItr)
PTFPartition	createFirstPartitionForChain(ObjectInspector oi, HiveConf hiveConf, boolean isMapSide) Create a new Partition.
java.lang.String	getName() Implements the getName function for the Node Interface.
static java.lang.String	getOperatorName()
OperatorType	getType() Return the type of the specific operator among the types in OperatorType.
protected void	initializeOp(Configuration jobConf) Operator specific initialization.
protected void	processInputPartition()
protected void	processMapFunction()
void	processOp(java.lang.Object row, int tag) Process the row.
protected void	reconstructQueryDef(HiveConf hiveConf) Initialize the visitor to use the QueryDefDeserializer Use the order defined in QueryDefWalker to visit the QueryDef
protected void	setupKeysWrapper(ObjectInspector inputOI)

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, assignCounterNameToEnum, augmentPlan, checkFatalErrors, cleanUpInputFileChanged, cleanUpInputFileChangedOp, clone, close, columnNamesRowResolvedCanBeObtained, dump, dump, endGroup, fatalErrorMessage, flush, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getConf, getConfiguration, getCounterNames, getCounterNameToEnum, getCounters, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getNextCntr, getNumChild, getNumParent, getOperatorId, getParentOperators, getSchema, getStats, getWrappedCounterName, incrCounter, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeCounters, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logStats, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, process, processGroup, removeChild, removeChildAndAdoptItsChildren, removeChildren, removeParent, replaceChild, replaceParent, reset, resetId, resetLastEnumUsed, resetStats, setAlias, setChildOperators, setColumnExprMap, setConf, setCounterNames, setCounterNameToEnum, setDone, setExecContext, setGroupKeyObject, setId, setInputObjInspectors, setOperatorId, setOutputCollector, setParentOperators, setReporter, setSchema, setUseBucketizedHiveInputFormat, startGroup, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString, updateCounters

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

currentKeys

protected transient KeyWrapper currentKeys

newKeys

protected transient KeyWrapper newKeys

Constructor Detail

PTFOperator

public PTFOperator()

Method Detail

initializeOp

protected void initializeOp(Configuration jobConf)
                     throws HiveException

Description copied from class: Operator

Operator specific initialization.

Overrides:

initializeOp in class Operator<PTFDesc>

Throws:

HiveException

closeOp

protected void closeOp(boolean abort)
                throws HiveException

Description copied from class: Operator

Operator specific close routine. Operators which inherents this class should overwrite this funtion for their specific cleanup routine.

Overrides:

closeOp in class Operator<PTFDesc>

Throws:

HiveException

processOp

public void processOp(java.lang.Object row,
                      int tag)
               throws HiveException

Description copied from class: Operator

Process the row.

Specified by:

processOp in class Operator<PTFDesc>

Parameters:

row - The object representing the row.

tag - The tag of the row usually means which parent this row comes from. Rows with the same tag should have exactly the same rowInspector all the time.

Throws:

HiveException

reconstructQueryDef

protected void reconstructQueryDef(HiveConf hiveConf)
                            throws HiveException

Initialize the visitor to use the QueryDefDeserializer Use the order defined in QueryDefWalker to visit the QueryDef

Parameters:

hiveConf -

Throws:

HiveException

setupKeysWrapper

protected void setupKeysWrapper(ObjectInspector inputOI)
                         throws HiveException

Throws:

HiveException

processInputPartition

protected void processInputPartition()
                              throws HiveException

Throws:

HiveException

processMapFunction

protected void processMapFunction()
                           throws HiveException

Throws:

HiveException

getName

public java.lang.String getName()

Description copied from class: Operator

Implements the getName function for the Node Interface.

Specified by:

getName in interface Node

Overrides:

getName in class Operator<PTFDesc>

Returns:

the name of the operator

getOperatorName

public static java.lang.String getOperatorName()

getType

public OperatorType getType()

Description copied from class: Operator

Return the type of the specific operator among the types in OperatorType.

Specified by:

getType in class Operator<PTFDesc>

Returns:

OperatorType.*

createFirstPartitionForChain

public PTFPartition createFirstPartitionForChain(ObjectInspector oi,
                                                 HiveConf hiveConf,
                                                 boolean isMapSide)
                                          throws HiveException

Create a new Partition. A partition has 2 OIs: the OI for the rows being put in and the OI for the rows coming out. You specify the output OI by giving the Serde to use to Serialize. Typically these 2 OIs are the same; but not always. For the first PTF in a chain the OI of the incoming rows is dictated by the Parent Op to this PTFOp. The output OI from the Partition is typically LazyBinaryStruct, but not always. In the case of Noop/NoopMap we keep the Strcuture the same as what is given to us.

The Partition we want to create here is for feeding the First table function in the chain. So for map-side processing use the Serde from the output Shape its InputDef. For reduce-side processing use the Serde from its RawInputShape(the shape after map-side processing).

Parameters:

oi -

hiveConf -

isMapSide -

Returns:

Throws:

HiveException

connectLeadLagFunctionsToPartition

public static void connectLeadLagFunctionsToPartition(PTFDesc ptfDesc,
                                                      PTFPartition.PTFPartitionIterator<java.lang.Object> pItr)
                                               throws HiveException

Throws:

HiveException