org.apache.hadoop.hive.ql.exec

Class GroupByOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<GroupByDesc>

org.apache.hadoop.hive.ql.exec.GroupByOperator

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable, Node

public class GroupByOperator
extends Operator<GroupByDesc>
implements java.io.Serializable

GroupBy operator implementation.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.OperatorFunc, Operator.ProgressCounter, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean[]	aggregationIsDistinct
protected ExprNodeEvaluator[][]	aggregationParameterFields
protected ObjectInspector[][]	aggregationParameterObjectInspectors
protected java.lang.Object[][]	aggregationParameterObjects
protected ObjectInspector[][]	aggregationParameterStandardObjectInspectors
protected GenericUDAFEvaluator.AggregationBuffer[]	aggregations
protected java.lang.Object[][]	aggregationsParametersLastInvoke
static java.lang.String	counterNameHashOut
protected ObjectInspector[]	currentKeyObjectInspectors
protected KeyWrapper	currentKeys
protected java.util.Map<java.lang.Integer,java.util.Set<java.lang.Integer>>	distinctKeyAggrs
protected java.util.HashMap<KeyWrapper,GenericUDAFEvaluator.AggregationBuffer[]>	hashAggregations
protected ExprNodeEvaluator[]	keyFields
protected ObjectInspector[]	keyObjectInspectors
protected java.util.HashSet<KeyWrapper>	keysCurrentGroup
static java.lang.management.MemoryMXBean	memoryMXBean
protected KeyWrapper	newKeys
protected java.util.List<java.lang.Integer>	nonDistinctAggrs
protected java.util.Map<java.lang.Integer,java.util.Set<java.lang.Integer>>	nonDistinctKeyAggrs
protected java.util.ArrayList<ObjectInspector>	objectInspectors

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

alias, beginTime, childOperators, childOperatorsArray, childOperatorsTag, colExprMap, conf, counterNames, counterNameToEnum, counters, done, fatalErrorCntr, groupKeyObject, id, inputObjInspectors, inputRows, isLogInfoEnabled, numInputRowsCntr, numOutputRowsCntr, operatorId, out, outputObjInspector, outputRows, parentOperators, reporter, state, statsMap, timeTakenCntr, totalTime

Constructor Summary

Constructors

Constructor and Description
GroupByOperator()

Method Summary

Modifier and Type	Method and Description
boolean	acceptLimitPushdown() we can push the limit above GBY (running in Reducer), since that will generate single row for each group.
void	closeOp(boolean abort) We need to forward all the aggregations to children.
void	endGroup()
void	flush() Forward all aggregations to children.
protected void	forward(java.lang.Object[] keys, GenericUDAFEvaluator.AggregationBuffer[] aggs) Forward a record of keys and aggregation results.
java.util.List<java.lang.String>	genColLists(java.util.HashMap<Operator<? extends OperatorDesc>,OpParseContext> opParseCtx)
protected java.util.List<java.lang.String>	getAdditionalCounters()
java.lang.String	getName() Implements the getName function for the Node Interface.
static java.lang.String	getOperatorName()
OperatorType	getType() Return the type of the specific operator among the types in OperatorType.
static javolution.util.FastBitSet	groupingSet2BitSet(int value)
protected void	initializeOp(Configuration hconf) Operator specific initialization.
protected GenericUDAFEvaluator.AggregationBuffer[]	newAggregations()
void	processOp(java.lang.Object row, int tag) Process the row.
protected void	resetAggregations(GenericUDAFEvaluator.AggregationBuffer[] aggs)
void	startGroup()
protected void	updateAggregations(GenericUDAFEvaluator.AggregationBuffer[] aggs, java.lang.Object row, ObjectInspector rowInspector, boolean hashAggr, boolean newEntryForHashAggr, java.lang.Object[][] lastInvoke)

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

allInitializedParentsAreClosed, areAllParentsInitialized, assignCounterNameToEnum, augmentPlan, checkFatalErrors, cleanUpInputFileChanged, cleanUpInputFileChangedOp, clone, close, columnNamesRowResolvedCanBeObtained, dump, dump, fatalErrorMessage, forward, getChildOperators, getChildren, getColumnExprMap, getConf, getConfiguration, getCounterNames, getCounterNameToEnum, getCounters, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getNextCntr, getNumChild, getNumParent, getOperatorId, getParentOperators, getSchema, getStats, getWrappedCounterName, incrCounter, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeCounters, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logStats, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, process, processGroup, removeChild, removeChildAndAdoptItsChildren, removeChildren, removeParent, replaceChild, replaceParent, reset, resetId, resetLastEnumUsed, resetStats, setAlias, setChildOperators, setColumnExprMap, setConf, setCounterNames, setCounterNameToEnum, setDone, setExecContext, setGroupKeyObject, setId, setInputObjInspectors, setOperatorId, setOutputCollector, setParentOperators, setReporter, setSchema, setUseBucketizedHiveInputFormat, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString, updateCounters

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

counterNameHashOut

public static final java.lang.String counterNameHashOut

See Also:

Constant Field Values

keyFields

protected transient ExprNodeEvaluator[] keyFields

keyObjectInspectors

protected transient ObjectInspector[] keyObjectInspectors

aggregationParameterFields

protected transient ExprNodeEvaluator[][] aggregationParameterFields

aggregationParameterObjectInspectors

protected transient ObjectInspector[][] aggregationParameterObjectInspectors

aggregationParameterStandardObjectInspectors

protected transient ObjectInspector[][] aggregationParameterStandardObjectInspectors

aggregationParameterObjects

protected transient java.lang.Object[][] aggregationParameterObjects

aggregationIsDistinct

protected transient boolean[] aggregationIsDistinct

distinctKeyAggrs

protected transient java.util.Map<java.lang.Integer,java.util.Set<java.lang.Integer>> distinctKeyAggrs

nonDistinctKeyAggrs

protected transient java.util.Map<java.lang.Integer,java.util.Set<java.lang.Integer>> nonDistinctKeyAggrs

nonDistinctAggrs

protected transient java.util.List<java.lang.Integer> nonDistinctAggrs

objectInspectors

protected transient java.util.ArrayList<ObjectInspector> objectInspectors

currentKeys

protected transient KeyWrapper currentKeys

newKeys

protected transient KeyWrapper newKeys

aggregations

protected transient GenericUDAFEvaluator.AggregationBuffer[] aggregations

aggregationsParametersLastInvoke

protected transient java.lang.Object[][] aggregationsParametersLastInvoke

hashAggregations

protected transient java.util.HashMap<KeyWrapper,GenericUDAFEvaluator.AggregationBuffer[]> hashAggregations

keysCurrentGroup

protected transient java.util.HashSet<KeyWrapper> keysCurrentGroup

currentKeyObjectInspectors

protected transient ObjectInspector[] currentKeyObjectInspectors

memoryMXBean

public static java.lang.management.MemoryMXBean memoryMXBean

Constructor Detail

GroupByOperator

public GroupByOperator()

Method Detail

groupingSet2BitSet

public static javolution.util.FastBitSet groupingSet2BitSet(int value)

initializeOp

protected void initializeOp(Configuration hconf)
                     throws HiveException

Description copied from class: Operator

Operator specific initialization.

Overrides:

initializeOp in class Operator<GroupByDesc>

Throws:

HiveException

newAggregations

protected GenericUDAFEvaluator.AggregationBuffer[] newAggregations()
                                                            throws HiveException

Throws:

HiveException

resetAggregations

protected void resetAggregations(GenericUDAFEvaluator.AggregationBuffer[] aggs)
                          throws HiveException

Throws:

HiveException

updateAggregations

protected void updateAggregations(GenericUDAFEvaluator.AggregationBuffer[] aggs,
                                  java.lang.Object row,
                                  ObjectInspector rowInspector,
                                  boolean hashAggr,
                                  boolean newEntryForHashAggr,
                                  java.lang.Object[][] lastInvoke)
                           throws HiveException

Throws:

HiveException

startGroup

public void startGroup()
                throws HiveException

Overrides:

startGroup in class Operator<GroupByDesc>

Throws:

HiveException

endGroup

public void endGroup()
              throws HiveException

Overrides:

endGroup in class Operator<GroupByDesc>

Throws:

HiveException

processOp

public void processOp(java.lang.Object row,
                      int tag)
               throws HiveException

Description copied from class: Operator

Process the row.

Specified by:

processOp in class Operator<GroupByDesc>

Parameters:

row - The object representing the row.

tag - The tag of the row usually means which parent this row comes from. Rows with the same tag should have exactly the same rowInspector all the time.

Throws:

HiveException

forward

protected void forward(java.lang.Object[] keys,
                       GenericUDAFEvaluator.AggregationBuffer[] aggs)
                throws HiveException

Forward a record of keys and aggregation results.

Parameters:

keys - The keys in the record

Throws:

HiveException

flush

public void flush()
           throws HiveException

Forward all aggregations to children. It is only used by DemuxOperator.

Overrides:

flush in class Operator<GroupByDesc>

Throws:

HiveException

closeOp

public void closeOp(boolean abort)
             throws HiveException

We need to forward all the aggregations to children.

Overrides:

closeOp in class Operator<GroupByDesc>

Throws:

HiveException

getAdditionalCounters

protected java.util.List<java.lang.String> getAdditionalCounters()

Overrides:

getAdditionalCounters in class Operator<GroupByDesc>

genColLists

public java.util.List<java.lang.String> genColLists(java.util.HashMap<Operator<? extends OperatorDesc>,OpParseContext> opParseCtx)

getName

public java.lang.String getName()

Description copied from class: Operator

Implements the getName function for the Node Interface.

Specified by:

getName in interface Node

Overrides:

getName in class Operator<GroupByDesc>

Returns:

the name of the operator

getOperatorName

public static java.lang.String getOperatorName()

getType

public OperatorType getType()

Description copied from class: Operator

Return the type of the specific operator among the types in OperatorType.

Specified by:

getType in class Operator<GroupByDesc>

Returns:

OperatorType.*

acceptLimitPushdown

public boolean acceptLimitPushdown()

we can push the limit above GBY (running in Reducer), since that will generate single row for each group. This doesn't necessarily hold for GBY (running in Mappers), so we don't push limit above it.

Overrides:

acceptLimitPushdown in class Operator<GroupByDesc>