parquet.avro

Class AvroParquetInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

parquet.hadoop.ParquetInputFormat<org.apache.avro.generic.IndexedRecord>

parquet.avro.AvroParquetInputFormat

public class AvroParquetInputFormat
extends ParquetInputFormat<org.apache.avro.generic.IndexedRecord>

A Hadoop InputFormat for Parquet files.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

org.apache.hadoop.mapreduce.lib.input.FileInputFormat.Counter

Field Summary

Fields inherited from class parquet.hadoop.ParquetInputFormat

READ_SUPPORT_CLASS, UNBOUND_RECORD_FILTER

Fields inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

DEFAULT_LIST_STATUS_NUM_THREADS, INPUT_DIR, INPUT_DIR_RECURSIVE, LIST_STATUS_NUM_THREADS, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE

Constructor Summary

Constructors

Constructor and Description
AvroParquetInputFormat()

Method Summary

Modifier and Type	Method and Description
static void	setAvroReadSchema(org.apache.hadoop.mapreduce.Job job, org.apache.avro.Schema avroReadSchema) Override the Avro schema to use for reading.
static void	setRequestedProjection(org.apache.hadoop.mapreduce.Job job, org.apache.avro.Schema requestedProjection) Set the subset of columns to read (projection pushdown).

Methods inherited from class parquet.hadoop.ParquetInputFormat

createRecordReader, getFooters, getFooters, getGlobalMetaData, getReadSupport, getReadSupportClass, getSplits, getSplits, getUnboundRecordFilter, listStatus, setReadSupportClass, setReadSupportClass, setUnboundRecordFilter

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getFormatMinSplitSize, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, isSplitable, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AvroParquetInputFormat

public AvroParquetInputFormat()

Method Detail

setRequestedProjection

public static void setRequestedProjection(org.apache.hadoop.mapreduce.Job job,
                                          org.apache.avro.Schema requestedProjection)

Set the subset of columns to read (projection pushdown). Specified as an Avro schema, the requested projection is converted into a Parquet schema for Parquet column projection.

This is useful if the full schema is large and you only want to read a few columns, since it saves time by not reading unused columns.

If a requested projection is set, then the Avro schema used for reading must be compatible with the projection. For instance, if a column is not included in the projection then it must either not be included or be optional in the read schema. Use setAvroReadSchema(org.apache.hadoop.mapreduce.Job, org.apache.avro.Schema) to set a read schema, if needed.

Parameters:

job -

requestedProjection -

See Also:

setAvroReadSchema(org.apache.hadoop.mapreduce.Job, org.apache.avro.Schema), AvroParquetOutputFormat.setSchema(org.apache.hadoop.mapreduce.Job, org.apache.avro.Schema)

setAvroReadSchema

public static void setAvroReadSchema(org.apache.hadoop.mapreduce.Job job,
                                     org.apache.avro.Schema avroReadSchema)

Override the Avro schema to use for reading. If not set, the Avro schema used for writing is used.

Differences between the read and write schemas are resolved using Avro's schema resolution rules.

Parameters:

job -

avroReadSchema -

See Also:

setRequestedProjection(org.apache.hadoop.mapreduce.Job, org.apache.avro.Schema), AvroParquetOutputFormat.setSchema(org.apache.hadoop.mapreduce.Job, org.apache.avro.Schema)