org.apache.hadoop.hive.ql.io

Class SymlinkTextInputFormat

java.lang.Object

org.apache.hadoop.hive.ql.io.SymbolicInputFormat

org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat

All Implemented Interfaces:

ContentSummaryInputFormat, ReworkMapredInputFormat

public class SymlinkTextInputFormat
extends SymbolicInputFormat
implements ContentSummaryInputFormat, ReworkMapredInputFormat

Symlink file is a text file which contains a list of filename / dirname. This input method reads symlink files from specified job input paths and takes the files / directories specified in those symlink files as actual map-reduce input. The target input data should be in TextInputFormat.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SymlinkTextInputFormat.SymlinkTextInputSplit This input split wraps the FileSplit generated from TextInputFormat.getSplits(), while setting the original link file path as job input path.

Constructor Summary

Constructors

Constructor and Description
SymlinkTextInputFormat()

Method Summary

Modifier and Type	Method and Description
void	configure(JobConf job)
ContentSummary	getContentSummary(Path p, JobConf job)
<any>	getRecordReader(InputSplit split, JobConf job, Reporter reporter)
InputSplit[]	getSplits(JobConf job, int numSplits) Parses all target paths from job input directory which contains symlink files, and splits the target data using TextInputFormat.
void	validateInput(JobConf job) For backward compatibility with hadoop 0.17.

Methods inherited from class org.apache.hadoop.hive.ql.io.SymbolicInputFormat

rework

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hive.ql.io.ReworkMapredInputFormat

rework

Constructor Detail

SymlinkTextInputFormat

public SymlinkTextInputFormat()

Method Detail

getRecordReader

public <any> getRecordReader(InputSplit split,
                             JobConf job,
                             Reporter reporter)
                      throws java.io.IOException

Throws:

java.io.IOException

getSplits

public InputSplit[] getSplits(JobConf job,
                              int numSplits)
                       throws java.io.IOException

Parses all target paths from job input directory which contains symlink files, and splits the target data using TextInputFormat.

Throws:

java.io.IOException

configure

public void configure(JobConf job)

validateInput

public void validateInput(JobConf job)
                   throws java.io.IOException

For backward compatibility with hadoop 0.17.

Throws:

java.io.IOException

getContentSummary

public ContentSummary getContentSummary(Path p,
                                        JobConf job)
                                 throws java.io.IOException

Specified by:

getContentSummary in interface ContentSummaryInputFormat

Throws:

java.io.IOException