org.apache.hive.hcatalog.templeton.tool

Class HDFSStorage

java.lang.Object

org.apache.hive.hcatalog.templeton.tool.HDFSStorage

All Implemented Interfaces:

TempletonStorage

public class HDFSStorage
extends java.lang.Object
implements TempletonStorage

HDFS implementation of templeton storage. This implementation assumes that all keys in key/value pairs are chosen such that they don't have any newlines in them.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hive.hcatalog.templeton.tool.TempletonStorage

TempletonStorage.Type

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	JOB_PATH
static java.lang.String	JOB_TRACKINGPATH
static java.lang.String	OVERHEAD_PATH
java.lang.String	storage_root

Fields inherited from interface org.apache.hive.hcatalog.templeton.tool.TempletonStorage

STORAGE_CLASS, STORAGE_ROOT

Constructor Summary

Constructors

Constructor and Description
HDFSStorage()

Method Summary

Modifier and Type	Method and Description
void	closeStorage() For storage methods that require a connection, this is a hint that it's time to close the connection.
boolean	delete(TempletonStorage.Type type, java.lang.String id) Delete a data grouping (all data for a jobid, all tracking data for a job, etc.).
java.util.List<java.lang.String>	getAll() Get the id of each data grouping in the storage system.
java.util.List<java.lang.String>	getAllForKey(java.lang.String key, java.lang.String value) Get the id of each data grouping that has the specific key/value pair.
java.util.List<java.lang.String>	getAllForType(TempletonStorage.Type type) Get the id of each data grouping of a given type in the storage system.
java.util.List<java.lang.String>	getAllForTypeAndKey(TempletonStorage.Type type, java.lang.String key, java.lang.String value) Get the id of each data grouping of a given type that has the specific key/value pair.
java.lang.String	getField(TempletonStorage.Type type, java.lang.String id, java.lang.String key) Get the value of one field for a given data type.
java.util.Map<java.lang.String,java.lang.String>	getFields(TempletonStorage.Type type, java.lang.String id) Get all the name/value pairs stored for this id.
java.lang.String	getPath(TempletonStorage.Type type) Get the path to storage based on the type.
static java.lang.String	getPath(TempletonStorage.Type type, java.lang.String root) Static method to get the path based on the type.
void	openStorage(Configuration config) For storage methods that require a connection, this is a hint that it's time to open a connection.
void	saveField(TempletonStorage.Type type, java.lang.String id, java.lang.String key, java.lang.String val) Save a single key/value pair for a specific job id.
void	startCleanup(Configuration config) Start the cleanup process for this storage type.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

storage_root

public java.lang.String storage_root

JOB_PATH

public static final java.lang.String JOB_PATH

See Also:

Constant Field Values

JOB_TRACKINGPATH

public static final java.lang.String JOB_TRACKINGPATH

See Also:

Constant Field Values

OVERHEAD_PATH

public static final java.lang.String OVERHEAD_PATH

See Also:

Constant Field Values

Constructor Detail

HDFSStorage

public HDFSStorage()

Method Detail

startCleanup

public void startCleanup(Configuration config)

Description copied from interface: TempletonStorage

Start the cleanup process for this storage type.

Specified by:

startCleanup in interface TempletonStorage

saveField

public void saveField(TempletonStorage.Type type,
                      java.lang.String id,
                      java.lang.String key,
                      java.lang.String val)
               throws NotFoundException

Description copied from interface: TempletonStorage

Save a single key/value pair for a specific job id.

Specified by:

saveField in interface TempletonStorage

Parameters:

type - The data type (as listed above)

id - The String id of this data grouping (jobid, etc.)

key - The name of the field to save

val - The value of the field to save

Throws:

NotFoundException

getField

public java.lang.String getField(TempletonStorage.Type type,
                                 java.lang.String id,
                                 java.lang.String key)

Description copied from interface: TempletonStorage

Get the value of one field for a given data type. If the type is UNKNOWN, search for the id in all types.

Specified by:

getField in interface TempletonStorage

Parameters:

type - The data type (as listed above)

id - The String id of this data grouping (jobid, etc.)

key - The name of the field to retrieve

Returns:

The value of the field requested, or null if not found.

getFields

public java.util.Map<java.lang.String,java.lang.String> getFields(TempletonStorage.Type type,
                                                                  java.lang.String id)

Description copied from interface: TempletonStorage

Get all the name/value pairs stored for this id. Be careful using getFields() -- optimistic locking will mean that your odds of a conflict are decreased if you read/write one field at a time. getFields() is intended for read-only usage. If the type is UNKNOWN, search for the id in all types.

Specified by:

getFields in interface TempletonStorage

Parameters:

type - The data type (as listed above)

id - The String id of this data grouping (jobid, etc.)

Returns:

A Map of key/value pairs found for this type/id.

delete

public boolean delete(TempletonStorage.Type type,
                      java.lang.String id)
               throws NotFoundException

Description copied from interface: TempletonStorage

Delete a data grouping (all data for a jobid, all tracking data for a job, etc.). If the type is UNKNOWN, search for the id in all types.

Specified by:

delete in interface TempletonStorage

Parameters:

type - The data type (as listed above)

id - The String id of this data grouping (jobid, etc.)

Returns:

True if successful, false if not, throws NotFoundException if the id wasn't found.

Throws:

NotFoundException

getAll

public java.util.List<java.lang.String> getAll()

Description copied from interface: TempletonStorage

Get the id of each data grouping in the storage system.

Specified by:

getAll in interface TempletonStorage

Returns:

An ArrayList of ids.

getAllForType

public java.util.List<java.lang.String> getAllForType(TempletonStorage.Type type)

Description copied from interface: TempletonStorage

Get the id of each data grouping of a given type in the storage system.

Specified by:

getAllForType in interface TempletonStorage

Parameters:

type - The data type (as listed above)

Returns:

An ArrayList of ids.

getAllForKey

public java.util.List<java.lang.String> getAllForKey(java.lang.String key,
                                                     java.lang.String value)

Description copied from interface: TempletonStorage

Get the id of each data grouping that has the specific key/value pair.

Specified by:

getAllForKey in interface TempletonStorage

Parameters:

key - The name of the field to search for

value - The value of the field to search for

Returns:

An ArrayList of ids.

getAllForTypeAndKey

public java.util.List<java.lang.String> getAllForTypeAndKey(TempletonStorage.Type type,
                                                            java.lang.String key,
                                                            java.lang.String value)

Description copied from interface: TempletonStorage

Get the id of each data grouping of a given type that has the specific key/value pair.

Specified by:

getAllForTypeAndKey in interface TempletonStorage

Parameters:

type - The data type (as listed above)

key - The name of the field to search for

value - The value of the field to search for

Returns:

An ArrayList of ids.

openStorage

public void openStorage(Configuration config)
                 throws java.io.IOException

Description copied from interface: TempletonStorage

For storage methods that require a connection, this is a hint that it's time to open a connection.

Specified by:

openStorage in interface TempletonStorage

Throws:

java.io.IOException

closeStorage

public void closeStorage()
                  throws java.io.IOException

Description copied from interface: TempletonStorage

For storage methods that require a connection, this is a hint that it's time to close the connection.

Specified by:

closeStorage in interface TempletonStorage

Throws:

java.io.IOException

getPath

public java.lang.String getPath(TempletonStorage.Type type)

Get the path to storage based on the type.

Parameters:

type -

getPath

public static java.lang.String getPath(TempletonStorage.Type type,
                                       java.lang.String root)

Static method to get the path based on the type.

Parameters:

type -

root -