org.apache.hive.hcatalog.templeton.tool

Class ZooKeeperStorage

java.lang.Object

org.apache.hive.hcatalog.templeton.tool.ZooKeeperStorage

All Implemented Interfaces:

TempletonStorage

public class ZooKeeperStorage
extends java.lang.Object
implements TempletonStorage

A storage implementation based on storing everything in ZooKeeper. This keeps everything in a central location that is guaranteed to be available and accessible. Data is stored with each key/value pair being a node in ZooKeeper.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hive.hcatalog.templeton.tool.TempletonStorage

TempletonStorage.Type

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	ENCODING
java.lang.String	job_path
java.lang.String	job_trackingpath
java.lang.String	overhead_path
java.lang.String	storage_root
static java.lang.String	TRACKINGDIR
static java.lang.String	ZK_HOSTS
static java.lang.String	ZK_SESSION_TIMEOUT

Fields inherited from interface org.apache.hive.hcatalog.templeton.tool.TempletonStorage

STORAGE_CLASS, STORAGE_ROOT

Constructor Summary

Constructors

Constructor and Description
ZooKeeperStorage()

Method Summary

Modifier and Type	Method and Description
void	close() Close this ZK connection.
void	closeStorage() For storage methods that require a connection, this is a hint that it's time to close the connection.
void	create(TempletonStorage.Type type, java.lang.String id) Create a node in ZooKeeper
boolean	delete(TempletonStorage.Type type, java.lang.String id) Delete a data grouping (all data for a jobid, all tracking data for a job, etc.).
java.util.List<java.lang.String>	getAll() Get the id of each data grouping in the storage system.
java.util.List<java.lang.String>	getAllForKey(java.lang.String key, java.lang.String value) Get the id of each data grouping that has the specific key/value pair.
java.util.List<java.lang.String>	getAllForType(TempletonStorage.Type type) Get the id of each data grouping of a given type in the storage system.
java.util.List<java.lang.String>	getAllForTypeAndKey(TempletonStorage.Type type, java.lang.String key, java.lang.String value) Get the id of each data grouping of a given type that has the specific key/value pair.
java.lang.String	getField(TempletonStorage.Type type, java.lang.String id, java.lang.String key) Get the value of one field for a given data type.
java.util.Map<java.lang.String,java.lang.String>	getFields(TempletonStorage.Type type, java.lang.String id) Get all the name/value pairs stored for this id.
java.lang.String	getPath(TempletonStorage.Type type) Get the path based on the job type.
static java.lang.String[]	getPaths(java.lang.String fullpath)
java.lang.String	makeFieldZnode(TempletonStorage.Type type, java.lang.String id, java.lang.String name) Make a ZK path to the named field.
java.lang.String	makeZnode(TempletonStorage.Type type, java.lang.String id) Make a ZK path to job
void	openStorage(Configuration config) For storage methods that require a connection, this is a hint that it's time to open a connection.
void	saveField(TempletonStorage.Type type, java.lang.String id, java.lang.String key, java.lang.String val) Save a single key/value pair for a specific job id.
void	startCleanup(Configuration config) Start the cleanup process for this storage type.
static org.apache.zookeeper.ZooKeeper	zkOpen(Configuration conf) Open a ZooKeeper connection for the JobState.
static org.apache.zookeeper.ZooKeeper	zkOpen(java.lang.String zkHosts, int zkSessionTimeout) Open a ZooKeeper connection for the JobState.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TRACKINGDIR

public static final java.lang.String TRACKINGDIR

See Also:

Constant Field Values

storage_root

public java.lang.String storage_root

job_path

public java.lang.String job_path

job_trackingpath

public java.lang.String job_trackingpath

overhead_path

public java.lang.String overhead_path

ZK_HOSTS

public static final java.lang.String ZK_HOSTS

See Also:

Constant Field Values

ZK_SESSION_TIMEOUT

public static final java.lang.String ZK_SESSION_TIMEOUT

See Also:

Constant Field Values

ENCODING

public static final java.lang.String ENCODING

See Also:

Constant Field Values

Constructor Detail

ZooKeeperStorage

public ZooKeeperStorage()

Method Detail

zkOpen

public static org.apache.zookeeper.ZooKeeper zkOpen(java.lang.String zkHosts,
                                                    int zkSessionTimeout)
                                             throws java.io.IOException

Open a ZooKeeper connection for the JobState.

Throws:

java.io.IOException

zkOpen

public static org.apache.zookeeper.ZooKeeper zkOpen(Configuration conf)
                                             throws java.io.IOException

Open a ZooKeeper connection for the JobState.

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Close this ZK connection.

Throws:

java.io.IOException

startCleanup

public void startCleanup(Configuration config)

Description copied from interface: TempletonStorage

Start the cleanup process for this storage type.

Specified by:

startCleanup in interface TempletonStorage

create

public void create(TempletonStorage.Type type,
                   java.lang.String id)
            throws java.io.IOException

Create a node in ZooKeeper

Throws:

java.io.IOException

getPath

public java.lang.String getPath(TempletonStorage.Type type)

Get the path based on the job type.

Parameters:

type -

getPaths

public static java.lang.String[] getPaths(java.lang.String fullpath)

makeFieldZnode

public java.lang.String makeFieldZnode(TempletonStorage.Type type,
                                       java.lang.String id,
                                       java.lang.String name)

Make a ZK path to the named field.

makeZnode

public java.lang.String makeZnode(TempletonStorage.Type type,
                                  java.lang.String id)

Make a ZK path to job

saveField

public void saveField(TempletonStorage.Type type,
                      java.lang.String id,
                      java.lang.String key,
                      java.lang.String val)
               throws NotFoundException

Description copied from interface: TempletonStorage

Save a single key/value pair for a specific job id.

Specified by:

saveField in interface TempletonStorage

Parameters:

type - The data type (as listed above)

id - The String id of this data grouping (jobid, etc.)

key - The name of the field to save

val - The value of the field to save

Throws:

NotFoundException

getField

public java.lang.String getField(TempletonStorage.Type type,
                                 java.lang.String id,
                                 java.lang.String key)

Description copied from interface: TempletonStorage

Get the value of one field for a given data type. If the type is UNKNOWN, search for the id in all types.

Specified by:

getField in interface TempletonStorage

Parameters:

type - The data type (as listed above)

id - The String id of this data grouping (jobid, etc.)

key - The name of the field to retrieve

Returns:

The value of the field requested, or null if not found.

getFields

public java.util.Map<java.lang.String,java.lang.String> getFields(TempletonStorage.Type type,
                                                                  java.lang.String id)

Description copied from interface: TempletonStorage

Get all the name/value pairs stored for this id. Be careful using getFields() -- optimistic locking will mean that your odds of a conflict are decreased if you read/write one field at a time. getFields() is intended for read-only usage. If the type is UNKNOWN, search for the id in all types.

Specified by:

getFields in interface TempletonStorage

Parameters:

type - The data type (as listed above)

id - The String id of this data grouping (jobid, etc.)

Returns:

A Map of key/value pairs found for this type/id.

delete

public boolean delete(TempletonStorage.Type type,
                      java.lang.String id)
               throws NotFoundException

Description copied from interface: TempletonStorage

Delete a data grouping (all data for a jobid, all tracking data for a job, etc.). If the type is UNKNOWN, search for the id in all types.

Specified by:

delete in interface TempletonStorage

Parameters:

type - The data type (as listed above)

id - The String id of this data grouping (jobid, etc.)

Returns:

True if successful, false if not, throws NotFoundException if the id wasn't found.

Throws:

NotFoundException

getAll

public java.util.List<java.lang.String> getAll()

Description copied from interface: TempletonStorage

Get the id of each data grouping in the storage system.

Specified by:

getAll in interface TempletonStorage

Returns:

An ArrayList of ids.

getAllForType

public java.util.List<java.lang.String> getAllForType(TempletonStorage.Type type)

Description copied from interface: TempletonStorage

Get the id of each data grouping of a given type in the storage system.

Specified by:

getAllForType in interface TempletonStorage

Parameters:

type - The data type (as listed above)

Returns:

An ArrayList of ids.

getAllForKey

public java.util.List<java.lang.String> getAllForKey(java.lang.String key,
                                                     java.lang.String value)

Description copied from interface: TempletonStorage

Get the id of each data grouping that has the specific key/value pair.

Specified by:

getAllForKey in interface TempletonStorage

Parameters:

key - The name of the field to search for

value - The value of the field to search for

Returns:

An ArrayList of ids.

getAllForTypeAndKey

public java.util.List<java.lang.String> getAllForTypeAndKey(TempletonStorage.Type type,
                                                            java.lang.String key,
                                                            java.lang.String value)

Description copied from interface: TempletonStorage

Get the id of each data grouping of a given type that has the specific key/value pair.

Specified by:

getAllForTypeAndKey in interface TempletonStorage

Parameters:

type - The data type (as listed above)

key - The name of the field to search for

value - The value of the field to search for

Returns:

An ArrayList of ids.

openStorage

public void openStorage(Configuration config)
                 throws java.io.IOException

Description copied from interface: TempletonStorage

For storage methods that require a connection, this is a hint that it's time to open a connection.

Specified by:

openStorage in interface TempletonStorage

Throws:

java.io.IOException

closeStorage

public void closeStorage()
                  throws java.io.IOException

Description copied from interface: TempletonStorage

For storage methods that require a connection, this is a hint that it's time to close the connection.

Specified by:

closeStorage in interface TempletonStorage

Throws:

java.io.IOException