org.apache.hadoop.contrib.bkjournal

Class BookKeeperJournalManager

java.lang.Object

org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager

All Implemented Interfaces:

Closeable, AutoCloseable, Storage.FormatConfirmable, JournalManager

public class BookKeeperJournalManager
extends Object
implements JournalManager

BookKeeper Journal Manager To use, add the following to hdfs-site.xml.

 <property>
   <name>dfs.namenode.edits.dir</name>
   <value>bookkeeper://zk1:2181;zk2:2181;zk3:2181/hdfsjournal</value>
 </property>

 <property>
   <name>dfs.namenode.edits.journal-plugin.bookkeeper</name>
   <value>org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager</value>
 </property>
 
 

The URI format for bookkeeper is bookkeeper://[zkEnsemble]/[rootZnode] [zookkeeper ensemble] is a list of semi-colon separated, zookeeper host:port pairs. In the example above there are 3 servers, in the ensemble, zk1, zk2 & zk3, each one listening on port 2181. [root znode] is the path of the zookeeper znode, under which the editlog information will be stored. Other configuration options are:

• dfs.namenode.bookkeeperjournal.output-buffer-size Number of bytes a bookkeeper journal stream will buffer before forcing a flush. Default is 1024.
• dfs.namenode.bookkeeperjournal.ensemble-size Number of bookkeeper servers in edit log ledger ensembles. This is the number of bookkeeper servers which need to be available for the ledger to be writable. Default is 3.
• dfs.namenode.bookkeeperjournal.quorum-size Number of bookkeeper servers in the write quorum. This is the number of bookkeeper servers which must have acknowledged the write of an entry before it is considered written. Default is 2.
• dfs.namenode.bookkeeperjournal.digestPw Password to use when creating ledgers.
• dfs.namenode.bookkeeperjournal.zk.session.timeout Session timeout for Zookeeper client from BookKeeper Journal Manager. Hadoop recommends that, this value should be less than the ZKFC session timeout value. Default value is 3000.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hdfs.server.namenode.JournalManager

JournalManager.CorruptionException

Field Summary

Fields

Modifier and Type	Field and Description
static String	BKJM_BOOKKEEPER_DIGEST_PW
static String	BKJM_BOOKKEEPER_DIGEST_PW_DEFAULT
static String	BKJM_BOOKKEEPER_ENSEMBLE_SIZE
static int	BKJM_BOOKKEEPER_ENSEMBLE_SIZE_DEFAULT
static String	BKJM_BOOKKEEPER_QUORUM_SIZE
static int	BKJM_BOOKKEEPER_QUORUM_SIZE_DEFAULT
static String	BKJM_OUTPUT_BUFFER_SIZE
static int	BKJM_OUTPUT_BUFFER_SIZE_DEFAULT
static String	BKJM_ZK_LEDGERS_AVAILABLE_PATH
static String	BKJM_ZK_LEDGERS_AVAILABLE_PATH_DEFAULT
static String	BKJM_ZK_SESSION_TIMEOUT
static int	BKJM_ZK_SESSION_TIMEOUT_DEFAULT

Constructor Summary

Constructors

Constructor and Description
BookKeeperJournalManager(Configuration conf, URI uri, NamespaceInfo nsInfo) Construct a Bookkeeper journal manager.

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Close the journal manager, freeing any resources it may hold.
void	finalizeLogSegment(long firstTxId, long lastTxId) Finalize a log segment.
void	format(NamespaceInfo ns) Format the underlying storage, removing any previously stored data.
boolean	hasSomeData()
void	purgeLogsOlderThan(long minTxIdToKeep) Remove all edit logs with transaction IDs lower than the given transaction ID.
void	recoverUnfinalizedSegments() Recover segments which have not been finalized.
void	selectInputStreams(Collection<EditLogInputStream> streams, long fromTxId, boolean inProgressOk)
void	selectInputStreams(Collection<EditLogInputStream> streams, long fromTxId, boolean inProgressOk, boolean forReading) Get a list of edit log input streams.
void	setOutputBufferCapacity(int size) Set the amount of memory that this stream should use to buffer edits.
EditLogOutputStream	startLogSegment(long txId) Start a new log segment in a BookKeeper ledger.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hdfs.server.common.Storage.FormatConfirmable

toString

Field Detail

BKJM_OUTPUT_BUFFER_SIZE

public static final String BKJM_OUTPUT_BUFFER_SIZE

See Also:

Constant Field Values

BKJM_OUTPUT_BUFFER_SIZE_DEFAULT

public static final int BKJM_OUTPUT_BUFFER_SIZE_DEFAULT

See Also:

Constant Field Values

BKJM_BOOKKEEPER_ENSEMBLE_SIZE

public static final String BKJM_BOOKKEEPER_ENSEMBLE_SIZE

See Also:

Constant Field Values

BKJM_BOOKKEEPER_ENSEMBLE_SIZE_DEFAULT

public static final int BKJM_BOOKKEEPER_ENSEMBLE_SIZE_DEFAULT

See Also:

Constant Field Values

BKJM_BOOKKEEPER_QUORUM_SIZE

public static final String BKJM_BOOKKEEPER_QUORUM_SIZE

See Also:

Constant Field Values

BKJM_BOOKKEEPER_QUORUM_SIZE_DEFAULT

public static final int BKJM_BOOKKEEPER_QUORUM_SIZE_DEFAULT

See Also:

Constant Field Values

BKJM_BOOKKEEPER_DIGEST_PW

public static final String BKJM_BOOKKEEPER_DIGEST_PW

See Also:

Constant Field Values

BKJM_BOOKKEEPER_DIGEST_PW_DEFAULT

public static final String BKJM_BOOKKEEPER_DIGEST_PW_DEFAULT

See Also:

Constant Field Values

BKJM_ZK_SESSION_TIMEOUT

public static final String BKJM_ZK_SESSION_TIMEOUT

See Also:

Constant Field Values

BKJM_ZK_SESSION_TIMEOUT_DEFAULT

public static final int BKJM_ZK_SESSION_TIMEOUT_DEFAULT

See Also:

Constant Field Values

BKJM_ZK_LEDGERS_AVAILABLE_PATH

public static final String BKJM_ZK_LEDGERS_AVAILABLE_PATH

See Also:

Constant Field Values

BKJM_ZK_LEDGERS_AVAILABLE_PATH_DEFAULT

public static final String BKJM_ZK_LEDGERS_AVAILABLE_PATH_DEFAULT

See Also:

Constant Field Values

Constructor Detail

BookKeeperJournalManager

public BookKeeperJournalManager(Configuration conf,
                        URI uri,
                        NamespaceInfo nsInfo)
                         throws IOException

Construct a Bookkeeper journal manager.

Throws:

IOException

Method Detail

format

public void format(NamespaceInfo ns)
            throws IOException

Description copied from interface: JournalManager

Format the underlying storage, removing any previously stored data.

Specified by:

format in interface JournalManager

Throws:

IOException

hasSomeData

public boolean hasSomeData()
                    throws IOException

Specified by:

hasSomeData in interface Storage.FormatConfirmable

Returns:

true if the storage seems to have some valid data in it, and the user should be required to confirm the format. Otherwise, false.

Throws:

IOException - if the storage cannot be accessed at all.

startLogSegment

public EditLogOutputStream startLogSegment(long txId)
                                    throws IOException

Start a new log segment in a BookKeeper ledger. First ensure that we have the write lock for this journal. Then create a ledger and stream based on that ledger. The ledger id is written to the inprogress znode, so that in the case of a crash, a recovery process can find the ledger we were writing to when we crashed.

Specified by:

startLogSegment in interface JournalManager

Parameters:

txId - First transaction id to be written to the stream

Throws:

IOException

finalizeLogSegment

public void finalizeLogSegment(long firstTxId,
                      long lastTxId)
                        throws IOException

Finalize a log segment. If the journal manager is currently writing to a ledger, ensure that this is the ledger of the log segment being finalized. Otherwise this is the recovery case. In the recovery case, ensure that the firstTxId of the ledger matches firstTxId for the segment we are trying to finalize.

Specified by:

finalizeLogSegment in interface JournalManager

Throws:

IOException

selectInputStreams

public void selectInputStreams(Collection<EditLogInputStream> streams,
                      long fromTxId,
                      boolean inProgressOk)
                        throws IOException

Throws:

IOException

selectInputStreams

public void selectInputStreams(Collection<EditLogInputStream> streams,
                      long fromTxId,
                      boolean inProgressOk,
                      boolean forReading)
                        throws IOException

Get a list of edit log input streams. The list will start with the stream that contains fromTxnId, and continue until the end of the journal being managed.

fromTxId - the first transaction id we want to read

inProgressOk - whether or not in-progress streams should be returned

forReading - whether or not the caller intends to read from the edit logs

Throws:

IOException - if the underlying storage has an error or is otherwise inaccessible

recoverUnfinalizedSegments

public void recoverUnfinalizedSegments()
                                throws IOException

Description copied from interface: JournalManager

Recover segments which have not been finalized.

Specified by:

recoverUnfinalizedSegments in interface JournalManager

Throws:

IOException

purgeLogsOlderThan

public void purgeLogsOlderThan(long minTxIdToKeep)
                        throws IOException

Remove all edit logs with transaction IDs lower than the given transaction ID.

Parameters:

minTxIdToKeep - the lowest transaction ID that should be retained

Throws:

IOException - in the event of error

close

public void close()
           throws IOException

Description copied from interface: JournalManager

Close the journal manager, freeing any resources it may hold.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface JournalManager

Throws:

IOException

setOutputBufferCapacity

public void setOutputBufferCapacity(int size)

Set the amount of memory that this stream should use to buffer edits. Setting this will only affect future output stream. Streams which have currently be created won't be affected.

Specified by:

setOutputBufferCapacity in interface JournalManager