org.apache.hadoop.hdfs.server.namenode

Class NNStorage

java.lang.Object

org.apache.hadoop.hdfs.server.common.StorageInfo

org.apache.hadoop.hdfs.server.common.Storage

org.apache.hadoop.hdfs.server.namenode.NNStorage

All Implemented Interfaces:

Closeable, AutoCloseable, StorageErrorReporter

@InterfaceAudience.Private
public class NNStorage
extends Storage
implements Closeable, StorageErrorReporter

NNStorage is responsible for management of the StorageDirectories used by the NameNode.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hdfs.server.common.Storage

Storage.FormatConfirmable, Storage.StorageDirectory, Storage.StorageDirType, Storage.StorageState

Field Summary

Fields

Modifier and Type	Field and Description
protected String	blockpoolID
protected long	mostRecentCheckpointTxId TxId of the last transaction that was included in the most recent fsimage file.
protected List<Storage.StorageDirectory>	removedStorageDirs list of failed (and thus removed) storages

Fields inherited from class org.apache.hadoop.hdfs.server.common.Storage

LAST_PRE_UPGRADE_LAYOUT_VERSION, LAST_UPGRADABLE_HADOOP_VERSION, LAST_UPGRADABLE_LAYOUT_VERSION, LAYOUT_VERSIONS_203, LOG, STORAGE_1_BBW, STORAGE_DIR_CURRENT, STORAGE_DIR_PREVIOUS, STORAGE_FILE_LOCK, STORAGE_FILE_VERSION, STORAGE_PREVIOUS_CKPT, STORAGE_TMP_FINALIZED, STORAGE_TMP_LAST_CKPT, STORAGE_TMP_PREVIOUS, STORAGE_TMP_REMOVED, storageDirs, storageType

Fields inherited from class org.apache.hadoop.hdfs.server.common.StorageInfo

clusterID, cTime, layoutVersion, namespaceID

Constructor Summary

Constructors

Constructor and Description
NNStorage(Configuration conf, Collection<URI> imageDirs, Collection<URI> editsDirs) Construct the NNStorage.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
String	determineClusterId() try to find current cluster id in the VERSION files returns first cluster id found in any VERSION file null in case none found
void	format()
void	format(NamespaceInfo nsInfo) Format all available storage directories.
String	getBlockPoolID()
static String	getCheckpointImageFileName(long txid)
static String	getFinalizedEditsFileName(long startTxId, long endTxId)
File	getFsImageName(long txid) Return the name of the image file.
File[]	getFsImageNameCheckpoint(long txid) Return the name of the image file that is uploaded by periodic checkpointing
File	getHighestFsImageName()
static String	getImageFileName(long txid)
static String	getInProgressEditsFileName(long startTxId)
long	getMostRecentCheckpointTxId()
NamespaceInfo	getNamespaceInfo()
static String	getTemporaryEditsFileName(long startTxId, long endTxId, long timestamp)
boolean	isPreUpgradableLayout(Storage.StorageDirectory sd) Return true if the layout of the given storage directory is from a version of Hadoop prior to the introduction of the "current" and "previous" directories which allow upgrade and rollback.
static String	newClusterID() Generate new clusterID.
static NamespaceInfo	newNamespaceInfo()
void	reportErrorOnFile(File f) Report that an IOE has occurred on some file which may or may not be within one of the NN image storage directories.
protected void	setFieldsFromProperties(Properties props, Storage.StorageDirectory sd) Get common storage fields.
protected void	setPropertiesFromFields(Properties props, Storage.StorageDirectory sd) Write version file into the storage directory.
void	writeTransactionIdFileToStorage(long txid) Write a small file in all available storage directories that indicates that the namespace has reached some given transaction ID.

Methods inherited from class org.apache.hadoop.hdfs.server.common.Storage

addStorageDir, checkVersionUpgradable, confirmFormat, deleteDir, dirIterable, dirIterator, dirIterator, getBuildVersion, getFiles, getNumStorageDirs, getRegistrationID, getSingularStorageDir, getStorageDir, is203LayoutVersion, listStorageDirectories, readPreviousVersionProperties, readProperties, readPropertiesFile, rename, setClusterId, setcTime, setLayoutVersion, setNamespaceID, setStorageType, unlockAll, writeAll, writeProperties, writeProperties

Methods inherited from class org.apache.hadoop.hdfs.server.common.StorageInfo

getClusterID, getCTime, getLayoutVersion, getNamespaceID, setStorageInfo, toColonSeparatedString, toString, versionSupportsFederation

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

blockpoolID

protected String blockpoolID

mostRecentCheckpointTxId

protected long mostRecentCheckpointTxId

TxId of the last transaction that was included in the most recent fsimage file. This does not include any transactions that have since been written to the edit log.

removedStorageDirs

protected final List<Storage.StorageDirectory> removedStorageDirs

list of failed (and thus removed) storages

Constructor Detail

NNStorage

public NNStorage(Configuration conf,
         Collection<URI> imageDirs,
         Collection<URI> editsDirs)
          throws IOException

Construct the NNStorage.

Parameters:

conf - Namenode configuration.

imageDirs - Directories the image can be stored in.

editsDirs - Directories the editlog can be stored in.

Throws:

IOException - if any directories are inaccessible.

Method Detail

isPreUpgradableLayout

public boolean isPreUpgradableLayout(Storage.StorageDirectory sd)
                              throws IOException

Description copied from class: Storage

Return true if the layout of the given storage directory is from a version of Hadoop prior to the introduction of the "current" and "previous" directories which allow upgrade and rollback.

Specified by:

isPreUpgradableLayout in class Storage

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getMostRecentCheckpointTxId

public long getMostRecentCheckpointTxId()

Returns:

the transaction ID of the last checkpoint.

writeTransactionIdFileToStorage

public void writeTransactionIdFileToStorage(long txid)

Write a small file in all available storage directories that indicates that the namespace has reached some given transaction ID. This is used when the image is loaded to avoid accidental rollbacks in the case where an edit log is fully deleted but there is no checkpoint. See TestNameEditsConfigs.testNameEditsConfigsFailure()

Parameters:

txid - the txid that has been reached

getFsImageNameCheckpoint

public File[] getFsImageNameCheckpoint(long txid)

Return the name of the image file that is uploaded by periodic checkpointing

Returns:

List of filenames to save checkpoints to.

getFsImageName

public File getFsImageName(long txid)

Return the name of the image file.

Returns:

The name of the first image file.

getHighestFsImageName

public File getHighestFsImageName()

format

public void format(NamespaceInfo nsInfo)
            throws IOException

Format all available storage directories.

Throws:

IOException

newNamespaceInfo

public static NamespaceInfo newNamespaceInfo()
                                      throws UnknownHostException

Throws:

UnknownHostException

format

public void format()
            throws IOException

Throws:

IOException

setFieldsFromProperties

protected void setFieldsFromProperties(Properties props,
                           Storage.StorageDirectory sd)
                                throws IOException

Description copied from class: Storage

Get common storage fields. Should be overloaded if additional fields need to be get.

Overrides:

setFieldsFromProperties in class Storage

Throws:

IOException

setPropertiesFromFields

protected void setPropertiesFromFields(Properties props,
                           Storage.StorageDirectory sd)
                                throws IOException

Write version file into the storage directory. The version file should always be written last. Missing or corrupted version file indicates that the checkpoint is not valid.

Overrides:

setPropertiesFromFields in class Storage

Parameters:

sd - storage directory

props - the Properties object to write into

Throws:

IOException

getCheckpointImageFileName

public static String getCheckpointImageFileName(long txid)

getImageFileName

public static String getImageFileName(long txid)

getInProgressEditsFileName

public static String getInProgressEditsFileName(long startTxId)

getFinalizedEditsFileName

public static String getFinalizedEditsFileName(long startTxId,
                               long endTxId)

getTemporaryEditsFileName

public static String getTemporaryEditsFileName(long startTxId,
                               long endTxId,
                               long timestamp)

reportErrorOnFile

public void reportErrorOnFile(File f)

Report that an IOE has occurred on some file which may or may not be within one of the NN image storage directories.

Specified by:

reportErrorOnFile in interface StorageErrorReporter

Parameters:

f - the file which had an error.

newClusterID

public static String newClusterID()

Generate new clusterID. clusterID is a persistent attribute of the cluster. It is generated when the cluster is created and remains the same during the life cycle of the cluster. When a new name node is formated, if this is a new cluster, a new clusterID is geneated and stored. Subsequent name node must be given the same ClusterID during its format to be in the same cluster. When a datanode register it receive the clusterID and stick with it. If at any point, name node or data node tries to join another cluster, it will be rejected.

Returns:

new clusterID

determineClusterId

public String determineClusterId()

try to find current cluster id in the VERSION files returns first cluster id found in any VERSION file null in case none found

Returns:

clusterId or null in case no cluster id found

getBlockPoolID

public String getBlockPoolID()

getNamespaceInfo

public NamespaceInfo getNamespaceInfo()