org.apache.hadoop.hdfs.server.datanode

Class DataNode

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.hdfs.server.datanode.DataNode

All Implemented Interfaces:

Configurable, ClientDatanodeProtocol, DataNodeMXBean, InterDatanodeProtocol

@InterfaceAudience.Private
public class DataNode
extends Configured
implements InterDatanodeProtocol, ClientDatanodeProtocol, DataNodeMXBean

DataNode is a class (and program) that stores a set of blocks for a DFS deployment. A single deployment can have one or many DataNodes. Each DataNode communicates regularly with a single NameNode. It also communicates with client code and other DataNodes from time to time. DataNodes store a series of named blocks. The DataNode allows client code to read these blocks, or to write new block data. The DataNode may also, in response to instructions from its NameNode, delete blocks or copy blocks to/from other DataNodes. The DataNode maintains just one critical table: block-> stream of bytes (of BLOCK_SIZE or less) This info is stored on a local disk. The DataNode reports the table's contents to the NameNode upon startup and every so often afterwards. DataNodes spend their lives in an endless loop of asking the NameNode for something to do. A NameNode cannot connect to a DataNode directly; a NameNode simply returns values from functions invoked by a DataNode. DataNodes maintain an open server socket so that client code or other DataNodes can read/write data. The host/port for this server is reported to the NameNode, which then sends that information to clients or other DataNodes that might be interested.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DataNode.ShortCircuitFdsUnsupportedException
static class	DataNode.ShortCircuitFdsVersionException

Field Summary

Fields

Modifier and Type	Field and Description
static String	DN_CLIENTTRACE_FORMAT
static String	EMPTY_DEL_HINT
RPC.Server	ipcServer
static org.apache.commons.logging.Log	LOG

Fields inherited from interface org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol

versionID

Fields inherited from interface org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol

versionID

Method Summary

Methods

Modifier and Type	Method and Description
void	checkDiskError() Check if there is a disk failure and if so, handle the error
protected void	checkDiskError(Exception e) Check if there is no space in disk
void	clearAllBlockSecretKeys()
static DataNode	createDataNode(String[] args, Configuration conf) Instantiate & Start a single datanode daemon and wait for it to finish.
static DataNode	createDataNode(String[] args, Configuration conf, SecureDataNodeStarter.SecureResources resources) Instantiate & Start a single datanode daemon and wait for it to finish.
static InterDatanodeProtocol	createInterDataNodeProtocolProxy(DatanodeID datanodeid, Configuration conf, int socketTimeout, boolean connectToDnViaHostname)
static InetSocketAddress	createSocketAddr(String target) Deprecated.
void	deleteBlockPool(String blockPoolId, boolean force) Delete the block pool directory.
DatanodeProtocolClientSideTranslatorPB	getActiveNamenodeForBP(String bpid) Get the NameNode corresponding to the given block pool.
Long	getBalancerBandwidth() Get current value of the max balancer bandwidth in bytes per second.
BlockLocalPathInfo	getBlockLocalPathInfo(ExtendedBlock block, Token<BlockTokenIdentifier> token) Retrieves the path names of the block file and metadata file stored on the local file system.
DataBlockScanner	getBlockScanner()
String	getClusterId() Gets the cluster id.
DatanodeID	getDatanodeId()
String	getDisplayName()
DNConf	getDnConf()
DatanodeRegistration	getDNRegistrationForBP(String bpid) get BP registration by blockPool id
HdfsBlocksMetadata	getHdfsBlocksMetadata(List<ExtendedBlock> blocks, List<Token<BlockTokenIdentifier>> tokens) Retrieves volume location information about a list of blocks on a datanode.
String	getHttpPort() Gets the http port.
static InetSocketAddress	getInfoAddr(Configuration conf) Determine the http server's effective addr
int	getInfoPort()
int	getInfoSecurePort()
int	getIpcPort()
String	getNamenodeAddresses() Returned information is a JSON representation of a map with name node host name as the key and block pool Id as the value.
long	getReplicaVisibleLength(ExtendedBlock block) Return the visible length of a replica.
String	getRpcPort() Gets the rpc port.
String	getVersion() Gets the version of Hadoop.
String	getVolumeInfo() Returned information is a JSON representation of a map with volume name as the key and value is a map of volume attribute keys to its values
int	getXceiverCount() Number of concurrent xceivers per node.
InetSocketAddress	getXferAddress() NB: The datanode can perform data transfer on the streaming address however clients are given the IPC IP address for data transfer, and that may be a different address.
int	getXferPort()
ReplicaRecoveryInfo	initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock) Initialize a replica recovery.
static DataNode	instantiateDataNode(String[] args, Configuration conf) Instantiate a single datanode object.
static DataNode	instantiateDataNode(String[] args, Configuration conf, SecureDataNodeStarter.SecureResources resources) Instantiate a single datanode object, along with its secure resources.
boolean	isBPServiceAlive(String bpid)
boolean	isConnectedToNN(InetSocketAddress addr)
boolean	isDatanodeFullyStarted() A datanode is considered to be fully started if all the BP threads are alive and all the block pools are initialized.
boolean	isDatanodeUp() A data node is considered to be up if one of the bp services is up
static void	main(String[] args)
protected Socket	newSocket() Creates either NIO or regular depending on socketWriteTimeout.
void	notifyNamenodeDeletedBlock(ExtendedBlock block) Notify the corresponding namenode to delete the block.
protected void	notifyNamenodeReceivedBlock(ExtendedBlock block, String delHint)
protected void	notifyNamenodeReceivingBlock(ExtendedBlock block)
Daemon	recoverBlocks(String who, Collection<BlockRecoveryCommand.RecoveringBlock> blocks)
void	refreshNamenodes() Refresh the list of federated namenodes from updated configuration Adds new namenodes and stops the deleted namenodes.
void	refreshNamenodes(Configuration conf)
void	reportBadBlocks(ExtendedBlock block) Report a bad block which is hosted on the local DN.
void	reportRemoteBadBlock(DatanodeInfo srcDataNode, ExtendedBlock block) Report a bad block on another DN (eg if we received a corrupt replica from a remote host).
void	runDatanodeDaemon() Start a single datanode daemon and wait for it to finish.
void	scheduleAllBlockReport(long delay) This methods arranges for the data node to send the block report at the next heartbeat.
static void	secureMain(String[] args, SecureDataNodeStarter.SecureResources resources)
static void	setNewStorageID(DatanodeID dnId)
void	shutdown() Shut down this instance of the datanode.
String	toString()
String	updateReplicaUnderRecovery(ExtendedBlock oldBlock, long recoveryId, long newLength) Update replica with the new generation stamp and length.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

DN_CLIENTTRACE_FORMAT

public static final String DN_CLIENTTRACE_FORMAT

See Also:

Constant Field Values

EMPTY_DEL_HINT

public static final String EMPTY_DEL_HINT

See Also:

Constant Field Values

ipcServer

public RPC.Server ipcServer

Method Detail

createSocketAddr

@Deprecated
public static InetSocketAddress createSocketAddr(String target)

Deprecated.

Use NetUtils.createSocketAddr(String) instead.

notifyNamenodeReceivedBlock

protected void notifyNamenodeReceivedBlock(ExtendedBlock block,
                               String delHint)

notifyNamenodeReceivingBlock

protected void notifyNamenodeReceivingBlock(ExtendedBlock block)

notifyNamenodeDeletedBlock

public void notifyNamenodeDeletedBlock(ExtendedBlock block)

Notify the corresponding namenode to delete the block.

reportBadBlocks

public void reportBadBlocks(ExtendedBlock block)
                     throws IOException

Report a bad block which is hosted on the local DN.

Throws:

IOException

reportRemoteBadBlock

public void reportRemoteBadBlock(DatanodeInfo srcDataNode,
                        ExtendedBlock block)
                          throws IOException

Report a bad block on another DN (eg if we received a corrupt replica from a remote host).

Parameters:

srcDataNode - the DN hosting the bad block

block - the block itself

Throws:

IOException

getInfoAddr

public static InetSocketAddress getInfoAddr(Configuration conf)

Determine the http server's effective addr

getXferPort

public int getXferPort()

getDisplayName

public String getDisplayName()

Returns:

name useful for logging

getXferAddress

public InetSocketAddress getXferAddress()

NB: The datanode can perform data transfer on the streaming address however clients are given the IPC IP address for data transfer, and that may be a different address.

Returns:

socket address for data transfer

getIpcPort

public int getIpcPort()

Returns:

the datanode's IPC port

getDNRegistrationForBP

public DatanodeRegistration getDNRegistrationForBP(String bpid)
                                            throws IOException

get BP registration by blockPool id

Parameters:

bpid -

Returns:

BP registration object

Throws:

IOException

newSocket

protected Socket newSocket()
                    throws IOException

Creates either NIO or regular depending on socketWriteTimeout.

Throws:

IOException

createInterDataNodeProtocolProxy

public static InterDatanodeProtocol createInterDataNodeProtocolProxy(DatanodeID datanodeid,
                                                     Configuration conf,
                                                     int socketTimeout,
                                                     boolean connectToDnViaHostname)
                                                              throws IOException

Throws:

IOException

setNewStorageID

public static void setNewStorageID(DatanodeID dnId)

getBlockLocalPathInfo

public BlockLocalPathInfo getBlockLocalPathInfo(ExtendedBlock block,
                                       Token<BlockTokenIdentifier> token)
                                         throws IOException

Description copied from interface: ClientDatanodeProtocol

Retrieves the path names of the block file and metadata file stored on the local file system. In order for this method to work, one of the following should be satisfied:

• The client user must be configured at the datanode to be able to use this method.
• When security is enabled, kerberos authentication must be used to connect to the datanode.

Specified by:

getBlockLocalPathInfo in interface ClientDatanodeProtocol

Parameters:

block - the specified block on the local datanode

token - the block access token.

Returns:

the BlockLocalPathInfo of a block

Throws:

IOException - on error

getHdfsBlocksMetadata

public HdfsBlocksMetadata getHdfsBlocksMetadata(List<ExtendedBlock> blocks,
                                       List<Token<BlockTokenIdentifier>> tokens)
                                         throws IOException,
                                                UnsupportedOperationException

Description copied from interface: ClientDatanodeProtocol

Retrieves volume location information about a list of blocks on a datanode. This is in the form of an opaque VolumeId for each configured data directory, which is not guaranteed to be the same across DN restarts.

Specified by:

getHdfsBlocksMetadata in interface ClientDatanodeProtocol

Parameters:

blocks - list of blocks on the local datanode

tokens - block access tokens corresponding to the requested blocks

Returns:

an HdfsBlocksMetadata that associates ExtendedBlocks with data directories

Throws:

IOException - if datanode is unreachable, or replica is not found on datanode

UnsupportedOperationException

shutdown

public void shutdown()

Shut down this instance of the datanode. Returns only after shutdown is complete. This method can only be called by the offerService thread. Otherwise, deadlock might occur.

checkDiskError

protected void checkDiskError(Exception e)
                       throws IOException

Check if there is no space in disk

Parameters:

e - that caused this checkDiskError call

Throws:

IOException

checkDiskError

public void checkDiskError()

Check if there is a disk failure and if so, handle the error

getXceiverCount

public int getXceiverCount()

Number of concurrent xceivers per node.

Specified by:

getXceiverCount in interface DataNodeMXBean

runDatanodeDaemon

public void runDatanodeDaemon()
                       throws IOException

Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting.

Throws:

IOException

isDatanodeUp

public boolean isDatanodeUp()

A data node is considered to be up if one of the bp services is up

instantiateDataNode

public static DataNode instantiateDataNode(String[] args,
                           Configuration conf)
                                    throws IOException

Instantiate a single datanode object. This must be run by invoking runDatanodeDaemon() subsequently.

Throws:

IOException

instantiateDataNode

public static DataNode instantiateDataNode(String[] args,
                           Configuration conf,
                           SecureDataNodeStarter.SecureResources resources)
                                    throws IOException

Instantiate a single datanode object, along with its secure resources. This must be run by invokingrunDatanodeDaemon() subsequently.

Throws:

IOException

createDataNode

public static DataNode createDataNode(String[] args,
                      Configuration conf)
                               throws IOException

Instantiate & Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting.

Throws:

IOException

createDataNode

@InterfaceAudience.Private
public static DataNode createDataNode(String[] args,
                                                Configuration conf,
                                                SecureDataNodeStarter.SecureResources resources)
                               throws IOException

Instantiate & Start a single datanode daemon and wait for it to finish. If this thread is specifically interrupted, it will stop waiting.

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object

scheduleAllBlockReport

public void scheduleAllBlockReport(long delay)

This methods arranges for the data node to send the block report at the next heartbeat.

getBlockScanner

public DataBlockScanner getBlockScanner()

Returns:

the block scanner.

secureMain

public static void secureMain(String[] args,
              SecureDataNodeStarter.SecureResources resources)

main

public static void main(String[] args)

recoverBlocks

public Daemon recoverBlocks(String who,
                   Collection<BlockRecoveryCommand.RecoveringBlock> blocks)

initReplicaRecovery

public ReplicaRecoveryInfo initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock)
                                        throws IOException

Description copied from interface: InterDatanodeProtocol

Initialize a replica recovery.

Specified by:

initReplicaRecovery in interface InterDatanodeProtocol

Returns:

actual state of the replica on this data-node or null if data-node does not have the replica.

Throws:

IOException

updateReplicaUnderRecovery

public String updateReplicaUnderRecovery(ExtendedBlock oldBlock,
                                long recoveryId,
                                long newLength)
                                  throws IOException

Update replica with the new generation stamp and length.

Specified by:

updateReplicaUnderRecovery in interface InterDatanodeProtocol

Throws:

IOException

getActiveNamenodeForBP

public DatanodeProtocolClientSideTranslatorPB getActiveNamenodeForBP(String bpid)
                                                              throws IOException

Get the NameNode corresponding to the given block pool.

Parameters:

bpid - Block pool Id

Returns:

Namenode corresponding to the bpid

Throws:

IOException - if unable to get the corresponding NameNode

getReplicaVisibleLength

public long getReplicaVisibleLength(ExtendedBlock block)
                             throws IOException

Description copied from interface: ClientDatanodeProtocol

Return the visible length of a replica.

Specified by:

getReplicaVisibleLength in interface ClientDatanodeProtocol

Throws:

IOException

getVersion

public String getVersion()

Description copied from interface: DataNodeMXBean

Gets the version of Hadoop.

Specified by:

getVersion in interface DataNodeMXBean

Returns:

the version of Hadoop

getRpcPort

public String getRpcPort()

Description copied from interface: DataNodeMXBean

Gets the rpc port.

Specified by:

getRpcPort in interface DataNodeMXBean

Returns:

the rpc port

getHttpPort

public String getHttpPort()

Description copied from interface: DataNodeMXBean

Gets the http port.

Specified by:

getHttpPort in interface DataNodeMXBean

Returns:

the http port

getInfoPort

public int getInfoPort()

Returns:

the datanode's http port

getInfoSecurePort

public int getInfoSecurePort()

Returns:

the datanode's https port

getNamenodeAddresses

public String getNamenodeAddresses()

Returned information is a JSON representation of a map with name node host name as the key and block pool Id as the value. Note that, if there are multiple NNs in an NA nameservice, a given block pool may be represented twice.

Specified by:

getNamenodeAddresses in interface DataNodeMXBean

Returns:

the namenode IP addresses that the datanode is talking to

getVolumeInfo

public String getVolumeInfo()

Returned information is a JSON representation of a map with volume name as the key and value is a map of volume attribute keys to its values

Specified by:

getVolumeInfo in interface DataNodeMXBean

Returns:

the volume info

getClusterId

public String getClusterId()

Description copied from interface: DataNodeMXBean

Gets the cluster id.

Specified by:

getClusterId in interface DataNodeMXBean

Returns:

the cluster id

refreshNamenodes

public void refreshNamenodes(Configuration conf)
                      throws IOException

Throws:

IOException

refreshNamenodes

public void refreshNamenodes()
                      throws IOException

Description copied from interface: ClientDatanodeProtocol

Refresh the list of federated namenodes from updated configuration Adds new namenodes and stops the deleted namenodes.

Specified by:

refreshNamenodes in interface ClientDatanodeProtocol

Throws:

IOException - on error

deleteBlockPool

public void deleteBlockPool(String blockPoolId,
                   boolean force)
                     throws IOException

Description copied from interface: ClientDatanodeProtocol

Delete the block pool directory. If force is false it is deleted only if it is empty, otherwise it is deleted along with its contents.

Specified by:

deleteBlockPool in interface ClientDatanodeProtocol

Parameters:

blockPoolId - Blockpool id to be deleted.

force - If false blockpool directory is deleted only if it is empty i.e. if it doesn't contain any block files, otherwise it is deleted along with its contents.

Throws:

IOException

isConnectedToNN

public boolean isConnectedToNN(InetSocketAddress addr)

Parameters:

addr - rpc address of the namenode

Returns:

true if the datanode is connected to a NameNode at the given address

isBPServiceAlive

public boolean isBPServiceAlive(String bpid)

Parameters:

bpid - block pool Id

Returns:

true - if BPOfferService thread is alive

isDatanodeFullyStarted

public boolean isDatanodeFullyStarted()

A datanode is considered to be fully started if all the BP threads are alive and all the block pools are initialized.

Returns:

true - if the data node is fully started

getDatanodeId

public DatanodeID getDatanodeId()

clearAllBlockSecretKeys

public void clearAllBlockSecretKeys()

getBalancerBandwidth

public Long getBalancerBandwidth()

Get current value of the max balancer bandwidth in bytes per second.

Returns:

bandwidth Blanacer bandwidth in bytes per second for this datanode.

getDnConf

public DNConf getDnConf()