org.apache.hadoop.hdfs.server.datanode.fsdataset

Interface FsDatasetSpi<V extends FsVolumeSpi>

All Superinterfaces:

FSDatasetMBean

@InterfaceAudience.Private
public interface FsDatasetSpi<V extends FsVolumeSpi>
extends FSDatasetMBean

This is a service provider interface for the underlying storage that stores replicas for a data node. The default implementation stores replicas on local drives.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	FsDatasetSpi.Factory<D extends FsDatasetSpi<?>> A factory for creating FsDatasetSpi objects.

Method Summary

Methods

Modifier and Type	Method and Description
void	addBlockPool(String bpid, Configuration conf) add new block pool ID
void	adjustCrcChannelPosition(ExtendedBlock b, ReplicaOutputStreams outs, int checksumSize) Sets the file pointer of the checksum stream so that the last checksum will be overwritten
ReplicaInPipelineInterface	append(ExtendedBlock b, long newGS, long expectedBlockLen) Append to a finalized replica and returns the meta info of the replica
void	checkAndUpdate(String bpid, long blockId, File diskFile, File diskMetaFile, FsVolumeSpi vol) Check whether the in-memory block record matches the block on the disk, and, in case that they are not matched, update the record or mark it as corrupted.
void	checkDataDir() Check if all the data directories are healthy
boolean	contains(ExtendedBlock block) Does the dataset contain the block?
ReplicaInPipelineInterface	convertTemporaryToRbw(ExtendedBlock temporary) Covert a temporary replica to a RBW.
ReplicaInPipelineInterface	createRbw(ExtendedBlock b) Creates a RBW replica and returns the meta info of the replica
RollingLogs	createRollingLogs(String bpid, String prefix) Create rolling logs.
ReplicaInPipelineInterface	createTemporary(ExtendedBlock b) Creates a temporary replica and returns the meta information of the replica
void	deleteBlockPool(String bpid, boolean force) Deletes the block pool directories.
void	finalizeBlock(ExtendedBlock b) Finalizes the block previously opened for writing using writeToBlock.
InputStream	getBlockInputStream(ExtendedBlock b, long seekOffset) Returns an input stream at specified offset of the specified block
BlockLocalPathInfo	getBlockLocalPathInfo(ExtendedBlock b) Get BlockLocalPathInfo for the given block.
String[]	getBlockPoolList()
BlockListAsLongs	getBlockReport(String bpid) Returns the block report - the full list of blocks stored under a block pool
List<Block>	getFinalizedBlocks(String bpid)
HdfsBlocksMetadata	getHdfsBlocksMetadata(List<ExtendedBlock> blocks) Get a HdfsBlocksMetadata corresponding to the list of blocks in blocks.
long	getLength(ExtendedBlock b) Returns the specified block's on-disk length (excluding metadata)
LengthInputStream	getMetaDataInputStream(ExtendedBlock b)
Replica	getReplica(String bpid, long blockId) Deprecated.
String	getReplicaString(String bpid, long blockId)
long	getReplicaVisibleLength(ExtendedBlock block) Get visible length of the specified replica.
Block	getStoredBlock(String bpid, long blkid)
ReplicaInputStreams	getTmpInputStreams(ExtendedBlock b, long blkoff, long ckoff) Returns an input stream at specified offset of the specified block The block is still in the tmp directory and is not finalized
V	getVolume(ExtendedBlock b)
Map<String,Object>	getVolumeInfoMap()
List<V>	getVolumes()
boolean	hasEnoughResource() Checks how many valid storage volumes there are in the DataNode.
ReplicaRecoveryInfo	initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock) Initialize a replica recovery.
void	invalidate(String bpid, Block[] invalidBlks) Invalidates the specified blocks
boolean	isValidBlock(ExtendedBlock b) Is the block valid?
boolean	isValidRbw(ExtendedBlock b) Is the block a valid RBW?
ReplicaInPipelineInterface	recoverAppend(ExtendedBlock b, long newGS, long expectedBlockLen) Recover a failed append to a finalized replica and returns the meta info of the replica
void	recoverClose(ExtendedBlock b, long newGS, long expectedBlockLen) Recover a failed pipeline close It bumps the replica's generation stamp and finalize it if RBW replica
ReplicaInPipelineInterface	recoverRbw(ExtendedBlock b, long newGS, long minBytesRcvd, long maxBytesRcvd) Recovers a RBW replica and returns the meta info of the replica
void	shutdown() Shutdown the FSDataset
void	shutdownBlockPool(String bpid) Shutdown and remove the block pool from underlying storage.
void	unfinalizeBlock(ExtendedBlock b) Unfinalizes the block previously opened for writing using writeToBlock.
String	updateReplicaUnderRecovery(ExtendedBlock oldBlock, long recoveryId, long newLength) Update replica's generation stamp and length and finalize it.

Methods inherited from interface org.apache.hadoop.hdfs.server.datanode.metrics.FSDatasetMBean

getBlockPoolUsed, getCapacity, getDfsUsed, getNumFailedVolumes, getRemaining, getStorageInfo

Method Detail

createRollingLogs

RollingLogs createRollingLogs(String bpid,
                            String prefix)
                              throws IOException

Create rolling logs.

Parameters:

prefix - the prefix of the log names.

Returns:

rolling logs

Throws:

IOException

getVolumes

List<V> getVolumes()

Returns:

a list of volumes.

getVolume

V getVolume(ExtendedBlock b)

Returns:

the volume that contains a replica of the block.

getVolumeInfoMap

Map<String,Object> getVolumeInfoMap()

Returns:

a volume information map (name => info).

getBlockPoolList

String[] getBlockPoolList()

Returns:

a list of block pools.

getFinalizedBlocks

List<Block> getFinalizedBlocks(String bpid)

Returns:

a list of finalized blocks for the given block pool.

checkAndUpdate

void checkAndUpdate(String bpid,
                  long blockId,
                  File diskFile,
                  File diskMetaFile,
                  FsVolumeSpi vol)

Check whether the in-memory block record matches the block on the disk, and, in case that they are not matched, update the record or mark it as corrupted.

getMetaDataInputStream

LengthInputStream getMetaDataInputStream(ExtendedBlock b)
                                         throws IOException

Parameters:

b - - the block

Returns:

a stream if the meta-data of the block exists; otherwise, return null.

Throws:

IOException

getLength

long getLength(ExtendedBlock b)
               throws IOException

Returns the specified block's on-disk length (excluding metadata)

Parameters:

b -

Returns:

the specified block's on-disk length (excluding metadta)

Throws:

IOException

getReplica

@Deprecated
Replica getReplica(String bpid,
                            long blockId)

Deprecated.

Get reference to the replica meta info in the replicasMap. To be called from methods that are synchronized on FSDataset

Parameters:

blockId -

Returns:

replica from the replicas map

getReplicaString

String getReplicaString(String bpid,
                      long blockId)

Returns:

replica meta information

getStoredBlock

Block getStoredBlock(String bpid,
                   long blkid)
                     throws IOException

Returns:

the generation stamp stored with the block.

Throws:

IOException

getBlockInputStream

InputStream getBlockInputStream(ExtendedBlock b,
                              long seekOffset)
                                throws IOException

Returns an input stream at specified offset of the specified block

Parameters:

b -

seekOffset -

Returns:

an input stream to read the contents of the specified block, starting at the offset

Throws:

IOException

getTmpInputStreams

ReplicaInputStreams getTmpInputStreams(ExtendedBlock b,
                                     long blkoff,
                                     long ckoff)
                                       throws IOException

Returns an input stream at specified offset of the specified block The block is still in the tmp directory and is not finalized

Parameters:

b -

blkoff -

ckoff -

Returns:

an input stream to read the contents of the specified block, starting at the offset

Throws:

IOException

createTemporary

ReplicaInPipelineInterface createTemporary(ExtendedBlock b)
                                           throws IOException

Creates a temporary replica and returns the meta information of the replica

Parameters:

b - block

Returns:

the meta info of the replica which is being written to

Throws:

IOException - if an error occurs

createRbw

ReplicaInPipelineInterface createRbw(ExtendedBlock b)
                                     throws IOException

Creates a RBW replica and returns the meta info of the replica

Parameters:

b - block

Returns:

the meta info of the replica which is being written to

Throws:

IOException - if an error occurs

recoverRbw

ReplicaInPipelineInterface recoverRbw(ExtendedBlock b,
                                    long newGS,
                                    long minBytesRcvd,
                                    long maxBytesRcvd)
                                      throws IOException

Recovers a RBW replica and returns the meta info of the replica

Parameters:

b - block

newGS - the new generation stamp for the replica

minBytesRcvd - the minimum number of bytes that the replica could have

maxBytesRcvd - the maximum number of bytes that the replica could have

Returns:

the meta info of the replica which is being written to

Throws:

IOException - if an error occurs

convertTemporaryToRbw

ReplicaInPipelineInterface convertTemporaryToRbw(ExtendedBlock temporary)
                                                 throws IOException

Covert a temporary replica to a RBW.

Parameters:

temporary - the temporary replica being converted

Returns:

the result RBW

Throws:

IOException

append

ReplicaInPipelineInterface append(ExtendedBlock b,
                                long newGS,
                                long expectedBlockLen)
                                  throws IOException

Append to a finalized replica and returns the meta info of the replica

Parameters:

b - block

newGS - the new generation stamp for the replica

expectedBlockLen - the number of bytes the replica is expected to have

Returns:

the meata info of the replica which is being written to

Throws:

IOException

recoverAppend

ReplicaInPipelineInterface recoverAppend(ExtendedBlock b,
                                       long newGS,
                                       long expectedBlockLen)
                                         throws IOException

Recover a failed append to a finalized replica and returns the meta info of the replica

Parameters:

b - block

newGS - the new generation stamp for the replica

expectedBlockLen - the number of bytes the replica is expected to have

Returns:

the meta info of the replica which is being written to

Throws:

IOException

recoverClose

void recoverClose(ExtendedBlock b,
                long newGS,
                long expectedBlockLen)
                  throws IOException

Recover a failed pipeline close It bumps the replica's generation stamp and finalize it if RBW replica

Parameters:

b - block

newGS - the new generation stamp for the replica

expectedBlockLen - the number of bytes the replica is expected to have

Throws:

IOException

finalizeBlock

void finalizeBlock(ExtendedBlock b)
                   throws IOException

Finalizes the block previously opened for writing using writeToBlock. The block size is what is in the parameter b and it must match the amount of data written

Parameters:

b -

Throws:

IOException

unfinalizeBlock

void unfinalizeBlock(ExtendedBlock b)
                     throws IOException

Unfinalizes the block previously opened for writing using writeToBlock. The temporary file associated with this block is deleted.

Parameters:

b -

Throws:

IOException

getBlockReport

BlockListAsLongs getBlockReport(String bpid)

Returns the block report - the full list of blocks stored under a block pool

Parameters:

bpid - Block Pool Id

Returns:

- the block report - the full list of blocks stored

contains

boolean contains(ExtendedBlock block)

Does the dataset contain the block?

isValidBlock

boolean isValidBlock(ExtendedBlock b)

Is the block valid?

Parameters:

b -

Returns:

- true if the specified block is valid

isValidRbw

boolean isValidRbw(ExtendedBlock b)

Is the block a valid RBW?

Parameters:

b -

Returns:

- true if the specified block is a valid RBW

invalidate

void invalidate(String bpid,
              Block[] invalidBlks)
                throws IOException

Invalidates the specified blocks

Parameters:

bpid - Block pool Id

invalidBlks - - the blocks to be invalidated

Throws:

IOException

checkDataDir

void checkDataDir()
                  throws DiskChecker.DiskErrorException

Check if all the data directories are healthy

Throws:

DiskChecker.DiskErrorException

shutdown

void shutdown()

Shutdown the FSDataset

adjustCrcChannelPosition

void adjustCrcChannelPosition(ExtendedBlock b,
                            ReplicaOutputStreams outs,
                            int checksumSize)
                              throws IOException

Sets the file pointer of the checksum stream so that the last checksum will be overwritten

Parameters:

b - block

outs - The streams for the data file and checksum file

checksumSize - number of bytes each checksum has

Throws:

IOException

hasEnoughResource

boolean hasEnoughResource()

Checks how many valid storage volumes there are in the DataNode.

Returns:

true if more than the minimum number of valid volumes are left in the FSDataSet.

getReplicaVisibleLength

long getReplicaVisibleLength(ExtendedBlock block)
                             throws IOException

Get visible length of the specified replica.

Throws:

IOException

initReplicaRecovery

ReplicaRecoveryInfo initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock)
                                        throws IOException

Initialize a replica recovery.

Returns:

actual state of the replica on this data-node or null if data-node does not have the replica.

Throws:

IOException

updateReplicaUnderRecovery

String updateReplicaUnderRecovery(ExtendedBlock oldBlock,
                                long recoveryId,
                                long newLength)
                                  throws IOException

Update replica's generation stamp and length and finalize it.

Returns:

the ID of storage that stores the block

Throws:

IOException

addBlockPool

void addBlockPool(String bpid,
                Configuration conf)
                  throws IOException

add new block pool ID

Parameters:

bpid - Block pool Id

conf - Configuration

Throws:

IOException

shutdownBlockPool

void shutdownBlockPool(String bpid)

Shutdown and remove the block pool from underlying storage.

Parameters:

bpid - Block pool Id to be removed

deleteBlockPool

void deleteBlockPool(String bpid,
                   boolean force)
                     throws IOException

Deletes the block pool directories. If force is false, directories are deleted only if no block files exist for the block pool. If force is true entire directory for the blockpool is deleted along with its contents.

Parameters:

bpid - BlockPool Id to be deleted.

force - If force is false, directories are deleted only if no block files exist for the block pool, otherwise entire directory for the blockpool is deleted along with its contents.

Throws:

IOException

getBlockLocalPathInfo

BlockLocalPathInfo getBlockLocalPathInfo(ExtendedBlock b)
                                         throws IOException

Get BlockLocalPathInfo for the given block.

Throws:

IOException

getHdfsBlocksMetadata

HdfsBlocksMetadata getHdfsBlocksMetadata(List<ExtendedBlock> blocks)
                                         throws IOException

Get a HdfsBlocksMetadata corresponding to the list of blocks in blocks.

Parameters:

blocks - List of blocks for which to return metadata

Returns:

metadata Metadata for the list of blocks

Throws:

IOException