org.apache.hadoop.hdfs

Class DFSInputStream

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.FSInputStream

org.apache.hadoop.hdfs.DFSInputStream

All Implemented Interfaces:

Closeable, AutoCloseable, ByteBufferReadable, CanSetDropBehind, CanSetReadahead, PositionedReadable, Seekable

@InterfaceAudience.Private
public class DFSInputStream
extends FSInputStream
implements ByteBufferReadable, CanSetDropBehind, CanSetReadahead

DFSInputStream provides bytes from a named file. It handles negotiation of the namenode and various datanodes as necessary.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DFSInputStream.ReadStatistics

Method Summary

Methods

Modifier and Type	Method and Description
int	available() Return the size of the remaining available bytes if the size is less than or equal to Integer.MAX_VALUE, otherwise, return Integer.MAX_VALUE.
void	close() Close it down!
List<LocatedBlock>	getAllBlocks() Return collection of blocks that has already been located.
protected BlockReader	getBlockReader(InetSocketAddress dnAddr, DatanodeInfo chosenNode, String file, ExtendedBlock block, Token<BlockTokenIdentifier> blockToken, long startOffset, long len, int bufferSize, boolean verifyChecksum, String clientName) Retrieve a BlockReader suitable for reading.
ExtendedBlock	getCurrentBlock() Returns the block containing the target position.
DatanodeInfo	getCurrentDatanode() Returns the datanode from which the stream is currently reading.
long	getFileLength()
long	getPos() Return the current offset from the start of the file
DFSInputStream.ReadStatistics	getReadStatistics() Get statistics about the reads which this DFSInputStream has done.
void	mark(int readLimit)
boolean	markSupported() We definitely don't support marks
int	read()
int	read(byte[] buf, int off, int len) Read the entire buffer.
int	read(ByteBuffer buf) Reads up to buf.remaining() bytes into buf.
int	read(long position, byte[] buffer, int offset, int length) Read bytes starting from the specified position.
void	reset()
void	seek(long targetPos) Seek to a new arbitrary location
boolean	seekToNewSource(long targetPos) Seek to given position on a node other than the current node.
void	setDropBehind(Boolean dropBehind) Configure whether the stream should drop the cache.
void	setReadahead(Long readahead) Set the readahead on this stream.
long	skip(long n)

Methods inherited from class org.apache.hadoop.fs.FSInputStream

readFully, readFully

Methods inherited from class java.io.InputStream

read

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getFileLength

public long getFileLength()

getCurrentDatanode

public DatanodeInfo getCurrentDatanode()

Returns the datanode from which the stream is currently reading.

getCurrentBlock

public ExtendedBlock getCurrentBlock()

Returns the block containing the target position.

getAllBlocks

public List<LocatedBlock> getAllBlocks()
                                throws IOException

Return collection of blocks that has already been located.

Throws:

IOException

close

public void close()
           throws IOException

Close it down!

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class InputStream

Throws:

IOException

read

public int read()
         throws IOException

Specified by:

read in class InputStream

Throws:

IOException

read

public int read(byte[] buf,
       int off,
       int len)
         throws IOException

Read the entire buffer.

Overrides:

read in class InputStream

Throws:

IOException

read

public int read(ByteBuffer buf)
         throws IOException

Description copied from interface: ByteBufferReadable

Reads up to buf.remaining() bytes into buf. Callers should use buf.limit(..) to control the size of the desired read.

After a successful call, buf.position() and buf.limit() should be unchanged, and therefore any data can be immediately read from buf. buf.mark() may be cleared or updated.

In the case of an exception, the values of buf.position() and buf.limit() are undefined, and callers should be prepared to recover from this eventuality.

Many implementations will throw UnsupportedOperationException, so callers that are not confident in support for this method from the underlying filesystem should be prepared to handle that exception.

Implementations should treat 0-length requests as legitimate, and must not signal an error upon their receipt.

Specified by:

read in interface ByteBufferReadable

Parameters:

buf - the ByteBuffer to receive the results of the read operation. Up to buf.limit() - buf.position() bytes may be read.

Returns:

the number of bytes available to read from buf

Throws:

IOException - if there is some error performing the read

getBlockReader

protected BlockReader getBlockReader(InetSocketAddress dnAddr,
                         DatanodeInfo chosenNode,
                         String file,
                         ExtendedBlock block,
                         Token<BlockTokenIdentifier> blockToken,
                         long startOffset,
                         long len,
                         int bufferSize,
                         boolean verifyChecksum,
                         String clientName)
                              throws IOException

Retrieve a BlockReader suitable for reading. This method will reuse the cached connection to the DN if appropriate. Otherwise, it will create a new connection. Throwing an IOException from this method is basically equivalent to declaring the DataNode bad, so we try to connect a lot of different ways before doing that.

Parameters:

dnAddr - Address of the datanode

chosenNode - Chosen datanode information

file - File location

block - The Block object

blockToken - The access token for security

startOffset - The read offset, relative to block head

len - The number of bytes to read

bufferSize - The IO buffer size (not the client buffer size)

verifyChecksum - Whether to verify checksum

clientName - Client name

Returns:

New BlockReader instance

Throws:

IOException

read

public int read(long position,
       byte[] buffer,
       int offset,
       int length)
         throws IOException

Read bytes starting from the specified position.

Specified by:

read in interface PositionedReadable

Overrides:

read in class FSInputStream

Parameters:

position - start read from this position

buffer - read buffer

offset - offset into buffer

length - number of bytes to read

Returns:

actual number of bytes read

Throws:

IOException

skip

public long skip(long n)
          throws IOException

Overrides:

skip in class InputStream

Throws:

IOException

seek

public void seek(long targetPos)
          throws IOException

Seek to a new arbitrary location

Specified by:

seek in interface Seekable

Specified by:

seek in class FSInputStream

Throws:

IOException

seekToNewSource

public boolean seekToNewSource(long targetPos)
                        throws IOException

Seek to given position on a node other than the current node. If a node other than the current node is found, then returns true. If another node could not be found, then returns false.

Specified by:

seekToNewSource in interface Seekable

Specified by:

seekToNewSource in class FSInputStream

Throws:

IOException

getPos

public long getPos()
            throws IOException

Description copied from class: FSInputStream

Return the current offset from the start of the file

Specified by:

getPos in interface Seekable

Specified by:

getPos in class FSInputStream

Throws:

IOException

available

public int available()
              throws IOException

Return the size of the remaining available bytes if the size is less than or equal to Integer.MAX_VALUE, otherwise, return Integer.MAX_VALUE.

Overrides:

available in class InputStream

Throws:

IOException

markSupported

public boolean markSupported()

We definitely don't support marks

Overrides:

markSupported in class InputStream

mark

public void mark(int readLimit)

Overrides:

mark in class InputStream

reset

public void reset()
           throws IOException

Overrides:

reset in class InputStream

Throws:

IOException

getReadStatistics

public DFSInputStream.ReadStatistics getReadStatistics()

Get statistics about the reads which this DFSInputStream has done.

setReadahead

public void setReadahead(Long readahead)
                  throws IOException

Description copied from interface: CanSetReadahead

Set the readahead on this stream.

Specified by:

setReadahead in interface CanSetReadahead

Parameters:

readahead - The readahead to use. null means to use the default.

Throws:

IOException - If there was an error changing the dropBehind setting. UnsupportedOperationException If this stream doesn't support setting readahead.

setDropBehind

public void setDropBehind(Boolean dropBehind)
                   throws IOException

Description copied from interface: CanSetDropBehind

Configure whether the stream should drop the cache.

Specified by:

setDropBehind in interface CanSetDropBehind

Parameters:

dropBehind - Whether to drop the cache. null means to use the default value.

Throws:

IOException - If there was an error changing the dropBehind setting. UnsupportedOperationException If this stream doesn't support setting the drop-behind.