org.apache.hadoop.hdfs

Class DFSOutputStream

java.lang.Object

java.io.OutputStream

org.apache.hadoop.fs.FSOutputSummer

org.apache.hadoop.hdfs.DFSOutputStream

All Implemented Interfaces:

Closeable, Flushable, AutoCloseable, CanSetDropBehind, Syncable

@InterfaceAudience.Private
public class DFSOutputStream
extends FSOutputSummer
implements Syncable, CanSetDropBehind

DFSOutputStream creates files from a stream of bytes. The client application writes data that is cached internally by this stream. Data is broken up into packets, each packet is typically 64K in size. A packet comprises of chunks. Each chunk is typically 512 bytes and has an associated checksum with it. When a client application fills up the currentPacket, it is enqueued into dataQueue. The DataStreamer thread picks up packets from the dataQueue, sends it to the first datanode in the pipeline and moves it from the dataQueue to the ackQueue. The ResponseProcessor receives acks from the datanodes. When an successful ack for a packet is received from all datanodes, the ResponseProcessor removes the corresponding packet from the ackQueue. In case of error, all outstanding packets and moved from ackQueue. A new pipeline is setup by eliminating the bad datanode from the original pipeline. The DataStreamer now starts sending packets from the dataQueue.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	checkClosed() Check if the implementing OutputStream is closed and should no longer accept writes.
void	close() Closes this output stream and releases any system resources associated with this stream.
int	getCurrentBlockReplication() Note that this is not a public API; use HdfsDataOutputStream.getCurrentBlockReplication() instead.
int	getNumCurrentReplicas() Deprecated. use HdfsDataOutputStream.getCurrentBlockReplication().
DatanodeInfo[]	getPipeline()
void	hflush() Flushes out to all replicas of the block.
void	hsync() Similar to posix fsync, flush out the data in client's user buffer all the way to the disk device (but the disk may have it in its cache).
void	hsync(EnumSet<HdfsDataOutputStream.SyncFlag> syncFlags) The expected semantics is all data have flushed out to all replicas and all replicas have done posix fsync equivalent - ie the OS has flushed it to the disk device (but the disk may have it in its cache).
void	setArtificialSlowdown(long period)
void	setChunksPerPacket(int value)
void	setDropBehind(Boolean dropBehind) Configure whether the stream should drop the cache.
void	sync() Deprecated.
protected void	writeChunk(byte[] b, int offset, int len, byte[] checksum)

Methods inherited from class org.apache.hadoop.fs.FSOutputSummer

convertToByteStream, flushBuffer, flushBuffer, getBufferedDataSize, resetChecksumChunk, write, write

Methods inherited from class java.io.OutputStream

flush, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

checkClosed

protected void checkClosed()
                    throws IOException

Description copied from class: FSOutputSummer

Check if the implementing OutputStream is closed and should no longer accept writes. Implementations should do nothing if this stream is not closed, and should throw an IOException if it is closed.

Specified by:

checkClosed in class FSOutputSummer

Throws:

IOException - if this stream is already closed.

getPipeline

public DatanodeInfo[] getPipeline()

writeChunk

protected void writeChunk(byte[] b,
              int offset,
              int len,
              byte[] checksum)
                   throws IOException

Specified by:

writeChunk in class FSOutputSummer

Throws:

IOException

sync

@Deprecated
public void sync()
          throws IOException

Deprecated.

Specified by:

sync in interface Syncable

Throws:

IOException

See Also:

Syncable.hflush()

hflush

public void hflush()
            throws IOException

Flushes out to all replicas of the block. The data is in the buffers of the DNs but not necessarily in the DN's OS buffers. It is a synchronous operation. When it returns, it guarantees that flushed data become visible to new readers. It is not guaranteed that data has been flushed to persistent store on the datanode. Block allocations are persisted on namenode.

Specified by:

hflush in interface Syncable

Throws:

IOException - if any error occurs

hsync

public void hsync()
           throws IOException

Description copied from interface: Syncable

Similar to posix fsync, flush out the data in client's user buffer all the way to the disk device (but the disk may have it in its cache).

Specified by:

hsync in interface Syncable

Throws:

IOException - if error occurs

hsync

public void hsync(EnumSet<HdfsDataOutputStream.SyncFlag> syncFlags)
           throws IOException

The expected semantics is all data have flushed out to all replicas and all replicas have done posix fsync equivalent - ie the OS has flushed it to the disk device (but the disk may have it in its cache). Note that only the current block is flushed to the disk device. To guarantee durable sync across block boundaries the stream should be created with CreateFlag.SYNC_BLOCK.

Parameters:

syncFlags - Indicate the semantic of the sync. Currently used to specify whether or not to update the block length in NameNode.

Throws:

IOException

getNumCurrentReplicas

@Deprecated
public int getNumCurrentReplicas()
                          throws IOException

Deprecated. use HdfsDataOutputStream.getCurrentBlockReplication().

Throws:

IOException

getCurrentBlockReplication

public int getCurrentBlockReplication()
                               throws IOException

Note that this is not a public API; use HdfsDataOutputStream.getCurrentBlockReplication() instead.

Returns:

the number of valid replicas of the current block

Throws:

IOException

close

public void close()
           throws IOException

Closes this output stream and releases any system resources associated with this stream.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class OutputStream

Throws:

IOException

setArtificialSlowdown

public void setArtificialSlowdown(long period)

setChunksPerPacket

public void setChunksPerPacket(int value)

setDropBehind

public void setDropBehind(Boolean dropBehind)
                   throws IOException

Description copied from interface: CanSetDropBehind

Configure whether the stream should drop the cache.

Specified by:

setDropBehind in interface CanSetDropBehind

Parameters:

dropBehind - Whether to drop the cache. null means to use the default value.

Throws:

IOException - If there was an error changing the dropBehind setting. UnsupportedOperationException If this stream doesn't support setting the drop-behind.