org.apache.hadoop.fs

Class ChecksumFs

java.lang.Object

org.apache.hadoop.fs.AbstractFileSystem

org.apache.hadoop.fs.FilterFs

org.apache.hadoop.fs.ChecksumFs

Direct Known Subclasses:

LocalFs

@InterfaceAudience.Private
@InterfaceStability.Evolving
public abstract class ChecksumFs
extends FilterFs

Abstract Checksumed Fs. It provide a basic implementation of a Checksumed Fs, which creates a checksum file for each raw file. It generates & verifies checksums at the client side.

Field Summary

Fields inherited from class org.apache.hadoop.fs.AbstractFileSystem

statistics

Constructor Summary

Constructors

Constructor and Description
ChecksumFs(AbstractFileSystem theFs)

Method Summary

Methods

Modifier and Type	Method and Description
FSDataOutputStream	createInternal(Path f, EnumSet<CreateFlag> createFlag, FsPermission absolutePermission, int bufferSize, short replication, long blockSize, Progressable progress, Options.ChecksumOpt checksumOpt, boolean createParent) The specification of this method matches that of AbstractFileSystem.create(Path, EnumSet, Options.CreateOpts...) except that the opts have been declared explicitly.
boolean	delete(Path f, boolean recursive) Implement the delete(Path, boolean) in checksum file system.
static double	getApproxChkSumLength(long size)
int	getBytesPerSum() Return the bytes Per Checksum.
Path	getChecksumFile(Path file) Return the name of the checksum file associated with a file.
long	getChecksumFileLength(Path file, long fileSize) Return the length of the checksum file given the size of the actual file.
static long	getChecksumLength(long size, int bytesPerSum) Calculated the length of the checksum file in bytes.
AbstractFileSystem	getRawFs() get the raw file system.
static boolean	isChecksumFile(Path file) Return true iff file is a checksum file name.
FileStatus[]	listStatus(Path f) The specification of this method matches that of FileContext.Util.listStatus(Path) except that Path f must be for this file system.
FSDataInputStream	open(Path f, int bufferSize) Opens an FSDataInputStream at the indicated Path.
void	renameInternal(Path src, Path dst) Rename files/dirs.
boolean	reportChecksumFailure(Path f, FSDataInputStream in, long inPos, FSDataInputStream sums, long sumsPos) Report a checksum error to the file system.
boolean	setReplication(Path src, short replication) Set replication for an existing file.
void	setVerifyChecksum(boolean inVerifyChecksum) Set whether to verify checksum.

Methods inherited from class org.apache.hadoop.fs.FilterFs

checkPath, createSymlink, getCanonicalServiceName, getDelegationTokens, getFileBlockLocations, getFileChecksum, getFileLinkStatus, getFileStatus, getFsStatus, getFsStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getMyFs, getServerDefaults, getStatistics, getUri, getUriDefaultPort, getUriPath, isValidName, listCorruptFileBlocks, makeQualified, mkdir, open, renameInternal, resolvePath, setOwner, setPermission, setTimes, supportsSymlinks

Methods inherited from class org.apache.hadoop.fs.AbstractFileSystem

checkScheme, clearStatistics, create, createFileSystem, equals, get, getAllStatistics, getStatistics, hashCode, listLocatedStatus, listStatusIterator, printStatistics, rename

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ChecksumFs

public ChecksumFs(AbstractFileSystem theFs)
           throws IOException,
                  URISyntaxException

Throws:

IOException

URISyntaxException

Method Detail

getApproxChkSumLength

public static double getApproxChkSumLength(long size)

setVerifyChecksum

public void setVerifyChecksum(boolean inVerifyChecksum)

Set whether to verify checksum.

Overrides:

setVerifyChecksum in class FilterFs

getRawFs

public AbstractFileSystem getRawFs()

get the raw file system.

getChecksumFile

public Path getChecksumFile(Path file)

Return the name of the checksum file associated with a file.

isChecksumFile

public static boolean isChecksumFile(Path file)

Return true iff file is a checksum file name.

getChecksumFileLength

public long getChecksumFileLength(Path file,
                         long fileSize)

Return the length of the checksum file given the size of the actual file.

getBytesPerSum

public int getBytesPerSum()

Return the bytes Per Checksum.

open

public FSDataInputStream open(Path f,
                     int bufferSize)
                       throws IOException,
                              UnresolvedLinkException

Opens an FSDataInputStream at the indicated Path.

Overrides:

open in class FilterFs

Parameters:

f - the file name to open

bufferSize - the size of the buffer to be used.

Throws:

IOException

UnresolvedLinkException

getChecksumLength

public static long getChecksumLength(long size,
                     int bytesPerSum)

Calculated the length of the checksum file in bytes.

Parameters:

size - the length of the data file in bytes

bytesPerSum - the number of bytes in a checksum block

Returns:

the number of bytes in the checksum file

createInternal

public FSDataOutputStream createInternal(Path f,
                                EnumSet<CreateFlag> createFlag,
                                FsPermission absolutePermission,
                                int bufferSize,
                                short replication,
                                long blockSize,
                                Progressable progress,
                                Options.ChecksumOpt checksumOpt,
                                boolean createParent)
                                  throws IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of AbstractFileSystem.create(Path, EnumSet, Options.CreateOpts...) except that the opts have been declared explicitly.

Overrides:

createInternal in class FilterFs

Throws:

IOException

setReplication

public boolean setReplication(Path src,
                     short replication)
                       throws IOException,
                              UnresolvedLinkException

Set replication for an existing file. Implement the abstract setReplication of FileSystem

Overrides:

setReplication in class FilterFs

Parameters:

src - file name

replication - new replication

Returns:

true if successful; false if file does not exist or is a directory

Throws:

IOException

UnresolvedLinkException

renameInternal

public void renameInternal(Path src,
                  Path dst)
                    throws IOException,
                           UnresolvedLinkException

Rename files/dirs.

Overrides:

renameInternal in class FilterFs

Throws:

IOException

UnresolvedLinkException

delete

public boolean delete(Path f,
             boolean recursive)
               throws IOException,
                      UnresolvedLinkException

Implement the delete(Path, boolean) in checksum file system.

Overrides:

delete in class FilterFs

Throws:

IOException

UnresolvedLinkException

reportChecksumFailure

public boolean reportChecksumFailure(Path f,
                            FSDataInputStream in,
                            long inPos,
                            FSDataInputStream sums,
                            long sumsPos)

Report a checksum error to the file system.

Parameters:

f - the file name containing the error

in - the stream open on the file

inPos - the position of the beginning of the bad data in the file

sums - the stream open on the checksum file

sumsPos - the position of the beginning of the bad data in the checksum file

Returns:

if retry is neccessary

listStatus

public FileStatus[] listStatus(Path f)
                        throws IOException,
                               UnresolvedLinkException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.Util.listStatus(Path) except that Path f must be for this file system.

Overrides:

listStatus in class FilterFs

Throws:

IOException

UnresolvedLinkException