org.apache.hadoop.hdfs

Class HftpFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.hdfs.HftpFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, Configurable, DelegationTokenRenewer.Renewable

Direct Known Subclasses:

HsftpFileSystem

@InterfaceAudience.Private
@InterfaceStability.Evolving
public class HftpFileSystem
extends FileSystem
implements DelegationTokenRenewer.Renewable

An implementation of a protocol for accessing filesystems over HTTP. The following implementation provides a limited, read-only interface to a filesystem over HTTP.

See Also:

ListPathsServlet, FileDataServlet

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HftpFileSystem.TokenManager

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
protected static ThreadLocal<SimpleDateFormat>	df
static String	HFTP_DATE_FORMAT
static String	HFTP_TIMEZONE
protected URI	nnUri
static Text	TOKEN_KIND
protected UserGroupInformation	ugi

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics

Constructor Summary

Constructors

Constructor and Description
HftpFileSystem()

Method Summary

Methods

Modifier and Type	Method and Description
protected String	addDelegationTokenParam(String query)
FSDataOutputStream	append(Path f, int bufferSize, Progressable progress) This optional operation is not yet supported.
protected URI	canonicalizeUri(URI uri) Canonicalize the given URI.
void	close() No more filesystem operations are needed.
FSDataOutputStream	create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
boolean	delete(Path f, boolean recursive) Delete a file.
String	getCanonicalServiceName() See the documentation of for the logic behind selecting the canonical service name.
ContentSummary	getContentSummary(Path f) Return the ContentSummary of a given Path.
static SimpleDateFormat	getDateFormat()
protected int	getDefaultPort() Get the default port for this file system.
Token<?>	getDelegationToken(String renewer) Get a new delegation token for this file system.
FileChecksum	getFileChecksum(Path f) Get the checksum of a file.
FileStatus	getFileStatus(Path f) Return a file status object that represents the path.
protected InetSocketAddress	getNamenodeAddr(URI uri) We generate the address with one of the following ports, in order of preference.
protected URI	getNamenodeUri(URI uri)
protected URL	getNamenodeURL(String path, String query) Return a URL pointing to given path on the namenode.
Token<?>	getRenewToken()
String	getScheme() Return the protocol scheme for the FileSystem.
protected String	getUnderlyingProtocol() Return the underlying protocol that is used to talk to the namenode.
URI	getUri() Returns a URI whose scheme and authority identify this FileSystem.
Path	getWorkingDirectory() Get the current working directory for the given file system
protected void	initDelegationToken()
void	initialize(URI name, Configuration conf) Called after a new FileSystem instance is constructed.
FileStatus[]	listStatus(Path f) List the statuses of the files/directories in the given path if the path is a directory.
boolean	mkdirs(Path f, FsPermission permission) Make the given file and all non-existent parents into directories.
FSDataInputStream	open(Path f, int buffersize) Opens an FSDataInputStream at the indicated Path.
protected HttpURLConnection	openConnection(String path, String query) Open an HTTP connection to the namenode to read file data and metadata.
boolean	rename(Path src, Path dst) Renames Path src to Path dst.
protected Token<DelegationTokenIdentifier>	selectDelegationToken(UserGroupInformation ugi)
<T extends TokenIdentifier> void	setDelegationToken(Token<T> token) Set delegation token.
void	setWorkingDirectory(Path f) Set the current working directory for the given file system.

Methods inherited from class org.apache.hadoop.fs.FileSystem

addDelegationTokens, append, append, cancelDeleteOnExit, checkPath, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAllStatistics, getBlockSize, getCanonicalUri, getChildFileSystems, getDefaultBlockSize, getDefaultBlockSize, getDefaultReplication, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileBlockLocations, getFileLinkStatus, getFileSystemClass, getFSofPath, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getUsed, globStatus, globStatus, isDirectory, isFile, isSymlinksEnabled, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, makeQualified, mkdirs, mkdirs, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, rename, renameSnapshot, resolveLink, resolvePath, setDefaultUri, setDefaultUri, setOwner, setPermission, setReplication, setTimes, setVerifyChecksum, setWriteChecksum, startLocalOutput, supportsSymlinks

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TOKEN_KIND

public static final Text TOKEN_KIND

ugi

protected UserGroupInformation ugi

nnUri

protected URI nnUri

HFTP_TIMEZONE

public static final String HFTP_TIMEZONE

See Also:

Constant Field Values

HFTP_DATE_FORMAT

public static final String HFTP_DATE_FORMAT

See Also:

Constant Field Values

df

protected static final ThreadLocal<SimpleDateFormat> df

Constructor Detail

HftpFileSystem

public HftpFileSystem()

Method Detail

getDateFormat

public static final SimpleDateFormat getDateFormat()

getDefaultPort

protected int getDefaultPort()

Description copied from class: FileSystem

Get the default port for this file system.

Overrides:

getDefaultPort in class FileSystem

Returns:

the default port or 0 if there isn't one

getNamenodeAddr

protected InetSocketAddress getNamenodeAddr(URI uri)

We generate the address with one of the following ports, in order of preference. 1. Port from the hftp URI e.g. hftp://namenode:4000/ will return 4000. 2. Port configured via DFS_NAMENODE_HTTP_PORT_KEY 3. DFS_NAMENODE_HTTP_PORT_DEFAULT i.e. 50070.

Parameters:

uri -

Returns:

getNamenodeUri

protected URI getNamenodeUri(URI uri)

getCanonicalServiceName

public String getCanonicalServiceName()

See the documentation of for the logic behind selecting the canonical service name.

Overrides:

getCanonicalServiceName in class FileSystem

Returns:

See Also:

SecurityUtil.buildDTServiceName(URI, int)

canonicalizeUri

protected URI canonicalizeUri(URI uri)

Description copied from class: FileSystem

Canonicalize the given URI. This is filesystem-dependent, but may for example consist of canonicalizing the hostname using DNS and adding the default port if not specified. The default implementation simply fills in the default port if not specified and if the filesystem has a default port.

Overrides:

canonicalizeUri in class FileSystem

Returns:

URI

See Also:

NetUtils.getCanonicalUri(URI, int)

getScheme

public String getScheme()

Return the protocol scheme for the FileSystem.

Overrides:

getScheme in class FileSystem

Returns:

hftp

initialize

public void initialize(URI name,
              Configuration conf)
                throws IOException

Description copied from class: FileSystem

Called after a new FileSystem instance is constructed.

Overrides:

initialize in class FileSystem

Parameters:

name - a uri whose authority section names the host, port, etc. for this FileSystem

conf - the configuration

Throws:

IOException

initDelegationToken

protected void initDelegationToken()
                            throws IOException

Throws:

IOException

selectDelegationToken

protected Token<DelegationTokenIdentifier> selectDelegationToken(UserGroupInformation ugi)

getRenewToken

public Token<?> getRenewToken()

Specified by:

getRenewToken in interface DelegationTokenRenewer.Renewable

Returns:

the renew token.

getUnderlyingProtocol

protected String getUnderlyingProtocol()

Return the underlying protocol that is used to talk to the namenode.

setDelegationToken

public <T extends TokenIdentifier> void setDelegationToken(Token<T> token)

Description copied from interface: DelegationTokenRenewer.Renewable

Set delegation token.

Specified by:

setDelegationToken in interface DelegationTokenRenewer.Renewable

getDelegationToken

public Token<?> getDelegationToken(String renewer)
                            throws IOException

Description copied from class: FileSystem

Get a new delegation token for this file system. This is an internal method that should have been declared protected but wasn't historically. Callers should use FileSystem.addDelegationTokens(String, Credentials)

Overrides:

getDelegationToken in class FileSystem

Parameters:

renewer - the account name that is allowed to renew the token.

Returns:

a new delegation token

Throws:

IOException

getUri

public URI getUri()

Description copied from class: FileSystem

Returns a URI whose scheme and authority identify this FileSystem.

Specified by:

getUri in class FileSystem

getNamenodeURL

protected URL getNamenodeURL(String path,
                 String query)
                      throws IOException

Return a URL pointing to given path on the namenode.

Parameters:

path - to obtain the URL for

query - string to append to the path

Returns:

namenode URL referring to the given path

Throws:

IOException - on error constructing the URL

openConnection

protected HttpURLConnection openConnection(String path,
                               String query)
                                    throws IOException

Open an HTTP connection to the namenode to read file data and metadata.

Parameters:

path - The path component of the URL

query - The query component of the URL

Throws:

IOException

addDelegationTokenParam

protected String addDelegationTokenParam(String query)
                                  throws IOException

Throws:

IOException

open

public FSDataInputStream open(Path f,
                     int buffersize)
                       throws IOException

Description copied from class: FileSystem

Opens an FSDataInputStream at the indicated Path.

Specified by:

open in class FileSystem

Parameters:

f - the file name to open

buffersize - the size of the buffer to be used.

Throws:

IOException

close

public void close()
           throws IOException

Description copied from class: FileSystem

No more filesystem operations are needed. Will release any held locks.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class FileSystem

Throws:

IOException

listStatus

public FileStatus[] listStatus(Path f)
                        throws IOException

Description copied from class: FileSystem

List the statuses of the files/directories in the given path if the path is a directory.

Specified by:

listStatus in class FileSystem

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws IOException

Description copied from class: FileSystem

Return a file status object that represents the path.

Specified by:

getFileStatus in class FileSystem

Parameters:

f - The path we want information from

Returns:

a FileStatus object

Throws:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

getFileChecksum

public FileChecksum getFileChecksum(Path f)
                             throws IOException

Description copied from class: FileSystem

Get the checksum of a file.

Overrides:

getFileChecksum in class FileSystem

Parameters:

f - The file path

Returns:

The file checksum. The default return value is null, which indicates that no checksum algorithm is implemented in the corresponding FileSystem.

Throws:

IOException

getWorkingDirectory

public Path getWorkingDirectory()

Description copied from class: FileSystem

Get the current working directory for the given file system

Specified by:

getWorkingDirectory in class FileSystem

Returns:

the directory pathname

setWorkingDirectory

public void setWorkingDirectory(Path f)

Description copied from class: FileSystem

Set the current working directory for the given file system. All relative paths will be resolved relative to it.

Specified by:

setWorkingDirectory in class FileSystem

append

public FSDataOutputStream append(Path f,
                        int bufferSize,
                        Progressable progress)
                          throws IOException

This optional operation is not yet supported.

Specified by:

append in class FileSystem

Parameters:

f - the existing file to be appended.

bufferSize - the size of the buffer to be used.

progress - for reporting progress if it is not null.

Throws:

IOException

create

public FSDataOutputStream create(Path f,
                        FsPermission permission,
                        boolean overwrite,
                        int bufferSize,
                        short replication,
                        long blockSize,
                        Progressable progress)
                          throws IOException

Description copied from class: FileSystem

Create an FSDataOutputStream at the indicated Path with write-progress reporting.

Specified by:

create in class FileSystem

Parameters:

f - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

Throws:

IOException

See Also:

FileSystem.setPermission(Path, FsPermission)

rename

public boolean rename(Path src,
             Path dst)
               throws IOException

Description copied from class: FileSystem

Renames Path src to Path dst. Can take place on local fs or remote DFS.

Specified by:

rename in class FileSystem

Parameters:

src - path to be renamed

dst - new path after rename

Returns:

true if rename is successful

Throws:

IOException - on failure

delete

public boolean delete(Path f,
             boolean recursive)
               throws IOException

Description copied from class: FileSystem

Delete a file.

Specified by:

delete in class FileSystem

Parameters:

f - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

Returns:

true if delete is successful else false.

Throws:

IOException

mkdirs

public boolean mkdirs(Path f,
             FsPermission permission)
               throws IOException

Description copied from class: FileSystem

Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Specified by:

mkdirs in class FileSystem

Parameters:

f - path to create

permission - to apply to f

Throws:

IOException

getContentSummary

public ContentSummary getContentSummary(Path f)
                                 throws IOException

Description copied from class: FileSystem

Return the ContentSummary of a given Path.

Overrides:

getContentSummary in class FileSystem

Parameters:

f - path to use

Throws:

IOException