tachyon.master

Class MasterInfo

java.lang.Object

tachyon.master.ImageWriter

tachyon.master.MasterInfo

public class MasterInfo
extends ImageWriter

A global view of filesystem in master.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	MasterInfo.MasterInfoHeartbeatExecutor Master info periodical status check.
class	MasterInfo.RecomputationScheduler

Field Summary

Fields

Modifier and Type	Field and Description
static String	COL

Constructor Summary

Constructors

Constructor and Description
MasterInfo(InetSocketAddress address, Journal journal)

Method Summary

Modifier and Type	Method and Description
ClientRawTableInfo	_getClientRawTableInfo(String path, Inode inode) Get the raw table info associated with the given id.
boolean	_rename(int fileId, String dstPath) Rename a file to the given path, inner method.
boolean	addCheckpoint(long workerId, int fileId, long length, String checkpointPath) Add a checkpoint to a file.
int	cacheBlock(long workerId, long workerUsedBytes, long blockId, long length) A worker cache a block in its memory.
void	completeFile(int fileId) Completes the checkpointing of a file.
int	createDependency(List<String> parents, List<String> children, String commandPrefix, List<ByteBuffer> data, String comment, String framework, String frameworkVersion, DependencyType dependencyType)
int	createFile(boolean recursive, String path, boolean directory, long blockSizeByte) Create a file.
int	createFile(String path, long blockSizeByte)
long	createNewBlock(int fileId) Creates a new block for the given file.
int	createRawTable(String path, int columns, ByteBuffer metadata) Creates a raw table.
boolean	delete(int fileId, boolean recursive) Delete a file based on the file's ID.
boolean	delete(String path, boolean recursive) Delete files based on the path.
long	getBlockIdBasedOnOffset(int fileId, long offset)
List<BlockInfo>	getBlockList(String path) Get the list of blocks of an InodeFile determined by path.
long	getCapacityBytes() Get the capacity of the whole system.
ClientBlockInfo	getClientBlockInfo(long blockId) Get the block info associated with the given id.
ClientDependencyInfo	getClientDependencyInfo(int dependencyId) Get the dependency info associated with the given id.
ClientFileInfo	getClientFileInfo(int fid) Get the file info associated with the given id.
ClientFileInfo	getClientFileInfo(String path) Get the file info for the file at the given path
ClientRawTableInfo	getClientRawTableInfo(int id) Get the raw table info associated with the given id.
ClientRawTableInfo	getClientRawTableInfo(String path) Get the raw table info for the table at the given path
int	getFileId(String path) Get the file id of the file.
List<ClientBlockInfo>	getFileLocations(int fileId) Get the block infos of a file with the given id.
List<ClientBlockInfo>	getFileLocations(String path) Get the block infos of a file with the given path.
List<ClientFileInfo>	getFilesInfo(String path) If the path is a directory, return all the direct entries in it.
List<String>	getInMemoryFiles() Get absolute paths of all in memory files.
Journal	getJournal() Get Journal instance for MasterInfo for Unit test only
InetSocketAddress	getMasterAddress() Get the master address.
long	getNewUserId() Get a new user id
int	getNumberOfFiles(String path) Get the number of files at a given path.
String	getPath(int fileId) Get the path of a file with the given id
List<Integer>	getPinIdList() Get a list of the pin id's.
List<Integer>	getPriorityDependencyList() Creates a list of high priority dependencies, which don't yet have checkpoints.
int	getRawTableId(String path) Get the id of the table at the given path.
long	getStarttimeMs() Get the master start time in milliseconds.
long	getUnderFsCapacityBytes() Get the capacity of the under file system.
long	getUnderFsFreeBytes() Get the amount of free space in the under file system.
long	getUnderFsUsedBytes() Get the amount of space used in the under file system.
long	getUsedBytes() Get the amount of space used by the workers.
List<String>	getWhiteList() Get the white list.
NetAddress	getWorker(boolean random, String host) Get the address of a worker.
int	getWorkerCount() Get the number of workers.
List<ClientWorkerInfo>	getWorkersInfo() Get info about all the workers.
void	init()
List<Integer>	listFiles(String path, boolean recursive) Get the id of the file at the given path.
void	loadImage(com.fasterxml.jackson.core.JsonParser parser, String path) Load the image from parser, which is created based on the path.
List<String>	ls(String path, boolean recursive) Get the names of the sub-directories at the given path.
boolean	mkdir(String path) Create a directory at the given path.
long	registerWorker(NetAddress workerNetAddress, long totalBytes, long usedBytes, List<Long> currentBlockIds) Register a worker at the given address, setting it up and associating it with a given list of blocks.
boolean	rename(int fileId, String dstPath) Rename a file to the given path.
boolean	rename(String srcPath, String dstPath) Rename a file to the given path.
void	reportLostFile(int fileId) Logs a lost file and sets it to be recovered.
void	requestFilesInDependency(int depId) Request that the files for the given dependency be recomputed.
void	setPinned(int fileId, boolean pinned) Sets the isPinned flag on the given inode and all of its children.
void	stop() Stops the heartbeat thread.
void	updateRawTableMetadata(int tableId, ByteBuffer metadata) Update the metadata of a table.
Command	workerHeartbeat(long workerId, long usedBytes, List<Long> removedBlockIds) The heartbeat of the worker.
void	writeImage(com.fasterxml.jackson.databind.ObjectWriter objWriter, DataOutputStream dos) Create an image of the dependencies and filesystem tree.

Methods inherited from class tachyon.master.ImageWriter

writeElement

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

COL

public static final String COL

See Also:

Constant Field Values

Constructor Detail

MasterInfo

public MasterInfo(InetSocketAddress address,
                  Journal journal)
           throws IOException

Throws:

IOException

Method Detail

_getClientRawTableInfo

public ClientRawTableInfo _getClientRawTableInfo(String path,
                                                 Inode inode)
                                          throws TableDoesNotExistException

Get the raw table info associated with the given id.

Parameters:

path - The path of the table

inode - The inode at the path

Returns:

the table info

Throws:

TableDoesNotExistException

_rename

public boolean _rename(int fileId,
                       String dstPath)
                throws FileDoesNotExistException,
                       InvalidPathException

Rename a file to the given path, inner method.

Parameters:

fileId - The id of the file to rename

dstPath - The new path of the file

Returns:

true if the rename succeeded, false otherwise

Throws:

FileDoesNotExistException - If the id doesn't point to an inode

InvalidPathException - if the source path is a prefix of the destination

addCheckpoint

public boolean addCheckpoint(long workerId,
                             int fileId,
                             long length,
                             String checkpointPath)
                      throws FileNotFoundException,
                             SuspectedFileSizeException,
                             BlockInfoException

Add a checkpoint to a file.

Parameters:

workerId - The worker which submitted the request. -1 if the request is not from a worker.

fileId - The file to add the checkpoint.

length - The length of the checkpoint.

checkpointPath - The path of the checkpoint.

Returns:

true if the checkpoint is added successfully, false if not.

Throws:

FileNotFoundException

SuspectedFileSizeException

BlockInfoException

cacheBlock

public int cacheBlock(long workerId,
                      long workerUsedBytes,
                      long blockId,
                      long length)
               throws FileDoesNotExistException,
                      SuspectedFileSizeException,
                      BlockInfoException

A worker cache a block in its memory.

Parameters:

workerId -

workerUsedBytes -

blockId -

length -

Returns:

the dependency id of the file if it has not been checkpointed. -1 means the file either does not have dependency or has already been checkpointed.

Throws:

FileDoesNotExistException

SuspectedFileSizeException

BlockInfoException

completeFile

public void completeFile(int fileId)
                  throws FileDoesNotExistException

Completes the checkpointing of a file.

Parameters:

fileId - The id of the file

Throws:

FileDoesNotExistException

createDependency

public int createDependency(List<String> parents,
                            List<String> children,
                            String commandPrefix,
                            List<ByteBuffer> data,
                            String comment,
                            String framework,
                            String frameworkVersion,
                            DependencyType dependencyType)
                     throws InvalidPathException,
                            FileDoesNotExistException

Throws:

InvalidPathException

FileDoesNotExistException

createFile

public int createFile(boolean recursive,
                      String path,
                      boolean directory,
                      long blockSizeByte)
               throws FileAlreadyExistException,
                      InvalidPathException,
                      BlockInfoException,
                      TachyonException

Create a file. // TODO Make this API better.

Throws:

FileAlreadyExistException

InvalidPathException

BlockInfoException

TachyonException

createFile

public int createFile(String path,
                      long blockSizeByte)
               throws FileAlreadyExistException,
                      InvalidPathException,
                      BlockInfoException,
                      TachyonException

Throws:

FileAlreadyExistException

InvalidPathException

BlockInfoException

TachyonException

createNewBlock

public long createNewBlock(int fileId)
                    throws FileDoesNotExistException

Creates a new block for the given file.

Parameters:

fileId - The id of the file

Returns:

the block id.

Throws:

FileDoesNotExistException

createRawTable

public int createRawTable(String path,
                          int columns,
                          ByteBuffer metadata)
                   throws FileAlreadyExistException,
                          InvalidPathException,
                          TableColumnException,
                          TachyonException

Creates a raw table.

Parameters:

path - The path to place the table at

columns - The number of columns in the table

metadata - Additional metadata about the table

Returns:

the file id of the table

Throws:

FileAlreadyExistException

InvalidPathException

TableColumnException

TachyonException

delete

public boolean delete(int fileId,
                      boolean recursive)
               throws TachyonException

Delete a file based on the file's ID.

Parameters:

fileId - the file to be deleted.

recursive - whether delete the file recursively or not.

Returns:

succeed or not

Throws:

TachyonException

delete

public boolean delete(String path,
                      boolean recursive)
               throws TachyonException

Delete files based on the path.

Parameters:

path - The file to be deleted.

recursive - whether delete the file recursively or not.

Returns:

succeed or not

Throws:

TachyonException

getBlockIdBasedOnOffset

public long getBlockIdBasedOnOffset(int fileId,
                                    long offset)
                             throws FileDoesNotExistException

Throws:

FileDoesNotExistException

getBlockList

public List<BlockInfo> getBlockList(String path)
                             throws InvalidPathException,
                                    FileDoesNotExistException

Get the list of blocks of an InodeFile determined by path.

Parameters:

path - The file.

Returns:

The list of the blocks of the file.

Throws:

InvalidPathException

FileDoesNotExistException

getCapacityBytes

public long getCapacityBytes()

Get the capacity of the whole system.

Returns:

the system's capacity in bytes.

getClientBlockInfo

public ClientBlockInfo getClientBlockInfo(long blockId)
                                   throws FileDoesNotExistException,
                                          IOException,
                                          BlockInfoException

Get the block info associated with the given id.

Parameters:

blockId - The id of the block return

Returns:

the block info

Throws:

FileDoesNotExistException

IOException

BlockInfoException

getClientDependencyInfo

public ClientDependencyInfo getClientDependencyInfo(int dependencyId)
                                             throws DependencyDoesNotExistException

Get the dependency info associated with the given id.

Parameters:

dependencyId - The id of the dependency

Returns:

the dependency info

Throws:

DependencyDoesNotExistException

getClientFileInfo

public ClientFileInfo getClientFileInfo(int fid)
                                 throws FileDoesNotExistException,
                                        InvalidPathException

Get the file info associated with the given id.

Parameters:

fid - The id of the file

Returns:

the file info

Throws:

FileDoesNotExistException

InvalidPathException

getClientFileInfo

public ClientFileInfo getClientFileInfo(String path)
                                 throws FileDoesNotExistException,
                                        InvalidPathException

Get the file info for the file at the given path

Parameters:

path - The path of the file

Returns:

the file info

Throws:

FileDoesNotExistException

InvalidPathException

getClientRawTableInfo

public ClientRawTableInfo getClientRawTableInfo(int id)
                                         throws TableDoesNotExistException

Get the raw table info associated with the given id.

Parameters:

id - The id of the table

Returns:

the table info

Throws:

TableDoesNotExistException

getClientRawTableInfo

public ClientRawTableInfo getClientRawTableInfo(String path)
                                         throws TableDoesNotExistException,
                                                InvalidPathException

Get the raw table info for the table at the given path

Parameters:

path - The path of the table

Returns:

the table info

Throws:

TableDoesNotExistException

InvalidPathException

getFileId

public int getFileId(String path)
              throws InvalidPathException

Get the file id of the file.

Parameters:

path - The path of the file

Returns:

The file id of the file. -1 if the file does not exist.

Throws:

InvalidPathException

getFileLocations

public List<ClientBlockInfo> getFileLocations(int fileId)
                                       throws FileDoesNotExistException,
                                              IOException

Get the block infos of a file with the given id. Throws an exception if the id names a directory.

Parameters:

fileId - The id of the file to look up

Returns:

the block infos of the file

Throws:

FileDoesNotExistException

IOException

getFileLocations

public List<ClientBlockInfo> getFileLocations(String path)
                                       throws FileDoesNotExistException,
                                              InvalidPathException,
                                              IOException

Get the block infos of a file with the given path. Throws an exception if the path names a directory.

Parameters:

path - The path of the file to look up

Returns:

the block infos of the file

Throws:

FileDoesNotExistException

InvalidPathException

IOException

getFilesInfo

public List<ClientFileInfo> getFilesInfo(String path)
                                  throws FileDoesNotExistException,
                                         InvalidPathException

If the path is a directory, return all the direct entries in it. If the path is a file, return its ClientFileInfo.

Parameters:

path - the target directory/file path

Returns:

A list of ClientFileInfo

Throws:

FileDoesNotExistException

InvalidPathException

getInMemoryFiles

public List<String> getInMemoryFiles()

Get absolute paths of all in memory files.

Returns:

absolute paths of all in memory files.

getJournal

public Journal getJournal()

Get Journal instance for MasterInfo for Unit test only

Returns:

Journal instance

getMasterAddress

public InetSocketAddress getMasterAddress()

Get the master address.

Returns:

the master address

getNewUserId

public long getNewUserId()

Get a new user id

Returns:

a new user id

getNumberOfFiles

public int getNumberOfFiles(String path)
                     throws InvalidPathException,
                            FileDoesNotExistException

Get the number of files at a given path.

Parameters:

path - The path to look at

Returns:

The number of files at the path. Returns 1 if the path specifies a file. If it's a directory, returns the number of items in the directory.

Throws:

InvalidPathException

FileDoesNotExistException

getPath

public String getPath(int fileId)
               throws FileDoesNotExistException

Get the path of a file with the given id

Parameters:

fileId - The id of the file to look up

Returns:

the path of the file

Throws:

FileDoesNotExistException - raise if the file does not exist.

getPinIdList

public List<Integer> getPinIdList()

Get a list of the pin id's.

Returns:

a list of pin id's

getPriorityDependencyList

public List<Integer> getPriorityDependencyList()

Creates a list of high priority dependencies, which don't yet have checkpoints.

Returns:

the list of dependency ids

getRawTableId

public int getRawTableId(String path)
                  throws InvalidPathException,
                         TableDoesNotExistException

Get the id of the table at the given path.

Parameters:

path - The path of the table

Returns:

the id of the table

Throws:

InvalidPathException

TableDoesNotExistException

getStarttimeMs

public long getStarttimeMs()

Get the master start time in milliseconds.

Returns:

the master start time in milliseconds

getUnderFsCapacityBytes

public long getUnderFsCapacityBytes()
                             throws IOException

Get the capacity of the under file system.

Returns:

the capacity in bytes

Throws:

IOException

getUnderFsFreeBytes

public long getUnderFsFreeBytes()
                         throws IOException

Get the amount of free space in the under file system.

Returns:

the free space in bytes

Throws:

IOException

getUnderFsUsedBytes

public long getUnderFsUsedBytes()
                         throws IOException

Get the amount of space used in the under file system.

Returns:

the space used in bytes

Throws:

IOException

getUsedBytes

public long getUsedBytes()

Get the amount of space used by the workers.

Returns:

the amount of space used in bytes

getWhiteList

public List<String> getWhiteList()

Get the white list.

Returns:

the white list

getWorker

public NetAddress getWorker(boolean random,
                            String host)

Get the address of a worker.

Parameters:

random - If true, select a random worker

host - If random is false, select a worker on this host

Returns:

the address of the selected worker, or null if no address could be found

getWorkerCount

public int getWorkerCount()

Get the number of workers.

Returns:

the number of workers

getWorkersInfo

public List<ClientWorkerInfo> getWorkersInfo()

Get info about all the workers.

Returns:

a list of worker infos

init

public void init()
          throws IOException

Throws:

IOException

listFiles

public List<Integer> listFiles(String path,
                               boolean recursive)
                        throws InvalidPathException,
                               FileDoesNotExistException

Get the id of the file at the given path. If recursive, it scans the subdirectories as well.

Parameters:

path - The path to start looking at

recursive - If true, recursively scan the subdirectories at the given path as well

Returns:

the list of the inode id's at the path

Throws:

InvalidPathException

FileDoesNotExistException

loadImage

public void loadImage(com.fasterxml.jackson.core.JsonParser parser,
                      String path)
               throws IOException

Load the image from parser, which is created based on the path. Assume this blocks the whole MasterInfo.

Parameters:

parser - the JsonParser to load the image

path - the file to load the image

Throws:

IOException

ls

public List<String> ls(String path,
                       boolean recursive)
                throws InvalidPathException,
                       FileDoesNotExistException

Get the names of the sub-directories at the given path.

Parameters:

path - The path to look at

recursive - If true, recursively add the paths of the sub-directories

Returns:

the list of paths

Throws:

InvalidPathException

FileDoesNotExistException

mkdir

public boolean mkdir(String path)
              throws FileAlreadyExistException,
                     InvalidPathException,
                     TachyonException

Create a directory at the given path.

Parameters:

path - The path to create a directory at

Returns:

true if and only if the directory was created; false otherwise

Throws:

FileAlreadyExistException

InvalidPathException

TachyonException

registerWorker

public long registerWorker(NetAddress workerNetAddress,
                           long totalBytes,
                           long usedBytes,
                           List<Long> currentBlockIds)
                    throws BlockInfoException

Register a worker at the given address, setting it up and associating it with a given list of blocks.

Parameters:

workerNetAddress - The address of the worker to register

totalBytes - The capacity of the worker in bytes

usedBytes - The number of bytes already used in the worker

currentBlockIds - The id's of the blocks held by the worker

Returns:

the new id of the registered worker

Throws:

BlockInfoException

rename

public boolean rename(int fileId,
                      String dstPath)
               throws FileDoesNotExistException,
                      InvalidPathException

Rename a file to the given path.

Parameters:

fileId - The id of the file to rename

dstPath - The new path of the file

Returns:

true if the rename succeeded, false otherwise

Throws:

FileDoesNotExistException

InvalidPathException

rename

public boolean rename(String srcPath,
                      String dstPath)
               throws FileDoesNotExistException,
                      InvalidPathException

Rename a file to the given path.

Parameters:

srcPath - The path of the file to rename

dstPath - The new path of the file

Returns:

true if the rename succeeded, false otherwise

Throws:

FileDoesNotExistException

InvalidPathException

reportLostFile

public void reportLostFile(int fileId)

Logs a lost file and sets it to be recovered.

Parameters:

fileId - The id of the file to be recovered

requestFilesInDependency

public void requestFilesInDependency(int depId)

Request that the files for the given dependency be recomputed.

Parameters:

depId - The dependency whose files are to be recomputed

stop

public void stop()

Stops the heartbeat thread.

setPinned

public void setPinned(int fileId,
                      boolean pinned)
               throws FileDoesNotExistException

Sets the isPinned flag on the given inode and all of its children.

Throws:

FileDoesNotExistException

updateRawTableMetadata

public void updateRawTableMetadata(int tableId,
                                   ByteBuffer metadata)
                            throws TableDoesNotExistException,
                                   TachyonException

Update the metadata of a table.

Parameters:

tableId - The id of the table to update

metadata - The new metadata to update the table with

Throws:

TableDoesNotExistException

TachyonException

workerHeartbeat

public Command workerHeartbeat(long workerId,
                               long usedBytes,
                               List<Long> removedBlockIds)
                        throws BlockInfoException

The heartbeat of the worker. It updates the information of the worker and removes the given block id's.

Parameters:

workerId - The id of the worker to deal with

usedBytes - The number of bytes used in the worker

removedBlockIds - The id's of the blocks that have been removed

Returns:

a command specifying an action to take

Throws:

BlockInfoException

writeImage

public void writeImage(com.fasterxml.jackson.databind.ObjectWriter objWriter,
                       DataOutputStream dos)
                throws IOException

Create an image of the dependencies and filesystem tree.

Parameters:

os - The output stream to write the image to

Throws:

IOException