public class BaseDataFileFactory extends java.lang.Object implements DataFactory, CacheableFactory, ModuleControl, ModuleSupportable, java.security.PrivilegedExceptionAction
RESOLVE (mikem - 2/19/98) - Currently only getContainerClass() is abstract, there are probably more routines which should be abstract. Also the other implementations should probably inherit from the abstract class, rather than from the DataFileFactory class. Also there probably should be a generic directory and the rest of the filesystem implementations parallel to it. I wanted to limit the changes going into the branch and then fix inheritance stuff in main.
The code in this class was moved over from DataFileFactory.java and then that file was made to inherit from this one.
DB_EX_LOCKFILE_NAME, DB_LOCKFILE_NAME, MODULE, TEMP_SEGMENT_NAME| Constructor and Description |
|---|
BaseDataFileFactory() |
| Modifier and Type | Method and Description |
|---|---|
long |
addAndLoadStreamContainer(RawTransaction t,
long segmentId,
java.util.Properties tableProperties,
RowSource rowSource)
Add and load a stream container
|
long |
addContainer(RawTransaction t,
long segmentId,
long input_containerid,
int mode,
java.util.Properties tableProperties,
int temporaryFlag)
Add a container with a specified page size to a segment.
|
void |
backupDataFiles(Transaction rt,
java.io.File backupDir)
Back up the data segment of the database.
|
void |
boot(boolean create,
java.util.Properties startParams)
Boot this module with the given properties.
|
private void |
bootLogFactory(boolean create,
java.util.Properties startParams) |
private static java.lang.String |
buildJvmVersion()
Return values of system properties that identify the JVM.
|
private static java.lang.String |
buildOSinfo()
Return values of system properties that identify the OS.
|
boolean |
canSupport(java.util.Properties startParams)
See if this implementation can support any attributes that are listed in properties.
|
void |
checkpoint()
Implement checkpoint operation, write/sync all pages in cache.
|
void |
createFinished()
Database creation finished
|
boolean |
databaseEncrypted()
Returns if data base is in encrypted mode.
|
int |
decrypt(byte[] ciphertext,
int offset,
int length,
byte[] cleartext,
int outputOffset)
Decrypt cleartext from ciphertext.
|
void |
decryptAllContainers(RawTransaction t)
Decrypts all the containers in the data segment.
|
void |
dropContainer(RawTransaction t,
ContainerKey ckey)
Drop a container.
|
void |
dropStreamContainer(RawTransaction t,
long segmentId,
long containerId)
Drop a stream container.
|
int |
encrypt(byte[] cleartext,
int offset,
int length,
byte[] ciphertext,
int outputOffset,
boolean newEngine)
Encrypt cleartext into ciphertext.
|
void |
encryptAllContainers(RawTransaction t)
Encrypt all the containers in the data segment.
|
(package private) void |
fileToRemove(StorageFile file,
boolean remove)
Add a file to the list of files to be removed post recovery.
|
private long |
findMaxContainerId()
Find the largest containerid is seg 0.
|
(package private) void |
flush(LogInstant instant)
Ask the log factory to flush up to this log instant.
|
void |
freezePersistentStore()
Backup restore - stop writing dirty pages or container to disk
|
StorageFile |
getAlternateContainerPath(ContainerKey containerId,
boolean stub)
Return an alternate path to container file relative to the root directory.
|
(package private) CacheManager |
getContainerCache() |
(package private) java.lang.String[] |
getContainerNames()
get all the names of the files in seg 0.
|
StorageFile |
getContainerPath(ContainerKey containerId,
boolean stub)
Return the path to a container file.
|
private StorageFile |
getContainerPath(ContainerKey containerId,
boolean stub,
int code) |
int |
getEncryptionBlockSize()
Returns the encryption block size used by the algorithm at time of
creation of an encrypted database
|
FileResource |
getFileHandler()
Get an object to handle non-transactional files.
|
UUID |
getIdentifier()
Return my unique identifier
|
int |
getIntParameter(java.lang.String parameterName,
java.util.Properties properties,
int defaultValue,
int minimumValue,
int maximumValue) |
private void |
getJBMSLockOnDB(UUID myUUID,
UUIDFactory uuidFactory,
java.lang.String databaseDirectory)
check to see if we are the only JBMS opened against this database.
|
(package private) LogFactory |
getLogFactory() |
private AllocationActions |
getLoggableAllocationActions()
Get the loggable allocation action associated with this implementation
|
private PageActions |
getLoggablePageActions() |
long |
getMaxContainerId()
Return an id which can be used to create a container.
|
(package private) long |
getNextId() |
(package private) CacheManager |
getPageCache() |
(package private) RawStoreFactory |
getRawStoreFactory() |
java.lang.String |
getRootDirectory()
Get the root directory of the data storage area.
|
StorageFactory |
getStorageFactory() |
java.lang.String |
getVersionedName(java.lang.String name,
long generationId) |
private boolean |
handleServiceType(java.lang.String type)
Does this factory support this service type.
|
void |
idle() |
boolean |
isReadOnly()
Is the store read-only.
|
private static java.lang.String |
jarClassPath(java.lang.Class cls)
Return a jar file by asking the class's
class loader for the location where the class was loaded from.
|
private void |
logMsg(java.lang.String msg) |
StandardException |
markCorrupt(StandardException originalError)
Really this is just a convience routine for callers that might not
have access to a log factory.
|
Cacheable |
newCacheable(CacheManager cm) |
(package private) Cacheable |
newContainerObject()
Produces new container objects.
|
protected Cacheable |
newRAFContainer(BaseDataFileFactory factory)
Creates a RAFContainer object.
|
ContainerHandle |
openContainer(RawTransaction t,
ContainerKey containerId,
LockingPolicy locking,
int mode)
Open a container that is not droped.
|
private RawContainerHandle |
openContainer(RawTransaction t,
ContainerKey identity,
LockingPolicy locking,
int mode,
boolean droppedOK) |
RawContainerHandle |
openDroppedContainer(RawTransaction t,
ContainerKey containerId,
LockingPolicy locking,
int mode)
Open a container that may have been dropped.
|
StreamContainerHandle |
openStreamContainer(RawTransaction t,
long segmentId,
long containerId,
boolean hold)
open an exsisting streamContainer
|
void |
postRecovery()
Called after recovery is performed.
|
private void |
privGetJBMSLockOnDB() |
private void |
privReleaseJBMSLockOnDB() |
private void |
privRestoreDataDirectory() |
(package private) int |
random()
return a secure random number
|
int |
reclaimSpace(Serviceable work,
ContextManager contextMgr)
Reclaim space used by this factory.
|
void |
reCreateContainerForRedoRecovery(RawTransaction t,
long segmentId,
long containerId,
ByteArray containerInfo)
re-Create a container during redo recovery.
|
private void |
releaseJBMSLockOnDB() |
void |
removeDroppedContainerFileStubs(LogInstant redoLWM)
Delete the stub files that are not required for recovery.
|
void |
removeOldVersionOfContainers()
Removes old versions of the containers after a cryptographic operation
on the database.
|
private void |
removeStubs()
Remove stubs in this database.
|
void |
removeStubsOK()
Tell the data factory it is OK to remove committed deleted containers
when the data factory shuts down.
|
private void |
removeTempDirectory() |
private void |
restoreDataDirectory(java.lang.String backupPath)
removes the data directory(seg*) from database home directory and
restores it from backup location.
|
java.lang.Object |
run() |
void |
setDatabaseEncrypted(boolean isEncrypted)
Sets whether the database is encrypted.
|
void |
setRawStoreFactory(RawStoreFactory rsf,
boolean create,
java.util.Properties startParams)
make data factory aware of which raw store factory it belongs to
Also need to boot the LogFactory
|
void |
setupCacheCleaner(DaemonService daemon)
Set up the cache cleaner for the container cache and the page cache.
|
void |
stop()
Stop the module.
|
void |
stubFileToRemoveAfterCheckPoint(StorageFile file,
LogInstant logInstant,
java.lang.Object identity)
keeps track of information about the stub files of the committed deleted
containers.
|
void |
unfreezePersistentStore()
Backup restore - start writing dirty pages or container to disk
|
void |
writeFinished()
Backup restore - write finished, if this is the last writer, allow the
persistent store to proceed.
|
void |
writeInProgress()
Backup restore - don't allow the persistent store to be frozen - or if
it is already frozen, block.
|
StorageFactory storageFactory
WritableStorageFactory writableStorageFactory
private long nextContainerId
private boolean databaseEncrypted
private CacheManager pageCache
private CacheManager containerCache
private LogFactory logFactory
private ProductVersionHolder jbmsVersion
private java.lang.String jvmVersion
private java.lang.String osInfo
private java.lang.String jarCPath
private RawStoreFactory rawStoreFactory
private java.lang.String dataDirectory
private boolean throwDBlckException
private UUID identifier
private final java.lang.Object freezeSemaphore
private boolean isFrozen
private int writersInProgress
private boolean removeStubsOK
private boolean isCorrupt
private boolean inCreateNoLog
private StorageRandomAccessFile fileLockOnDB
private StorageFile exFileLock
private HeaderPrintWriter istream
private static final java.lang.String LINE
boolean dataNotSyncedAtAllocation
boolean dataNotSyncedAtCheckpoint
private PageActions loggablePageActions
private AllocationActions loggableAllocActions
private boolean readOnly
private boolean supportsRandomAccess
private FileResource fileHandler
private java.util.Hashtable droppedTableStubInfo
private java.util.Hashtable postRecoveryRemovedFiles
private int actionCode
private static final int REMOVE_TEMP_DIRECTORY_ACTION
private static final int GET_CONTAINER_PATH_ACTION
private static final int GET_ALTERNATE_CONTAINER_PATH_ACTION
private static final int FIND_MAX_CONTAINER_ID_ACTION
private static final int DELETE_IF_EXISTS_ACTION
private static final int GET_PATH_ACTION
private static final int POST_RECOVERY_REMOVE_ACTION
private static final int REMOVE_STUBS_ACTION
private static final int BOOT_ACTION
private static final int GET_LOCK_ON_DB_ACTION
private static final int RELEASE_LOCK_ON_DB_ACTION
private static final int RESTORE_DATA_DIRECTORY_ACTION
private static final int GET_CONTAINER_NAMES_ACTION
private ContainerKey containerId
private boolean stub
private StorageFile actionFile
private UUID myUUID
private UUIDFactory uuidFactory
private java.lang.String databaseDirectory
private java.io.File backupRoot
private java.lang.String[] bfilelist
public boolean canSupport(java.util.Properties startParams)
ModuleSupportable
The module can check for attributes in the properties to
see if it can fulfill the required behaviour. E.g. the raw
store may define an attribute called RawStore.Recoverable.
If a temporary raw store is required the property RawStore.recoverable=false
would be added to the properties before calling bootServiceModule. If a
raw store cannot support this attribute its canSupport method would
return null. Also see the Monitor class's prologue to see how the
identifier is used in looking up properties.
Actually a better way maybe to have properties of the form
RawStore.Attributes.mandatory=recoverable,smallfootprint and
RawStore.Attributes.requested=oltp,fast
canSupport in interface ModuleSupportablepublic void boot(boolean create,
java.util.Properties startParams)
throws StandardException
ModuleControlAn implementation's boot method can throw StandardException. If it is thrown the module is not registered by the monitor and therefore cannot be found through a findModule(). In this case the module's stop() method is not called, thus throwing this exception must free up any resources.
When create is true the contents of the properties object
will be written to the service.properties of the persistent
service. Thus any code that requires an entry in service.properties
must explicitly place the value in this properties set
using the put method.
Typically the properties object contains one or more default
properties sets, which are not written out to service.properties.
These default sets are how callers modify the create process. In a
JDBC connection database create the first set of defaults is a properties
object that contains the attributes that were set on the jdbc:derby: URL.
This attributes properties set has the second default properties set as
its default. This set (which could be null) contains the properties
that the user set on their DriverManager.getConnection() call, and are thus
not owned by Derby code, and thus must not be modified by Derby
code.
When create is false the properties object contains all the properties set in the service.properties file plus a limited number of attributes from the JDBC URL attributes or connection properties set. This avoids properties set by the user compromising the boot process. An example of a property passed in from the JDBC world is the bootPassword for encrypted databases.
Code should not hold onto the passed in properties reference after boot time as its contents may change underneath it. At least after the complete boot is completed, the links to all the default sets will be removed.
boot in interface ModuleControlStandardException - Module cannot be started.Monitor,
ModuleFactorypublic void stop()
ModuleControlstop in interface ModuleControlstop in interface DataFactoryMonitor,
ModuleFactorypublic Cacheable newCacheable(CacheManager cm)
newCacheable in interface CacheableFactorypublic void createFinished()
throws StandardException
createFinished in interface DataFactoryStandardException - Standard Derby exception policy.public ContainerHandle openContainer(RawTransaction t, ContainerKey containerId, LockingPolicy locking, int mode) throws StandardException
DataFactoryopenContainer in interface DataFactoryt - the raw transaction that is opening the containercontainerId - the container's identitylocking - the locking policymode - see the different mode in @see ContainerHandle
then will return a null handle if the container is dropped.StandardException - Standard Derby error policypublic RawContainerHandle openDroppedContainer(RawTransaction t, ContainerKey containerId, LockingPolicy locking, int mode) throws StandardException
DataFactoryopenDroppedContainer in interface DataFactoryStandardException - Standard Derby error policyDataFactory.openDroppedContainer(org.apache.derby.iapi.store.raw.xact.RawTransaction, org.apache.derby.iapi.store.raw.ContainerKey, org.apache.derby.iapi.store.raw.LockingPolicy, int)private RawContainerHandle openContainer(RawTransaction t, ContainerKey identity, LockingPolicy locking, int mode, boolean droppedOK) throws StandardException
StandardException - Standard Derby error policyDataFactory.openContainer(org.apache.derby.iapi.store.raw.xact.RawTransaction, org.apache.derby.iapi.store.raw.ContainerKey, org.apache.derby.iapi.store.raw.LockingPolicy, int)public long addContainer(RawTransaction t, long segmentId, long input_containerid, int mode, java.util.Properties tableProperties, int temporaryFlag) throws StandardException
addContainer in interface DataFactoryt - the transaction that is creating the containersegmentId - the segment where the container is to gomode - whether or not to LOGGED or not. The effect of this mode
is only for this addContainer call, not persisently stored
throughout the lifetime of the containertableProperties - properties of the container that is persistently
stored throughout the lifetime of the containerStandardException - Standard Derby error policypublic long addAndLoadStreamContainer(RawTransaction t, long segmentId, java.util.Properties tableProperties, RowSource rowSource) throws StandardException
addAndLoadStreamContainer in interface DataFactoryt - the transaction that is creating the containersegmentId - the segment where the container is to gotableProperties - properties of the container that is persistently
stored throughout the lifetime of the containerrowSource - the data to load the container withStandardException - Standard Derby error policypublic StreamContainerHandle openStreamContainer(RawTransaction t, long segmentId, long containerId, boolean hold) throws StandardException
openStreamContainer in interface DataFactoryStandardException - Standard Derby error policyDataFactory.openStreamContainer(org.apache.derby.iapi.store.raw.xact.RawTransaction, long, long, boolean)public void dropStreamContainer(RawTransaction t, long segmentId, long containerId) throws StandardException
Synchronisation
This call will remove the container.
dropStreamContainer in interface DataFactoryStandardException - Standard Derby error policypublic void reCreateContainerForRedoRecovery(RawTransaction t, long segmentId, long containerId, ByteArray containerInfo) throws StandardException
reCreateContainerForRedoRecovery in interface DataFactoryStandardException - Standard Derby Error policypublic void dropContainer(RawTransaction t, ContainerKey ckey) throws StandardException
Synchronisation
This call will mark the container as dropped and then obtain an CX lock (table level exclusive lock) on the container. Once a container has been marked as dropped it cannot be retrieved by an openContainer() call unless explicitly with droppedOK.
Once the exclusive lock has been obtained the container is removed and all its pages deallocated. The container will be fully removed at the commit time of the transaction.
dropContainer in interface DataFactoryStandardException - Standard Derby error policypublic void checkpoint()
throws StandardException
The derby write ahead log algorithm uses checkpoint of the data cache to determine points of the log no longer required by restart recovery.
This implementation uses the 2 cache interfaces to force all dirty pages to disk: WRITE DIRTY PAGES TO OS: In the first step all pages in the page cache are written, but not synced (pagecache.cleanAll). The cachemanager cleanAll() interface guarantees that every dirty page that exists when this call is first made will have it's clean() method called. The data cache (CachedPage.clean()), will call writePage but not sync the page. By using the java write then sync, the checkpoint is usually doing async I/O, allowing the OS to schedule multiple I/O's to the file as efficiently as it can. Note that it has been observed that checkpoints can flood the I/O system because these writes are not synced, see DERBY-799 - checkpoint should probably somehow restrict the rate it sends out those I/O's - it was observed a simple sleep every N writes fixed most of the problem. FORCE THOSE DIRTY WRITES TO DISK: To force the I/O's to disk, the system calls each open dirty file and uses the java interface to sync any outstanding dirty pages to disk (containerCache.cleanAll()). The open container cache does this work in RAFContainer.clean() by writing it's header out and syncing the file. (Note if any change is made to checkpoint to sync the writes vs. syncing the file, one probably still needs to write the container header out and sync it).
checkpoint in interface DataFactoryStandardException - Standard exception policy.public void idle()
throws StandardException
idle in interface DataFactoryStandardExceptionpublic void setRawStoreFactory(RawStoreFactory rsf, boolean create, java.util.Properties startParams) throws StandardException
DataFactorysetRawStoreFactory in interface DataFactoryStandardException - cannot boot the log factorypublic UUID getIdentifier()
getIdentifier in interface DataFactoryDataFactory.getIdentifier()public int reclaimSpace(Serviceable work, ContextManager contextMgr) throws StandardException
DataFactoryreclaimSpace in interface DataFactoryStandardException - Standard Derby exception policypublic StandardException markCorrupt(StandardException originalError)
markCorrupt in interface Corruptablepublic FileResource getFileHandler()
DataFactorygetFileHandler in interface DataFactorypublic void removeStubsOK()
DataFactoryremoveStubsOK in interface DataFactorypublic int getIntParameter(java.lang.String parameterName,
java.util.Properties properties,
int defaultValue,
int minimumValue,
int maximumValue)
CacheManager getContainerCache()
CacheManager getPageCache()
void flush(LogInstant instant) throws StandardException
StandardException - cannot sync log fileLogFactory getLogFactory()
RawStoreFactory getRawStoreFactory()
public java.lang.String getRootDirectory()
Cacheable newContainerObject()
Concrete implementations of a DataFactory must implement this routine to indicate what kind of containers are produced. This class produces file-based containers - RAFContainer objects for files that support random access and InputStreamContainer object for others, such as data files in JARs.
protected Cacheable newRAFContainer(BaseDataFileFactory factory)
private PageActions getLoggablePageActions() throws StandardException
StandardExceptionprivate AllocationActions getLoggableAllocationActions()
private void removeTempDirectory()
public StorageFile getContainerPath(ContainerKey containerId, boolean stub)
Return the path to a container file that is relative to the root directory.
The format of the name of an existing container file is: segNNN/cXXX.dat The format of the name of a stub describing a dropped container file is: segNNN/dXXX.dat NNN = segment number, currently 0 is where normal db files are found. XXX = The hex representation of the container number The store will always create containers with this format name, but the store will also recognize the following two formats when attempting to open files - as some copy tools have uppercased our filesnames when moving across operating systems: The format of the name of an existing container file is: segNNN/CXXX.DAT The format of the name of a stub describing a dropped container file is: segNNN/DXXX.DAT
containerId - The container being opened/createdstub - True if the file name for the stub is requested,
otherwise the file name for the data fileprivate StorageFile getContainerPath(ContainerKey containerId, boolean stub, int code)
public StorageFile getAlternateContainerPath(ContainerKey containerId, boolean stub)
containerId - The container being opened/createdstub - True if the file name for the stub is requested, otherwise the file name for the data fileprivate void removeStubs()
public void stubFileToRemoveAfterCheckPoint(StorageFile file, LogInstant logInstant, java.lang.Object identity)
public void removeDroppedContainerFileStubs(LogInstant redoLWM) throws StandardException
removeDroppedContainerFileStubs in interface DataFactoryStandardException - Standard Derby error policyprivate long findMaxContainerId()
Do a file list of the files in seg0 and return the highest numbered file found.
Until I figure out some reliable place to store this information across a boot of the system, this is what is used following a boot to assign the next conglomerate id when a new conglomerate is created. It is only called at most once, and then the value is cached by calling store code.
private void bootLogFactory(boolean create,
java.util.Properties startParams)
throws StandardException
StandardExceptionprivate boolean handleServiceType(java.lang.String type)
private void getJBMSLockOnDB(UUID myUUID, UUIDFactory uuidFactory, java.lang.String databaseDirectory) throws StandardException
StandardException - another JBMS is already attached to the
database at this directoryprivate void privGetJBMSLockOnDB()
throws StandardException
StandardExceptionprivate void releaseJBMSLockOnDB()
private void privReleaseJBMSLockOnDB()
throws java.io.IOException
java.io.IOExceptionprivate void logMsg(java.lang.String msg)
public final boolean databaseEncrypted()
DataFactorydatabaseEncrypted in interface DataFactorypublic void setDatabaseEncrypted(boolean isEncrypted)
setDatabaseEncrypted in interface DataFactoryisEncrypted - true if the database is encrypted,
false otherwisepublic int encrypt(byte[] cleartext,
int offset,
int length,
byte[] ciphertext,
int outputOffset,
boolean newEngine)
throws StandardException
DataFactoryencrypt in interface DataFactoryStandardException - Standard Derby Error PolicyCipherProvider.encrypt(byte[], int, int, byte[], int)public int decrypt(byte[] ciphertext,
int offset,
int length,
byte[] cleartext,
int outputOffset)
throws StandardException
DataFactorydecrypt in interface DataFactoryStandardException - Standard Derby Error PolicyCipherProvider.decrypt(byte[], int, int, byte[], int)public void decryptAllContainers(RawTransaction t) throws StandardException
decryptAllContainers in interface DataFactoryt - the transaction that is decrypting the containerStandardException - Standard Derby Error Policypublic void encryptAllContainers(RawTransaction t) throws StandardException
encryptAllContainers in interface DataFactoryt - the transaction that is encrypting the containers.StandardException - Standard Derby Error Policypublic void removeOldVersionOfContainers()
throws StandardException
removeOldVersionOfContainers in interface DataFactoryStandardExceptionprivate static java.lang.String jarClassPath(java.lang.Class cls)
cls - the Class to ask to print the class name of an objectprivate static java.lang.String buildOSinfo()
private static java.lang.String buildJvmVersion()
public int getEncryptionBlockSize()
getEncryptionBlockSize in interface DataFactorypublic java.lang.String getVersionedName(java.lang.String name,
long generationId)
public long getMaxContainerId()
throws StandardException
Return an id number with is greater than any existing container in the current database. Caller will use this to allocate future container numbers - most likely caching the value and then incrementing it as it is used.
getMaxContainerId in interface DataFactoryStandardException - Standard exception policy.long getNextId()
int random()
void fileToRemove(StorageFile file, boolean remove)
public void postRecovery()
throws StandardException
postRecovery in interface DataFactoryStandardException - Standard Derby Error Policypublic void setupCacheCleaner(DaemonService daemon)
setupCacheCleaner in interface DataFactorydaemon - daemon service to use for background cleaningpublic void freezePersistentStore()
throws StandardException
DataFactoryfreezePersistentStore in interface DataFactoryStandardException - Standard Derby error policypublic void unfreezePersistentStore()
DataFactoryunfreezePersistentStore in interface DataFactorypublic void writeInProgress()
throws StandardException
DataFactorywriteInProgress in interface DataFactoryStandardException - Standard Derby error policypublic void writeFinished()
DataFactorywriteFinished in interface DataFactorypublic void backupDataFiles(Transaction rt, java.io.File backupDir) throws StandardException
DataFactorybackupDataFiles in interface DataFactoryStandardExceptionjava.lang.String[] getContainerNames()
private void restoreDataDirectory(java.lang.String backupPath)
throws StandardException
StandardExceptionprivate void privRestoreDataDirectory()
throws StandardException
StandardExceptionpublic boolean isReadOnly()
isReadOnly in interface DataFactorypublic StorageFactory getStorageFactory()
getStorageFactory in interface DataFactorypublic final java.lang.Object run()
throws java.io.IOException,
StandardException
run in interface java.security.PrivilegedExceptionActionjava.io.IOExceptionStandardExceptionApache Derby V10.10 Internals - Copyright © 2004,2014 The Apache Software Foundation. All Rights Reserved.