opennlp.tools.tokenize

Class TokenizerModel

java.lang.Object

opennlp.tools.util.model.BaseModel

opennlp.tools.tokenize.TokenizerModel

All Implemented Interfaces:

ArtifactProvider

public final class TokenizerModel
extends BaseModel

The TokenizerModel is the model used by a learnable Tokenizer.

See Also:

TokenizerME

Field Summary

Fields inherited from class opennlp.tools.util.model.BaseModel

artifactMap, FACTORY_NAME, MANIFEST_ENTRY, toolFactory, TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY

Constructor Summary

Constructors

Constructor and Description
TokenizerModel(AbstractModel tokenizerModel, Map<String,String> manifestInfoEntries, TokenizerFactory tokenizerFactory) Initializes the current instance.
TokenizerModel(File modelFile)
TokenizerModel(InputStream in) Initializes the current instance.
TokenizerModel(String language, AbstractModel tokenizerMaxentModel, boolean useAlphaNumericOptimization) Deprecated. Use TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory) instead and pass in a TokenizerFactory.
TokenizerModel(String language, AbstractModel tokenizerMaxentModel, boolean useAlphaNumericOptimization, Map<String,String> manifestInfoEntries) Deprecated. Use TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory) instead and pass in a TokenizerFactory.
TokenizerModel(String language, AbstractModel tokenizerMaxentModel, Dictionary abbreviations, boolean useAlphaNumericOptimization, Map<String,String> manifestInfoEntries) Deprecated. Use TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory) instead and pass in a TokenizerFactory.
TokenizerModel(URL modelURL)

Method Summary

Modifier and Type	Method and Description
Dictionary	getAbbreviations()
protected Class<? extends BaseToolFactory>	getDefaultFactory() Sub-classes should override this method if their module has a default BaseToolFactory sub-class.
TokenizerFactory	getFactory()
AbstractModel	getMaxentModel()
static void	main(String[] args)
boolean	useAlphaNumericOptimization()
protected void	validateArtifactMap() Validates the parsed artifacts.

Methods inherited from class opennlp.tools.util.model.BaseModel

checkArtifactMap, createArtifactSerializers, createArtifactSerializers, getArtifact, getArtifactSerializer, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serialize, setManifestProperty

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerModel

public TokenizerModel(AbstractModel tokenizerModel,
                      Map<String,String> manifestInfoEntries,
                      TokenizerFactory tokenizerFactory)

Initializes the current instance.

Parameters:

tokenizerModel - the model

manifestInfoEntries - the manifest

tokenizerFactory - the factory

TokenizerModel

public TokenizerModel(String language,
                      AbstractModel tokenizerMaxentModel,
                      Dictionary abbreviations,
                      boolean useAlphaNumericOptimization,
                      Map<String,String> manifestInfoEntries)

Deprecated. Use TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory) instead and pass in a TokenizerFactory.

Initializes the current instance.

Parameters:

tokenizerMaxentModel -

useAlphaNumericOptimization -

TokenizerModel

public TokenizerModel(String language,
                      AbstractModel tokenizerMaxentModel,
                      boolean useAlphaNumericOptimization,
                      Map<String,String> manifestInfoEntries)

Deprecated. Use TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory) instead and pass in a TokenizerFactory.

Initializes the current instance.

Parameters:

language -

tokenizerMaxentModel -

useAlphaNumericOptimization -

manifestInfoEntries -

TokenizerModel

public TokenizerModel(String language,
                      AbstractModel tokenizerMaxentModel,
                      boolean useAlphaNumericOptimization)

Deprecated. Use TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory) instead and pass in a TokenizerFactory.

Initializes the current instance.

Parameters:

language -

tokenizerMaxentModel -

useAlphaNumericOptimization -

TokenizerModel

public TokenizerModel(InputStream in)
               throws IOException,
                      InvalidFormatException

Initializes the current instance.

Parameters:

in -

Throws:

IOException

InvalidFormatException

TokenizerModel

public TokenizerModel(File modelFile)
               throws IOException,
                      InvalidFormatException

Throws:

IOException

InvalidFormatException

TokenizerModel

public TokenizerModel(URL modelURL)
               throws IOException,
                      InvalidFormatException

Throws:

IOException

InvalidFormatException

Method Detail

validateArtifactMap

protected void validateArtifactMap()
                            throws InvalidFormatException

Description copied from class: BaseModel

Validates the parsed artifacts. If something is not valid subclasses should throw an InvalidFormatException. Note: Subclasses should generally invoke super.validateArtifactMap at the beginning of this method.

Overrides:

validateArtifactMap in class BaseModel

Throws:

InvalidFormatException

getFactory

public TokenizerFactory getFactory()

getDefaultFactory

protected Class<? extends BaseToolFactory> getDefaultFactory()

Description copied from class: BaseModel

Sub-classes should override this method if their module has a default BaseToolFactory sub-class.

Overrides:

getDefaultFactory in class BaseModel

Returns:

the default BaseToolFactory for the module, or null if none.

getMaxentModel

public AbstractModel getMaxentModel()

getAbbreviations

public Dictionary getAbbreviations()

useAlphaNumericOptimization

public boolean useAlphaNumericOptimization()

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException