org.apache.lucene.analysis.uima

Class BaseUIMATokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.analysis.uima.BaseUIMATokenizer

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

UIMAAnnotationsTokenizer, UIMATypeAwareAnnotationsTokenizer

public abstract class BaseUIMATokenizer
extends Tokenizer

Abstract base implementation of a Tokenizer which is able to analyze the given input with a UIMA AnalysisEngine

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

AttributeSource.State

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.uima.analysis_engine.AnalysisEngine	ae
protected org.apache.uima.cas.CAS	cas
protected org.apache.uima.cas.FSIterator<org.apache.uima.cas.text.AnnotationFS>	iterator

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BaseUIMATokenizer(AttributeFactory factory, String descriptorPath, Map<String,Object> configurationParameters)

Method Summary

Modifier and Type	Method and Description
protected void	analyzeInput() analyzes the tokenizer input using the given analysis engine
protected abstract void	initializeIterator() initialize the FSIterator which is used to build tokens at each incrementToken() method call
void	reset() This method is called by a consumer before it begins consumption using TokenStream.incrementToken().

Methods inherited from class org.apache.lucene.analysis.Tokenizer

close, correctOffset, setReader

Methods inherited from class org.apache.lucene.analysis.TokenStream

end, incrementToken

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

iterator

protected org.apache.uima.cas.FSIterator<org.apache.uima.cas.text.AnnotationFS> iterator

ae

protected org.apache.uima.analysis_engine.AnalysisEngine ae

cas

protected org.apache.uima.cas.CAS cas

Constructor Detail

BaseUIMATokenizer

protected BaseUIMATokenizer(AttributeFactory factory,
                            String descriptorPath,
                            Map<String,Object> configurationParameters)

Method Detail

analyzeInput

protected void analyzeInput()
                     throws org.apache.uima.resource.ResourceInitializationException,
                            org.apache.uima.analysis_engine.AnalysisEngineProcessException,
                            IOException

analyzes the tokenizer input using the given analysis engine

cas will be filled with extracted metadata (UIMA annotations, feature structures)

Throws:

IOException - If there is a low-level I/O error.

org.apache.uima.resource.ResourceInitializationException

org.apache.uima.analysis_engine.AnalysisEngineProcessException

initializeIterator

protected abstract void initializeIterator()
                                    throws IOException

initialize the FSIterator which is used to build tokens at each incrementToken() method call

Throws:

IOException - If there is a low-level I/O error.

reset

public void reset()
           throws IOException

Description copied from class: TokenStream

This method is called by a consumer before it begins consumption using TokenStream.incrementToken().

Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.

If you override this method, always call super.reset(), otherwise some internal state will not be correctly reset (e.g., Tokenizer will throw IllegalStateException on further usage).

Overrides:

reset in class Tokenizer

Throws:

IOException