org.opensolaris.opengrok.analysis

Class JFlexTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.opensolaris.opengrok.analysis.JFlexTokenizer

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

Direct Known Subclasses:

CSharpSymbolTokenizer, CSymbolTokenizer, CxxSymbolTokenizer, FortranSymbolTokenizer, JavaScriptSymbolTokenizer, JavaSymbolTokenizer, LispSymbolTokenizer, PerlSymbolTokenizer, PhpSymbolTokenizer, PlainFullTokenizer, PlainSymbolTokenizer, PythonSymbolTokenizer, ShSymbolTokenizer, TclSymbolTokenizer, TroffFullTokenizer, VBSymbolTokenizer

public abstract class JFlexTokenizer
extends org.apache.lucene.analysis.Tokenizer

this class was created because of lucene 2.4.1 update which introduced char[] in Tokens instead of String lucene 3.0.0 uses AttributeSource instead of Tokens to make things even easier :-D Generally this is a "template" for all new Tokenizers, so be carefull when changing it, it will impact almost ALL symbol tokenizers in OpenGrok ... Created on August 24, 2009

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.lucene.analysis.tokenattributes.OffsetAttribute	offsetAtt
protected org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute	posIncrAtt
protected org.apache.lucene.analysis.tokenattributes.TermAttribute	termAtt

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Constructor Summary

Constructors

Constructor and Description
JFlexTokenizer()

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
boolean	incrementToken() This will reinitalize internal AttributeImpls, or it returns false if end of input Reader ...
void	reInit(char[] contents, int length) Reinitialize the tokenizer with new contents.
protected void	setAttribs(java.lang.String str, int start, int end)
abstract void	yyclose()
abstract boolean	yylex()
abstract void	yyreset(java.io.Reader reader)

Methods inherited from class org.apache.lucene.analysis.Tokenizer

correctOffset, reset

Methods inherited from class org.apache.lucene.analysis.TokenStream

end, reset

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

termAtt

protected org.apache.lucene.analysis.tokenattributes.TermAttribute termAtt

offsetAtt

protected org.apache.lucene.analysis.tokenattributes.OffsetAttribute offsetAtt

posIncrAtt

protected org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute posIncrAtt

Constructor Detail

JFlexTokenizer

public JFlexTokenizer()

Method Detail

yylex

public abstract boolean yylex()
                       throws java.io.IOException

Throws:

java.io.IOException

yyreset

public abstract void yyreset(java.io.Reader reader)

yyclose

public abstract void yyclose()
                      throws java.io.IOException

Throws:

java.io.IOException

reInit

public final void reInit(char[] contents,
          int length)

Reinitialize the tokenizer with new contents.

Parameters:

contents - a char buffer with text to tokenize

length - the number of characters to use from the char buffer

close

public final void close()
                 throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Overrides:

close in class org.apache.lucene.analysis.Tokenizer

Throws:

java.io.IOException

incrementToken

public boolean incrementToken()
                       throws java.io.IOException

This will reinitalize internal AttributeImpls, or it returns false if end of input Reader ...

Specified by:

incrementToken in class org.apache.lucene.analysis.TokenStream

Returns:

false if no more tokens, otherwise true

Throws:

java.io.IOException

setAttribs

protected void setAttribs(java.lang.String str,
              int start,
              int end)