org.opensolaris.opengrok.analysis.plain

Class PlainSymbolTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.opensolaris.opengrok.analysis.JFlexTokenizer

org.opensolaris.opengrok.analysis.plain.PlainSymbolTokenizer

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public class PlainSymbolTokenizer
extends JFlexTokenizer

This class is a scanner generated by JFlex 1.4.3 on 10/28/13 12:18 PM from the specification file /builddir/build/BUILD/opengrok-0.11.1-src/src/org/opensolaris/opengrok/analysis/plain/PlainSymbolTokenizer.lex

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State

Field Summary

Fields

Modifier and Type	Field and Description
static int	YYEOF This character denotes the end of file
static int	YYINITIAL lexical states

Fields inherited from class org.opensolaris.opengrok.analysis.JFlexTokenizer

offsetAtt, posIncrAtt, termAtt

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Constructor Summary

Constructors

Constructor and Description
PlainSymbolTokenizer(java.io.InputStream in) Creates a new scanner.
PlainSymbolTokenizer(java.io.Reader in) Creates a new scanner There is also a java.io.InputStream version of this constructor.

Method Summary

Methods

Modifier and Type	Method and Description
void	yybegin(int newState) Enters a new lexical state
char	yycharat(int pos) Returns the character at position pos from the matched text.
void	yyclose() Closes the input stream.
int	yylength() Returns the length of the matched text region.
boolean	yylex() Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
void	yypushback(int number) Pushes the specified amount of characters back into the input stream.
void	yyreset(java.io.Reader reader) Resets the scanner to read from a new input stream.
int	yystate() Returns the current lexical state.
java.lang.String	yytext() Returns the text matched by the current regular expression.

Methods inherited from class org.opensolaris.opengrok.analysis.JFlexTokenizer

close, incrementToken, reInit, setAttribs

Methods inherited from class org.apache.lucene.analysis.Tokenizer

correctOffset, reset

Methods inherited from class org.apache.lucene.analysis.TokenStream

end, reset

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

YYEOF

public static final int YYEOF

This character denotes the end of file

See Also:

Constant Field Values

YYINITIAL

public static final int YYINITIAL

lexical states

See Also:

Constant Field Values

Constructor Detail

PlainSymbolTokenizer

public PlainSymbolTokenizer(java.io.Reader in)

Creates a new scanner There is also a java.io.InputStream version of this constructor.

Parameters:

in - the java.io.Reader to read input from.

PlainSymbolTokenizer

public PlainSymbolTokenizer(java.io.InputStream in)

Creates a new scanner. There is also java.io.Reader version of this constructor.

Parameters:

in - the java.io.Inputstream to read input from.

Method Detail

yyclose

public final void yyclose()
                   throws java.io.IOException

Closes the input stream.

Specified by:

yyclose in class JFlexTokenizer

Throws:

java.io.IOException

yyreset

public final void yyreset(java.io.Reader reader)

Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL.

Specified by:

yyreset in class JFlexTokenizer

Parameters:

reader - the new input stream

yystate

public final int yystate()

Returns the current lexical state.

yybegin

public final void yybegin(int newState)

Enters a new lexical state

Parameters:

newState - the new lexical state

yytext

public final java.lang.String yytext()

Returns the text matched by the current regular expression.

yycharat

public final char yycharat(int pos)

Returns the character at position pos from the matched text. It is equivalent to yytext().charAt(pos), but faster

Parameters:

pos - the position of the character to fetch. A value from 0 to yylength()-1.

Returns:

the character at position pos

yylength

public final int yylength()

Returns the length of the matched text region.

yypushback

public void yypushback(int number)

Pushes the specified amount of characters back into the input stream. They will be read again by then next call of the scanning method

Parameters:

number - the number of characters to be read again. This number must not be greater than yylength()!

yylex

public boolean yylex()
              throws java.io.IOException

Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.

Specified by:

yylex in class JFlexTokenizer

Returns:

the next token

Throws:

java.io.IOException - if any I/O-Error occurs