org.apache.lucene.analysis.custom

Class CustomAnalyzer

java.lang.Object

org.apache.lucene.analysis.Analyzer

org.apache.lucene.analysis.custom.CustomAnalyzer

All Implemented Interfaces:

Closeable, AutoCloseable

public final class CustomAnalyzer
extends Analyzer

A general-purpose Analyzer that can be created with a builder-style API. Under the hood it uses the factory classes TokenizerFactory, TokenFilterFactory, and CharFilterFactory.

You can create an instance of this Analyzer using the builder:

 Analyzer ana = CustomAnalyzer.builder(Paths.get("/path/to/config/dir"))
   .withTokenizer("standard")
   .addTokenFilter("standard")
   .addTokenFilter("lowercase")
   .addTokenFilter("stop", "ignoreCase", "false", "words", "stopwords.txt", "format", "wordset")
   .build();
 

The parameters passed to components are also used by Apache Solr and are documented on their corresponding factory classes. Refer to documentation of subclasses of TokenizerFactory, TokenFilterFactory, and CharFilterFactory.

The list of names to be used for components can be looked up through: TokenizerFactory.availableTokenizers(), TokenFilterFactory.availableTokenFilters(), and CharFilterFactory.availableCharFilters().

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CustomAnalyzer.Builder Builder for CustomAnalyzer.

Nested classes/interfaces inherited from class org.apache.lucene.analysis.Analyzer

Analyzer.ReuseStrategy, Analyzer.TokenStreamComponents

Field Summary

Fields inherited from class org.apache.lucene.analysis.Analyzer

GLOBAL_REUSE_STRATEGY, PER_FIELD_REUSE_STRATEGY

Method Summary

Modifier and Type	Method and Description
static CustomAnalyzer.Builder	builder() Returns a builder for custom analyzers that loads all resources from classpath.
static CustomAnalyzer.Builder	builder(Path configDir) Returns a builder for custom analyzers that loads all resources from the given file system base directory.
static CustomAnalyzer.Builder	builder(ResourceLoader loader) Returns a builder for custom analyzers that loads all resources using the given ResourceLoader.
protected Analyzer.TokenStreamComponents	createComponents(String fieldName) Creates a new Analyzer.TokenStreamComponents instance for this analyzer.
List<CharFilterFactory>	getCharFilterFactories() Returns the list of char filters that are used in this analyzer.
int	getOffsetGap(String fieldName) Just like Analyzer.getPositionIncrementGap(java.lang.String), except for Token offsets instead.
int	getPositionIncrementGap(String fieldName) Invoked before indexing a IndexableField instance if terms have already been added to that field.
List<TokenFilterFactory>	getTokenFilterFactories() Returns the list of token filters that are used in this analyzer.
TokenizerFactory	getTokenizerFactory() Returns the tokenizer that is used in this analyzer.
protected Reader	initReader(String fieldName, Reader reader) Override this if you want to add a CharFilter chain.
String	toString()

Methods inherited from class org.apache.lucene.analysis.Analyzer

close, getReuseStrategy, getVersion, setVersion, tokenStream, tokenStream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

builder

public static CustomAnalyzer.Builder builder()

Returns a builder for custom analyzers that loads all resources from classpath. All path names given must be absolute with package prefixes.

builder

public static CustomAnalyzer.Builder builder(Path configDir)

Returns a builder for custom analyzers that loads all resources from the given file system base directory. Place, e.g., stop word files there. Files that are not in the given directory are loaded from classpath.

builder

public static CustomAnalyzer.Builder builder(ResourceLoader loader)

Returns a builder for custom analyzers that loads all resources using the given ResourceLoader.

initReader

protected Reader initReader(String fieldName,
                            Reader reader)

Description copied from class: Analyzer

Override this if you want to add a CharFilter chain.

The default implementation returns reader unchanged.

Overrides:

initReader in class Analyzer

Parameters:

fieldName - IndexableField name being indexed

reader - original Reader

Returns:

reader, optionally decorated with CharFilter(s)

createComponents

protected Analyzer.TokenStreamComponents createComponents(String fieldName)

Description copied from class: Analyzer

Creates a new Analyzer.TokenStreamComponents instance for this analyzer.

Specified by:

createComponents in class Analyzer

Parameters:

fieldName - the name of the fields content passed to the Analyzer.TokenStreamComponents sink as a reader

Returns:

the Analyzer.TokenStreamComponents for this analyzer.

getPositionIncrementGap

public int getPositionIncrementGap(String fieldName)

Description copied from class: Analyzer

Invoked before indexing a IndexableField instance if terms have already been added to that field. This allows custom analyzers to place an automatic position increment gap between IndexbleField instances using the same field name. The default value position increment gap is 0. With a 0 position increment gap and the typical default token position increment of 1, all terms in a field, including across IndexableField instances, are in successive positions, allowing exact PhraseQuery matches, for instance, across IndexableField instance boundaries.

Overrides:

getPositionIncrementGap in class Analyzer

Parameters:

fieldName - IndexableField name being indexed.

Returns:

position increment gap, added to the next token emitted from Analyzer.tokenStream(String,Reader). This value must be >= 0.

getOffsetGap

public int getOffsetGap(String fieldName)

Description copied from class: Analyzer

Just like Analyzer.getPositionIncrementGap(java.lang.String), except for Token offsets instead. By default this returns 1. This method is only called if the field produced at least one token for indexing.

Overrides:

getOffsetGap in class Analyzer

Parameters:

fieldName - the field just indexed

Returns:

offset gap, added to the next token emitted from Analyzer.tokenStream(String,Reader). This value must be >= 0.

getCharFilterFactories

public List<CharFilterFactory> getCharFilterFactories()

Returns the list of char filters that are used in this analyzer.

getTokenizerFactory

public TokenizerFactory getTokenizerFactory()

Returns the tokenizer that is used in this analyzer.

getTokenFilterFactories

public List<TokenFilterFactory> getTokenFilterFactories()

Returns the list of token filters that are used in this analyzer.

toString

public String toString()

Overrides:

toString in class Object