org.apache.lucene.analysis.standard

Class StandardAnalyzer

java.lang.Object

org.apache.lucene.analysis.Analyzer

org.apache.lucene.analysis.standard.StandardAnalyzer

public class StandardAnalyzer
extends Analyzer

Filters StandardTokenizer with StandardFilter, LowerCaseFilter and StopFilter, using a list of English stop words.

Version:

$Id: StandardAnalyzer.java 692634 2008-09-06 10:58:33Z mikemccand $

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_MAX_TOKEN_LENGTH Default maximum allowed token length
static java.lang.String[]	STOP_WORDS An array containing some common English words that are usually not useful for searching.

Constructor Summary

Constructors

Constructor and Description
StandardAnalyzer() Builds an analyzer with the default stop words (STOP_WORDS).
StandardAnalyzer(boolean replaceInvalidAcronym) Deprecated. Remove in 3.X and make true the only valid value
StandardAnalyzer(java.io.File stopwords) Builds an analyzer with the stop words from the given file.
StandardAnalyzer(java.io.File stopwords, boolean replaceInvalidAcronym) Deprecated. Remove in 3.X and make true the only valid value
StandardAnalyzer(java.io.Reader stopwords) Builds an analyzer with the stop words from the given reader.
StandardAnalyzer(java.io.Reader stopwords, boolean replaceInvalidAcronym) Deprecated. Remove in 3.X and make true the only valid value
StandardAnalyzer(java.util.Set stopWords) Builds an analyzer with the given stop words.
StandardAnalyzer(java.util.Set stopwords, boolean replaceInvalidAcronym) Deprecated. Remove in 3.X and make true the only valid value
StandardAnalyzer(java.lang.String[] stopWords) Builds an analyzer with the given stop words.
StandardAnalyzer(java.lang.String[] stopwords, boolean replaceInvalidAcronym) Deprecated. Remove in 3.X and make true the only valid value

Method Summary

Methods

Modifier and Type	Method and Description
static boolean	getDefaultReplaceInvalidAcronym() Deprecated. This will be removed (hardwired to true) in 3.0
int	getMaxTokenLength()
boolean	isReplaceInvalidAcronym() Deprecated. This will be removed (hardwired to true) in 3.0
TokenStream	reusableTokenStream(java.lang.String fieldName, java.io.Reader reader) Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method.
static void	setDefaultReplaceInvalidAcronym(boolean replaceInvalidAcronym) Deprecated. This will be removed (hardwired to true) in 3.0
void	setMaxTokenLength(int length) Set maximum allowed token length.
void	setReplaceInvalidAcronym(boolean replaceInvalidAcronym) Deprecated. This will be removed (hardwired to true) in 3.0
TokenStream	tokenStream(java.lang.String fieldName, java.io.Reader reader) Constructs a StandardTokenizer filtered by a StandardFilter, a LowerCaseFilter and a StopFilter.

Methods inherited from class org.apache.lucene.analysis.Analyzer

close, getPositionIncrementGap, getPreviousTokenStream, setPreviousTokenStream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

STOP_WORDS

public static final java.lang.String[] STOP_WORDS

An array containing some common English words that are usually not useful for searching.

DEFAULT_MAX_TOKEN_LENGTH

public static final int DEFAULT_MAX_TOKEN_LENGTH

Default maximum allowed token length

See Also:

Constant Field Values

Constructor Detail

StandardAnalyzer

public StandardAnalyzer()

Builds an analyzer with the default stop words (STOP_WORDS).

StandardAnalyzer

public StandardAnalyzer(java.util.Set stopWords)

Builds an analyzer with the given stop words.

StandardAnalyzer

public StandardAnalyzer(java.lang.String[] stopWords)

Builds an analyzer with the given stop words.

StandardAnalyzer

public StandardAnalyzer(java.io.File stopwords)
                 throws java.io.IOException

Builds an analyzer with the stop words from the given file.

Throws:

java.io.IOException

See Also:

WordlistLoader.getWordSet(File)

StandardAnalyzer

public StandardAnalyzer(java.io.Reader stopwords)
                 throws java.io.IOException

Builds an analyzer with the stop words from the given reader.

Throws:

java.io.IOException

See Also:

WordlistLoader.getWordSet(Reader)

StandardAnalyzer

public StandardAnalyzer(boolean replaceInvalidAcronym)

Deprecated. Remove in 3.X and make true the only valid value

Parameters:

replaceInvalidAcronym - Set to true if this analyzer should replace mischaracterized acronyms in the StandardTokenizer See https://issues.apache.org/jira/browse/LUCENE-1068

StandardAnalyzer

public StandardAnalyzer(java.io.Reader stopwords,
                boolean replaceInvalidAcronym)
                 throws java.io.IOException

Deprecated. Remove in 3.X and make true the only valid value

Parameters:

stopwords - The stopwords to use

replaceInvalidAcronym - Set to true if this analyzer should replace mischaracterized acronyms in the StandardTokenizer See https://issues.apache.org/jira/browse/LUCENE-1068

Throws:

java.io.IOException

StandardAnalyzer

public StandardAnalyzer(java.io.File stopwords,
                boolean replaceInvalidAcronym)
                 throws java.io.IOException

Deprecated. Remove in 3.X and make true the only valid value

Parameters:

stopwords - The stopwords to use

replaceInvalidAcronym - Set to true if this analyzer should replace mischaracterized acronyms in the StandardTokenizer See https://issues.apache.org/jira/browse/LUCENE-1068

Throws:

java.io.IOException

StandardAnalyzer

public StandardAnalyzer(java.lang.String[] stopwords,
                boolean replaceInvalidAcronym)
                 throws java.io.IOException

Deprecated. Remove in 3.X and make true the only valid value

Parameters:

stopwords - The stopwords to use

replaceInvalidAcronym - Set to true if this analyzer should replace mischaracterized acronyms in the StandardTokenizer See https://issues.apache.org/jira/browse/LUCENE-1068

Throws:

java.io.IOException

StandardAnalyzer

public StandardAnalyzer(java.util.Set stopwords,
                boolean replaceInvalidAcronym)
                 throws java.io.IOException

Deprecated. Remove in 3.X and make true the only valid value

Parameters:

stopwords - The stopwords to use

replaceInvalidAcronym - Set to true if this analyzer should replace mischaracterized acronyms in the StandardTokenizer See https://issues.apache.org/jira/browse/LUCENE-1068

Throws:

java.io.IOException

Method Detail

getDefaultReplaceInvalidAcronym

public static boolean getDefaultReplaceInvalidAcronym()

Deprecated. This will be removed (hardwired to true) in 3.0

Returns:

true if new instances of StandardTokenizer will replace mischaracterized acronyms See https://issues.apache.org/jira/browse/LUCENE-1068

setDefaultReplaceInvalidAcronym

public static void setDefaultReplaceInvalidAcronym(boolean replaceInvalidAcronym)

Deprecated. This will be removed (hardwired to true) in 3.0

Parameters:

replaceInvalidAcronym - Set to true to have new instances of StandardTokenizer replace mischaracterized acronyms by default. Set to false to preseve the previous (before 2.4) buggy behavior. Alternatively, set the system property org.apache.lucene.analysis.standard.StandardAnalyzer.replaceInvalidAcronym to false. See https://issues.apache.org/jira/browse/LUCENE-1068

tokenStream

public TokenStream tokenStream(java.lang.String fieldName,
                      java.io.Reader reader)

Constructs a StandardTokenizer filtered by a StandardFilter, a LowerCaseFilter and a StopFilter.

Specified by:

tokenStream in class Analyzer

setMaxTokenLength

public void setMaxTokenLength(int length)

Set maximum allowed token length. If a token is seen that exceeds this length then it is discarded. This setting only takes effect the next time tokenStream or reusableTokenStream is called.

getMaxTokenLength

public int getMaxTokenLength()

See Also:

setMaxTokenLength(int)

reusableTokenStream

public TokenStream reusableTokenStream(java.lang.String fieldName,
                              java.io.Reader reader)
                                throws java.io.IOException

Description copied from class: Analyzer

Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method. Callers that do not need to use more than one TokenStream at the same time from this analyzer should use this method for better performance.

Overrides:

reusableTokenStream in class Analyzer

Throws:

java.io.IOException

isReplaceInvalidAcronym

public boolean isReplaceInvalidAcronym()

Deprecated. This will be removed (hardwired to true) in 3.0

Returns:

true if this Analyzer is replacing mischaracterized acronyms in the StandardTokenizer See https://issues.apache.org/jira/browse/LUCENE-1068

setReplaceInvalidAcronym

public void setReplaceInvalidAcronym(boolean replaceInvalidAcronym)

Deprecated. This will be removed (hardwired to true) in 3.0

Parameters:

replaceInvalidAcronym - Set to true if this Analyzer is replacing mischaracterized acronyms in the StandardTokenizer See https://issues.apache.org/jira/browse/LUCENE-1068