org.apache.lucene.analysis.compound

Class HyphenationCompoundWordTokenFilter

java.lang.Object

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.TokenFilter

org.apache.lucene.analysis.compound.CompoundWordTokenFilterBase

org.apache.lucene.analysis.compound.HyphenationCompoundWordTokenFilter

public class HyphenationCompoundWordTokenFilter
extends CompoundWordTokenFilterBase

A TokenFilter that decomposes compound words found in many germanic languages "Donaudampfschiff" becomes Donau, dampf, schiff so that you can find "Donaudampfschiff" even when you only enter "schiff" It uses a hyphenation grammar and a word dictionary to achieve this.

Field Summary

Fields inherited from class org.apache.lucene.analysis.compound.CompoundWordTokenFilterBase

DEFAULT_MAX_SUBWORD_SIZE, DEFAULT_MIN_SUBWORD_SIZE, DEFAULT_MIN_WORD_SIZE, dictionary, maxSubwordSize, minSubwordSize, minWordSize, onlyLongestMatch, tokens

Fields inherited from class org.apache.lucene.analysis.TokenFilter

input

Constructor Summary

Constructors

Constructor and Description
HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, java.util.Set dictionary)
HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, java.util.Set dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, java.lang.String[] dictionary)
HyphenationCompoundWordTokenFilter(TokenStream input, HyphenationTree hyphenator, java.lang.String[] dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	decomposeInternal(Token token)
static HyphenationTree	getHyphenationTree(java.io.File hyphenationFile) Create a hyphenator tree
static HyphenationTree	getHyphenationTree(java.io.Reader hyphenationReader) Create a hyphenator tree
static HyphenationTree	getHyphenationTree(java.lang.String hyphenationFilename) Create a hyphenator tree

Methods inherited from class org.apache.lucene.analysis.compound.CompoundWordTokenFilterBase

addAllLowerCase, createToken, decompose, makeDictionary, makeLowerCaseCopy, next

Methods inherited from class org.apache.lucene.analysis.TokenFilter

close, reset

Methods inherited from class org.apache.lucene.analysis.TokenStream

next

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HyphenationCompoundWordTokenFilter

public HyphenationCompoundWordTokenFilter(TokenStream input,
                                  HyphenationTree hyphenator,
                                  java.lang.String[] dictionary,
                                  int minWordSize,
                                  int minSubwordSize,
                                  int maxSubwordSize,
                                  boolean onlyLongestMatch)

Parameters:

input - the token stream to process

hyphenator - the hyphenation pattern tree to use for hyphenation

dictionary - the word dictionary to match against

minWordSize - only words longer than this get processed

minSubwordSize - only subwords longer than this get to the output stream

maxSubwordSize - only subwords shorter than this get to the output stream

onlyLongestMatch - Add only the longest matching subword to the stream

HyphenationCompoundWordTokenFilter

public HyphenationCompoundWordTokenFilter(TokenStream input,
                                  HyphenationTree hyphenator,
                                  java.lang.String[] dictionary)

Parameters:

input - the token stream to process

hyphenator - the hyphenation pattern tree to use for hyphenation

dictionary - the word dictionary to match against

HyphenationCompoundWordTokenFilter

public HyphenationCompoundWordTokenFilter(TokenStream input,
                                  HyphenationTree hyphenator,
                                  java.util.Set dictionary)

Parameters:

input - the token stream to process

hyphenator - the hyphenation pattern tree to use for hyphenation

dictionary - the word dictionary to match against. If this is a CharArraySet it must have set ignoreCase=false and only contain lower case strings.

HyphenationCompoundWordTokenFilter

public HyphenationCompoundWordTokenFilter(TokenStream input,
                                  HyphenationTree hyphenator,
                                  java.util.Set dictionary,
                                  int minWordSize,
                                  int minSubwordSize,
                                  int maxSubwordSize,
                                  boolean onlyLongestMatch)

Parameters:

input - the token stream to process

hyphenator - the hyphenation pattern tree to use for hyphenation

dictionary - the word dictionary to match against. If this is a CharArraySet it must have set ignoreCase=false and only contain lower case strings.

minWordSize - only words longer than this get processed

minSubwordSize - only subwords longer than this get to the output stream

maxSubwordSize - only subwords shorter than this get to the output stream

onlyLongestMatch - Add only the longest matching subword to the stream

Method Detail

getHyphenationTree

public static HyphenationTree getHyphenationTree(java.lang.String hyphenationFilename)
                                          throws java.lang.Exception

Create a hyphenator tree

Parameters:

hyphenationFilename - the filename of the XML grammar to load

Returns:

An object representing the hyphenation patterns

Throws:

java.lang.Exception

getHyphenationTree

public static HyphenationTree getHyphenationTree(java.io.File hyphenationFile)
                                          throws java.lang.Exception

Create a hyphenator tree

Parameters:

hyphenationFile - the file of the XML grammar to load

Returns:

An object representing the hyphenation patterns

Throws:

java.lang.Exception

getHyphenationTree

public static HyphenationTree getHyphenationTree(java.io.Reader hyphenationReader)
                                          throws java.lang.Exception

Create a hyphenator tree

Parameters:

hyphenationReader - the reader of the XML grammar to load from

Returns:

An object representing the hyphenation patterns

Throws:

java.lang.Exception

decomposeInternal

protected void decomposeInternal(Token token)

Specified by:

decomposeInternal in class CompoundWordTokenFilterBase