org.apache.lucene.analysis.ru

Class RussianLetterTokenizer

java.lang.Object

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.analysis.CharTokenizer

org.apache.lucene.analysis.ru.RussianLetterTokenizer

public class RussianLetterTokenizer
extends CharTokenizer

A RussianLetterTokenizer is a tokenizer that extends LetterTokenizer by additionally looking up letters in a given "russian charset". The problem with LeterTokenizer is that it uses Character.isLetter() method, which doesn't know how to detect letters in encodings like CP1252 and KOI8 (well-known problems with 0xD7 and 0xF7 chars)

Version:

$Id: RussianLetterTokenizer.java 564236 2007-08-09 15:21:19Z gsingers $

Field Summary

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Constructor Summary

Constructors

Constructor and Description
RussianLetterTokenizer(java.io.Reader in, char[] charset)

Method Summary

Methods

Modifier and Type	Method and Description
protected boolean	isTokenChar(char c) Collects only characters which satisfy Character.isLetter(char).

Methods inherited from class org.apache.lucene.analysis.CharTokenizer

next, normalize, reset

Methods inherited from class org.apache.lucene.analysis.Tokenizer

close

Methods inherited from class org.apache.lucene.analysis.TokenStream

next, reset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RussianLetterTokenizer

public RussianLetterTokenizer(java.io.Reader in,
                      char[] charset)

Method Detail

isTokenChar

protected boolean isTokenChar(char c)

Collects only characters which satisfy Character.isLetter(char).

Specified by:

isTokenChar in class CharTokenizer