public class DefaultIcuTokenizerConfig extends java.lang.Object implements IcuTokenizerConfig
IcuTokenizerConfig
that is generally applicable
to many languages.
Generally tokenizes Unicode text according to UAX#29
(BreakIterator.getWordInstance(ULocale.ROOT)
),
but with the following tailorings:
Modifier and Type | Field and Description |
---|---|
static java.lang.String |
WORD_HANGUL
Token type for words containing Korean hangul.
|
static java.lang.String |
WORD_HIRAGANA
Token type for words containing Japanese hiragana.
|
static java.lang.String |
WORD_IDEO
Token type for words containing ideographic characters.
|
static java.lang.String |
WORD_KATAKANA
Token type for words containing Japanese katakana.
|
static java.lang.String |
WORD_LETTER
Token type for words that contain letters.
|
static java.lang.String |
WORD_NUMBER
Token type for words that appear to be numbers.
|
Constructor and Description |
---|
DefaultIcuTokenizerConfig(boolean cjkAsWords,
boolean myanmarAsWords)
Creates a new config.
|
Modifier and Type | Method and Description |
---|---|
boolean |
combineCJ() |
com.ibm.icu.text.BreakIterator |
getBreakIterator(int script)
Return a breakiterator capable of processing a given script.
|
java.lang.String |
getType(int script,
int ruleStatus)
Return a token type value for a given script and BreakIterator
rule status.
|
public static final java.lang.String WORD_IDEO
public static final java.lang.String WORD_HIRAGANA
public static final java.lang.String WORD_KATAKANA
public static final java.lang.String WORD_HANGUL
public static final java.lang.String WORD_LETTER
public static final java.lang.String WORD_NUMBER
public DefaultIcuTokenizerConfig(boolean cjkAsWords, boolean myanmarAsWords)
cjkAsWords
- true if cjk text should undergo dictionary-based segmentation,
otherwise text will be segmented according to UAX#29 defaults.
If this is true, all Han+Hiragana+Katakana words will be tagged as
IDEOGRAPHIC.myanmarAsWords
- true if Myanmar text should undergo dictionary-based segmentation,
otherwise it will be tokenized as syllables.public boolean combineCJ()
combineCJ
in interface IcuTokenizerConfig
public com.ibm.icu.text.BreakIterator getBreakIterator(int script)
IcuTokenizerConfig
getBreakIterator
in interface IcuTokenizerConfig
script
- scriptpublic java.lang.String getType(int script, int ruleStatus)
IcuTokenizerConfig
getType
in interface IcuTokenizerConfig
script
- scriptruleStatus
- rule status