java - 使用 ICUTokenizer JAVA 标记泰语句子

Question

我正在尝试使用以下代码从泰语句子中获取所有标记。它抛出异常。谁能指点我在 JAVA 中标记泰语？

    import org.apache.lucene.analysis.Analyzer.TokenStreamComponents;
import org.apache.lucene.analysis.TokenFilter;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.icu.ICUNormalizer2Filter;
import org.apache.lucene.analysis.icu.segmentation.ICUTokenizer;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;

public class Tokenizer{

    public static void main(String[] args) throws IOException {
        ICUTokenizer tokenizer = new ICUTokenizer(new StringReader("การที่ได้ต้องแสดงว่างานดี"));
        TokenFilter filter = new ICUNormalizer2Filter(tokenizer);
        TokenStreamComponents tt = new TokenStreamComponents(tokenizer, filter);
        TokenStream ts = tt.getTokenStream();
        CharTermAttribute cattr  = ts.addAttribute(CharTermAttribute.class);
        ts.reset();
        while(ts.incrementToken()){
            System.out.println(cattr.toString()+"-----");
        }
    }
}

异常情况如下

Exception in thread "main" java.lang.ExceptionInInitializerError
    at org.apache.lucene.analysis.icu.segmentation.ICUTokenizer.<init>(ICUTokenizer.java:72)
    at com.tokenizer.tt.main(tt.java:22)
Caused by: java.lang.RuntimeException: java.io.IOException: ICU data file error: Not an ICU data file
    at org.apache.lucene.analysis.icu.segmentation.DefaultICUTokenizerConfig.readBreakIterator(DefaultICUTokenizerConfig.java:128)
    at org.apache.lucene.analysis.icu.segmentation.DefaultICUTokenizerConfig.<clinit>(DefaultICUTokenizerConfig.java:66)
    ... 2 more
Caused by: java.io.IOException: ICU data file error: Not an ICU data file
    at com.ibm.icu.impl.ICUBinary.readHeader(ICUBinary.java:577)
    at com.ibm.icu.text.RBBIDataWrapper.get(RBBIDataWrapper.java:173)
    at com.ibm.icu.text.RuleBasedBreakIterator.getInstanceFromCompiledRules(RuleBasedBreakIterator.java:71)
    at org.apache.lucene.analysis.icu.segmentation.DefaultICUTokenizerConfig.readBreakIterator(DefaultICUTokenizerConfig.java:123)
    ... 3 more

Answer 1

终于想通了如何在java程序中使用ICU4J

import java.io.IOException;
import java.io.Reader;
import java.io.StringReader;
import org.apache.lucene.analysis.icu.segmentation.ICUTokenizer;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;

public class icuEstes {

public static void main(String[] args) throws IOException {
    Reader reader = new StringReader("การที่ได้ต้องแสดงว่างานดี  This is a test ກວ່າດອກ");
    ICUTokenizer icut = new ICUTokenizer();
    icut.setReader(reader);
    icut.addAttribute(CharTermAttribute.class);
    icut.reset();
    while (icut.incrementToken()) {
        System.out.println(icut.toString());
        System.out.println(icut.getAttribute(CharTermAttribute.class));
    }
    icut.close();
}}