本文整理了Java中org.languagetool.tokenizers.WordTokenizer.getTokenizingCharacters()方法的一些代码示例，展示了WordTokenizer.getTokenizingCharacters()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。WordTokenizer.getTokenizingCharacters()方法的具体详情如下：
包路径：org.languagetool.tokenizers.WordTokenizer
类名称：WordTokenizer
方法名：getTokenizingCharacters

WordTokenizer.getTokenizingCharacters介绍

暂无

代码示例

代码示例来源：origin: languagetool-org/languagetool

@Override
public List<String> tokenize(String text) {
 List<String> l = new ArrayList<>();
 StringTokenizer st = new StringTokenizer(text, getTokenizingCharacters(), true);
 while (st.hasMoreElements()) {
  l.add(st.nextToken());
 }
 return joinEMailsAndUrls(l);
}

代码示例来源：origin: org.languagetool/language-tl

@Override
 public String getTokenizingCharacters() {
  return super.getTokenizingCharacters() + "-";
 } 
}

代码示例来源：origin: org.languagetool/language-pl

public PolishWordTokenizer() {
 plTokenizing = super.getTokenizingCharacters() + "–";   // n-dash
}

代码示例来源：origin: org.languagetool/language-nl

public DutchWordTokenizer() {
 //remove the apostrophe etc. from the standard tokenizing characters:
 String chars = super.getTokenizingCharacters();
 for (String quote : QUOTES) {
  chars = chars.replace(quote, "");
 }
 nlTokenizingChars = chars;
}

代码示例来源：origin: org.languagetool/languagetool-core

@Override
public List<String> tokenize(String text) {
 List<String> l = new ArrayList<>();
 StringTokenizer st = new StringTokenizer(text, getTokenizingCharacters(), true);
 while (st.hasMoreElements()) {
  l.add(st.nextToken());
 }
 return joinEMailsAndUrls(l);
}