java - Apache StrTokenizer 如何转义字符串文字中的引号和逗号

Question

我有这段代码来解析一些 csv，考虑到双引号会转义字符串文字中的引号(如 Apache 文档中所述)

private void test() {
    char quote = '\'';
    char delim = ',';
    // should be split into [comma, comma], [quote ', comma]
    String inputListValues = "'comma, comma', 'quote '', comma'";
    StrTokenizer st = new StrTokenizer(inputListValues, delim, quote);
    List<String> vals = new ArrayList<String>();
    while (st.hasNext()) {
        vals.add(st.nextToken().trim());
    }
    System.out.println(vals);

    // should be split into [quote ', comma], [comma, comma]
    String inputListValues2 = "'quote '', comma', 'comma, comma'";
    StrTokenizer st2 = new StrTokenizer(inputListValues2, delim, quote);
    List<String> vals2 = new ArrayList<String>();
    while (st2.hasNext()) {
        vals2.add(st2.nextToken().trim());
    }
    System.out.println(vals2);
}

输出为

vals    ArrayList<E>  (id=1088) 
    [0] "comma, comma" (id=1063)    
    [1] "'quote ''" (id=1036)   
    [2] "comma'" (id=2123)  

vals2   ArrayList<E>  (id=2296) 
    [0] "quote ', comma" (id=1920)  
    [1] "'comma" (id=1852)  
    [2] "comma'" (id=1316)  

我期待解析 2 个项目:[quote ', comma], [comma, comma]

如果它根本不起作用，那将是一回事，但似乎更改顺序会导致解析改变行为。

有人知道吗？我即将使用另一个库或正则表达式。

Answer 1

这是因为我开始使用它时考虑的是“csv 解析器”，但事实并非如此。文档说

"a, ", b ,", c" - Three tokens "a, " , " b ", ", c" (quoted text untouched)

所以空格是 token 的一部分。我添加然后使用 setTrimmerMatcher，因为对于修剪器匹配器:

These characters are trimmed off on each side of the delimiter until the token or quote is found.

代码最终是

StrTokenizer st = new StrTokenizer(toTokenize, DELIM_CHAR, QUOTE_CHAR);
// by default this is a STRING matching, not csv parser, so spaces count as part of the token
//  ie "a, ", b ,", c" - Three tokens "a, " , " b ", ", c" (quoted text untouched)
// thus we set the trimmer matcher, which "are trimmed off on each side of the delimiter until the token or quote is found."
st.setTrimmerMatcher(StrMatcher.trimMatcher());