r - 在 R 文本挖掘中从文本语料库中删除 "english"停用词时 UTF-8 无效

Question

在进行文本挖掘时，我在从包含 500 个文档的文本语料库中删除停用词时出错。我在 Ubuntu 14.04 LTS 和文本挖掘包 0.6-1 上使用 R 3.1.3。这是代码，请帮助。

unsup.corpus = Corpus(DirSource(directory.location, encoding = "UTF-8"),
                      readerControl = list(language = "en_US"))


document.collection = unsup.corpus    
document.collection = tm_map(document.collection, stripWhitespace, mc.cores = 1)    
document.collection = tm_map(document.collection, content_transformer(tolower), mc.cores = 1)    
document.collection = tm_map(document.collection, removeNumbers, mc.cores = 1)    
document.collection = tm_map(document.collection, removePunctuation, mc.cores = 1)

document.collection = tm_map(document.collection, removeWords, stopwords("english"), mc.cores = 1)

###### Error #

Error in gsub(sprintf("(*UCP)\b(%s)\b", paste(sort(words, decreasing = TRUE), : input string 21 is invalid UTF-8

Answer 1

你可以做的一件事是

document.collection = 
        tm_map(document.collection[-21], removeWords, stopwords("english"), mc.cores = 1) 

这摆脱了带有问题字符的“字符串”。

如果你想独立处理问题字符串，你可以调用

document.collection[-21] 

并对细节进行一些调查。