Python Nltk :UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 50: invalid continuation byte

Question

Traceback (most recent call last):
  File "C:/Users/rohanhm.2014/PycharmProjects/untitled1/abc", line 11, in <module>
    docs2 = [[w.lower() for w in doc]for doc in docs]
  File "C:/Users/rohanhm.2014/PycharmProjects/untitled1/abc", line 11, in <listcomp>
    docs2 = [[w.lower() for w in doc]for doc in docs]
  File "C:/Users/rohanhm.2014/PycharmProjects/untitled1/", line 11, in <listcomp>
    docs2 = [[w.lower() for w in doc]for doc in docs]
  File "C:\Python34\lib\site-packages\nltk\corpus\reader\util.py", line 291, in iterate_from
['PROJECT', 'FINAL', 'REPORT', 'Revision', 'History', 'Date', 'Version', 'Author', 'Validated', 'by', 'Purpose', '4', '-', 'Dec', '-', '13', '0', '.', '1', 'EA', 'Initial', 'Document', '1', '/', '8', '/', '2014', '0', '.', '2', 'EA', '&', 'AHE', 'Combined', 'the', 'copy', 'for', 'both', 'MOE', 'and', 'MOA', '.', '1', '/', '8', '/', '2014', '0', '.', '3']
    tokens = self.read_block(self._stream)
  File "C:\Python34\lib\site-packages\nltk\corpus\reader\plaintext.py", line 117, in _read_word_block
    words.extend(self._word_tokenizer.tokenize(stream.readline()))
  File "C:\Python34\lib\site-packages\nltk\data.py", line 1095, in readline
    new_chars = self._read(readsize)
  File "C:\Python34\lib\site-packages\nltk\data.py", line 1322, in _read
    chars, bytes_decoded = self._incr_decode(bytes)
  File "C:\Python34\lib\site-packages\nltk\data.py", line 1352, in _incr_decode
    return self.decode(bytes, 'strict')
  File "C:\Python34\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)

UnicodeDecodeError: 'utf-8' 编解码器无法解码位置 50 中的字节 0xe9:无效的连续字节

我正在尝试使用 NLTK 对文本进行预处理。但是我一直遇到这个错误。一些想法会有所帮助

Answer 1

一些代码行会很有用。然而，我的直觉告诉你语料库阅读器对象应该处理另一种编码而不是 utf8，可能是 latin-1。

corpus = nltk.corpus.reader.PlaintextCorpusReader(
    "/path/to/files", r'.*', encoding='latin-1')

另见此处:UnicodeDecodeError, invalid continuation byte