Python re.sub、re.split 无法分割长段落中的单词

Question

我正在尝试从存储在光盘上的 HTML 文档中创建单词列表。当我尝试拆分单词并将它们添加到我的单词向量中时，我最终陷入了困惑。

 def get_word_vector(self):
  line = self.soup.get_text()
  re.sub("s/(\\u[a-e0-9][a-e0-9][a-e0-9]//|\\n)","",line)
  for word in line.split("\s+"):
   for the_word in word.split("[,.\"\\/?!@#$%^&*\{\}\[\]]+"):
    if the_word not in self.word_vector:
     self.word_vector[the_word]=0
    self.word_vector[the_word]+=1
    self.doc_length=self.doc_length+1
  for keys in self.word_vector:
   print "%r: %r" % (keys, self.word_vector[keys]) #So I can see whats happening

当我在 wiki 页面上测试这个时，我得到了(小样本):

"Horse Markings"\n"Horse and Pony Head Markings"\n"Horse and Pony Leg Markings"\n"Identifying Horse parts and markings," Adapted From: Horses For Dummies, 2nd Edition.\n\n\n\n\n\n\n[hide]

作为单个“单词”。该文档正在阅读 BS4，如下所示:

  self.soup = BeautifulSoup(open(fullpath,"r"))

我不明白为什么会发生这种情况。我猜正则表达式失败是因为它错误？？？

Answer 1

还有一个替代选项:通过 get_text() 获取文本然后使用 nltk.tokenize从文本中获取单词列表。这里的重点不是重新发明轮子，而是使用专门的工具来完成特定的工作:BeautifulSoup 用于 HTML 解析，nltk 用于文本处理:

from urllib2 import urlopen
from bs4 import BeautifulSoup
from nltk.tokenize import RegexpTokenizer

soup = BeautifulSoup(urlopen('http://en.wikipedia.org/wiki/Stack_Overflow'))
tokenizer = RegexpTokenizer(r'\w+')
print tokenizer.tokenize(soup.get_text())

打印:

[u'Stack', u'Overflow', u'Wikipedia', u'the', u'free', u'encyclopedia', ... ]