python - 从 NLTK 中的 Text.similar() 和 ContextIndex.similar_words() 生成的单词按频率排序？

Question

我正在使用这两个函数来查找相似的词，它们返回不同的列表。我想知道这些函数是否按最频繁关联到最不频繁关联排序？

Answer 1

ContextIndex.similar_words(word)计算每个单词的相似度分数作为每个上下文中频率乘积的总和。 Text.similar()简单地计算单词共享的独特上下文的数量。

similar_words() 似乎包含 NLTK 2.0 中的错误。请参阅 nltk/text.py 中的定义:

def similar_words(self, word, n=20):
    scores = defaultdict(int)
    for c in self._word_to_contexts[self._key(word)]:
        for w in self._context_to_words[c]:
            if w != word:
                print w, c, self._context_to_words[c][word], self._context_to_words[c][w]
                scores[w] += self._context_to_words[c][word] * self._context_to_words[c][w]
    return sorted(scores, key=scores.get)[:n]

返回的单词列表应按相似度得分降序排列。将返回语句替换为:

return sorted(scores, key=scores.get)[::-1][:n]

在 similar() 中，对 similar_words() 的调用被注释掉了，可能是因为这个错误。

def similar(self, word, num=20):
    if '_word_context_index' not in self.__dict__:
        print 'Building word-context index...'
        self._word_context_index = ContextIndex(self.tokens,
                                                filter=lambda x:x.isalpha(),
                                                key=lambda s:s.lower())

#   words = self._word_context_index.similar_words(word, num)

    word = word.lower()
    wci = self._word_context_index._word_to_contexts
    if word in wci.conditions():
        contexts = set(wci[word])
        fd = FreqDist(w for w in wci.conditions() for c in wci[w]
                      if c in contexts and not w == word)
        words = fd.keys()[:num]
        print tokenwrap(words)
    else:
        print "No matches"

注意:在 FreqDist 中，与 dict 不同，keys() 返回一个排序列表。

示例:

import nltk

text = nltk.Text(word.lower() for word in nltk.corpus.brown.words())
text.similar('woman')

similar_words = text._word_context_index.similar_words('woman')
print ' '.join(similar_words)

输出:

man day time year car moment world family house boy child country
job state girl place war way case question   # Text.similar()

#man ('a', 'who') 9 39   # output from similar_words(); see following explanation
#girl ('a', 'who') 9 6
#[...]

man number time world fact end year state house way day use part
kind boy matter problem result girl group   # ContextIndex.similar_words()

fd，similar() 中的频率分布，是每个单词上下文数量的统计:

fd = [('man', 52), ('day', 30), ('time', 30), ('year', 28), ('car', 24), ('moment', 24), ('world', 23) ...]

对于每个上下文中的每个单词，similar_words() 计算频率乘积的总和:

man ('a', 'who') 9 39  # 'a man who' occurs 39 times in text;
                       # 'a woman who' occurs 9 times
                       # Similarity score for the context is the product:
                       #     score['man'] = 9 * 39
girl ('a', 'who') 9 6
writer ('a', 'who') 9 4
boy ('a', 'who') 9 3
child ('a', 'who') 9 2
dealer ('a', 'who') 9 2
...
man ('a', 'and') 6 11  # score += 6 * 11
...
man ('a', 'he') 4 6    # score += 4 * 6
...
[49 more occurrences of 'man']