nlp - Brown Corpus在基于WordNet的语义相似度测量中的作用是什么

Question

我遇到了几种使用 WordNet 的结构和层次结构来测量语义相似性的方法，例如江康拉思测度(JNC)、雷斯尼克测度(RES)、林测度(LIN)等。

使用 NLTK 测量它们的方式是:

sim2=wn.jcn_similarity(entry1,entry2,brown_ic)
sim3=entry1.res_similarity(entry2, brown_ic)
sim4=entry1.lin_similarity(entry2,brown_ic)

如果 WordNet 是计算语义相似度的基础，那么这里的 Brown Corpus 有什么用呢？

Answer 1

看一下 NLTK howto for wordnet. 处的解释

具体来说，*_ic 符号是信息内容。

synset1.res_similarity(synset2, ic): Resnik Similarity: Return a score denoting how similar two word senses are, based on the Information Content (IC) of the Least Common Subsumer (most specific ancestor node). Note that for any similarity measure that uses information content, the result is dependent on the corpus used to generate the information content and the specifics of how the information content was created.

关于 here 信息内容的更多信息:

The conventional way of measuring the IC of word senses is to combine knowledge of their hierarchical structure from an ontology like WordNet with statistics on their actual usage in text as derived from a large corpus