用于加权倒排索引的 Python 3 字典

Question

首先，这是家庭作业，所以我只想提出建议。我正在编写一个生成加权倒排索引的程序。加权倒排索引是以词为键的字典；该值是一个列表列表，列表中的每个项目都包含文档编号，以及该词在文档中出现的次数。

例如，

{"a": [[1, 2],[2,1]]}
The word "a" appears twice in document 1 and once in document 2.

我正在练习两个小文件。

文件1.txt:

    Where should I go
    When I want to have
    A smoke,
    A pancake, 
    and a nap.

文件2.txt:

I do not know
Where my pancake is
I want to take a nap.

这是我的程序代码:

def cleanData(myFile):
    file = open(myFile, "r")

    data = file.read()
    wordList = []

    #All numbers and end-of-sentence punctuation
    #replaced with the empty string
    #No replacement of apostrophes
    formattedData = data.strip().lower().replace(",","")\
                 .replace(".","").replace("!","").replace("?","")\
                 .replace(";","").replace(":","").replace('"',"")\
                 .replace("1","").replace("2","").replace("3","")\
                 .replace("4","").replace("5","").replace("6","")\
                 .replace("7","").replace("8","").replace("9","")\
                 .replace("0","")

    words = formattedData.split() #creates a list of all words in the document
    for word in words:
        wordList.append(word)     #adds each word in a document to the word list
    return wordList

def main():

fullDict = {}

files = ["file1.txt", "file2.txt"]
docNumber = 1

for file in files:
    wordList = cleanData(file)

    for word in wordList:
        if word not in fullDict:
            fullDict[word] = []
            fileList = [docNumber, 1]
            fullDict[word].append(fileList)
        else:
            listOfValues = list(fullDict.values())
            for x in range(len(listOfValues)):
                if docNumber == listOfValues[x][0]:
                    listOfValues[x][1] +=1
                    fullDict[word] = listOfValues
                    break
            fileList = [docNumber,1]
            fullDict[word].append(fileList)

    docNumber +=1
return fullDict

我想做的是生成这样的东西:

{"a": [[1,3],[2,1]], "nap": [[1,1],[2,1]]}

我得到的是这样的:

{"a": [[1,1],[1,1],[1,1],[2,1]], "nap": [[1,1],[2,1]]}

它记录了所有文档中每个词的所有出现，但它单独记录了重复。我想不通。任何帮助，将不胜感激!先感谢您。 :)

Answer 1

您的代码中存在两个主要问题。

问题一

        listOfValues = list(fullDict.values())
        for x in range(len(listOfValues)):
            if docNumber == listOfValues[x][0]:

在这里，您获取字典的所有值，不考虑当前单词，并增加计数，但您应该增加与当前单词对应的列表中的计数。所以，你应该把它改成

listOfValues = fullDict[word]

问题2

        fileList = [docNumber,1]
        fullDict[word].append(fileList)

除了增加所有单词的计数外，您总是向 fullDict 添加一个新值。但您应该添加它，前提是 docNumber 不在 listOfValues 中。因此，您可以在 for 循环中使用 else，如下所示

    for word in wordList:
        if word not in fullDict:
            ....
        else:
            listOfValues = fullDict[word]
            for x in range(len(listOfValues)):
                ....
            else:
                fileList = [docNumber, 1]
                fullDict[word].append(fileList)

进行这两项更改后，我得到了以下输出

{'a': [[1, 3], [2, 1]],
 'and': [[1, 1]],
 'do': [[2, 1]],
 'go': [[1, 1]],
 'have': [[1, 1]],
 'i': [[1, 2], [2, 2]],
 'is': [[2, 1]],
 'know': [[2, 1]],
 'my': [[2, 1]],
 'nap': [[1, 1], [2, 1]],
 'not': [[2, 1]],
 'pancake': [[1, 1], [2, 1]],
 'should': [[1, 1]],
 'smoke': [[1, 1]],
 'take': [[2, 1]],
 'to': [[1, 1], [2, 1]],
 'want': [[1, 1], [2, 1]],
 'when': [[1, 1]],
 'where': [[1, 1], [2, 1]]}

---

很少有改进代码的建议。

1. 您实际上可以使用字典来代替列表来存储文档编号和计数。这会让你的生活更轻松。

2. 您可以使用 collections.Counter 而不是手动计数.

3. 您可以使用一个简单的正则表达式，而不是使用多个替换，就像这样

   formattedData = re.sub(r'[.!?;:"0-9]', '', data.strip().lower())
   

如果我要清理cleanData，我会这样做

import re
def cleanData(myFile):
    with open(myFile, "r") as input_file:
        data = input_file.read()
    return re.sub(r'[.!?;:"0-9]', '', data.strip().lower()).split()

在 main 循环中，您可以像这样使用 Brad Budlong 建议的改进

def main():
    fullDict = {}
    files = ["file1.txt", "file2.txt"]
    for docNumber, currentFile in enumerate(files, 1):
        for word in cleanData(currentFile):
            if word not in fullDict:
                fullDict[word] = [[docNumber, 1]]
            else:
                for x in fullDict[word]:
                    if docNumber == x[0]:
                        x[1] += 1
                        break
                else:
                    fullDict[word].append([docNumber, 1])
    return fullDict