Python Pandas 列和模糊匹配+替换

Question

介绍

您好，我正在进行一个项目，该项目要求我用值替换 pandas 文本列中的字典键 - 但可能存在拼写错误。具体来说，我在 pandas 文本列中匹配名称，并将它们替换为“名字”。例如，我会将“tommy”替换为“First Name”。

但是，我意识到在我的字典无法替换的字符串列中存在名称和文本拼写错误的问题。例如“tommmmy”有额外的 m，在我的字典中不是名字。

#Create df 
d = {'message' : pd.Series(['awesome', 'my name is tommmy , please help with...', 'hi tommy , we understand your quest...'])}
names = ["tommy", "zelda", "marcon"]

#create dict 
namesdict = {r'(^|\s){}($|\s)'.format(el): r'\1FirstName\2' for el in names}

#replace 
d['message'].replace(namesdict, regex = True)



  #output 
    Out: 
0                                       awesome
1    my name is tommmy , please help with...
2    hi FirstName , we understand your quest...
dtype: object

所以 "tommmy"与 -> 中的 "tommy"不匹配，我需要处理拼写错误。我考虑过在实际的字典键和值替换之前尝试这样做，比如扫描 pandas 数据框并用适当的名称替换字符串列(“消息”)中的单词。我见过类似的方法，在特定字符串上使用索引，如 this one

但如何使用正确拼写列表匹配和替换 pandas df 中句子中的单词？我可以在 df.series 替换参数中执行此操作吗？我应该坚持使用正则表达式字符串替换吗？*

任何建议表示赞赏。

更新，尝试 Yannis 的回答

我正在尝试 Yannis 的回答，但我需要使用来自外部来源的列表，特别是美国人口普查的名字以进行匹配。但它的全名与我下载的字符串不匹配。

d = {'message' : pd.Series(['awesome', 'my name is tommy , please help with...', 'hi tommy , we understand your quest...'])}

import requests 
r = requests.get('http://deron.meranda.us/data/census-derived-all-first.txt')

#US Census first names (5000 +) 
firstnamelist = re.findall(r'\n(.*?)\s', r.text, re.DOTALL)


#turn list to string, force lower case
fnstring = ', '.join('"{0}"'.format(w) for w in firstnamelist )
fnstring  = ','.join(firstnamelist)
fnstring  = (fnstring.lower())


##turn to list, prepare it so it matches the name preceded by either the beginning of the string or whitespace.  
names = [x.strip() for x in fnstring.split(',')]




#import jellyfish 
import difflib 


def best_match(tokens, names):
    for i,t in enumerate(tokens):
        closest = difflib.get_close_matches(t, names, n=1)
        if len(closest) > 0:
            return i, closest[0]
    return None

def fuzzy_replace(x, y):
    
    names = y # just a simple replacement list
    tokens = x.split()
    res = best_match(tokens, y)
    if res is not None:
        pos, replacement = res
        tokens[pos] = "FirstName"
        return u" ".join(tokens)
    return x

d["message"].apply(lambda x: fuzzy_replace(x, names))

结果:

Out: 
0                                        FirstName
1    FirstName name is tommy , please help with...
2    FirstName tommy , we understand your quest...

但如果我使用像这样的较小列表，它会起作用:

names = ["tommy", "caitlyn", "kat", "al", "hope"]
d["message"].apply(lambda x: fuzzy_replace(x, names))

是不是名称列表较长导致了问题？

Answer 1

编辑:

将我的解决方案更改为使用 difflib。核心思想是标记您的输入文本并将每个标记与名称列表匹配。如果 best_match 找到一个匹配项，那么它会报告位置(和最佳匹配字符串)，这样您就可以用“FirstName”或您想要的任何内容替换标记。请参阅下面的完整示例:

import pandas as pd
import difflib

df = pd.DataFrame(data=[(0,"my name is tommmy , please help with"), (1, "hi FirstName , we understand your quest")], columns=["A", "message"])

def best_match(tokens, names):
    for i,t in enumerate(tokens):
        closest = difflib.get_close_matches(t, names, n=1)
        if len(closest) > 0:
            return i, closest[0]
    return None

def fuzzy_replace(x):
    names = ["tommy", "john"] # just a simple replacement list
    tokens = x.split()
    res = best_match(tokens, names)
    if res is not None:
        pos, replacement = res
        tokens[pos] = "FirstName"
        return u" ".join(tokens)
    return x

df.message.apply(lambda x: fuzzy_replace(x))

你应该得到的输出如下

0    my name is FirstName , please help with
1    hi FirstName , we understand your quest
Name: message, dtype: object

编辑2

经过讨论，我决定再试一次，使用 NLTK 进行词性标注，并仅针对名称列表的 NNP 标签(专有名词)运行模糊匹配。问题是有时标记器没有得到正确的标记，例如“嗨”也可能被标记为专有名词。但是，如果名称列表是小写的，则 get_close_matches 不会将 Hi 与名称匹配，但会匹配所有其他名称。我建议不要将 df["message"] 小写以增加 NLTK 正确标记名称的机会。一个人也可以玩 StanfordNER，但没有任何东西可以 100% 工作。这是代码:

import pandas as pd
import difflib
from nltk import pos_tag, wordpunct_tokenize
import requests 
import re

r = requests.get('http://deron.meranda.us/data/census-derived-all-first.txt')

# US Census first names (5000 +) 
firstnamelist = re.findall(r'\n(.*?)\s', r.text, re.DOTALL)

# turn list to string, force lower case
# simplified things here
names = [w.lower() for w in firstnamelist]


df = pd.DataFrame(data=[(0,"My name is Tommmy, please help with"), 
                        (1, "Hi Tommy , we understand your question"),
                        (2, "I don't talk to Johhn any longer"),
                        (3, 'Michale says this is stupid')
                       ], columns=["A", "message"])

def match_names(token, tag):
    print token, tag
    if tag == "NNP":
        best_match = difflib.get_close_matches(token, names, n=1)
        if len(best_match) > 0:
            return "FirstName" # or best_match[0] if you want to return the name found
        else:
            return token
    else:
        return token

def fuzzy_replace(x):
    tokens = wordpunct_tokenize(x)
    pos_tokens = pos_tag(tokens)
    # Every token is a tuple (token, tag)
    result = [match_names(token, tag) for token, tag in pos_tokens]
    x = u" ".join(result)
    return x

df['message'].apply(lambda x: fuzzy_replace(x))

然后我得到了输出:

0       My name is FirstName , please help with
1    Hi FirstName , we understand your question
2        I don ' t talk to FirstName any longer
3                 FirstName says this is stupid
Name: message, dtype: object