python - 在大数据集的 pandas 数据框中搜索和替换

Question

我有一个大小为 100 万的数据集，类型为 dataframe。

Id      description 1      bc single phase acr 2      conditioning accum 3      dsply value ac

and dictionary of size 2927 which looks like as follow:

Key     Valueaccum   accumulatorbb      baseboarddsply   display

executed the following code to replace the dictionary key found in dataframe with its value

dataset=dataset.replace(dict, regex=True)

但它会消耗更多时间来执行，即 104.07914903743769 秒用于 2000 数据集并具有 8GB RAM我需要将此代码应用于数百万个数据集。那么谁能告诉我如何减少执行时间？还有其他方法可以完成任务吗？

Answer 1

我看到预编译正则表达式有大约 15% 的改进。

但要获得最佳性能，请参阅 @unutbu's excellent solution .

import pandas as pd
import re

rep_dict = {'accum': 'accumulator', 'bb': 'baseboard', 'dsply': 'display'}
pattern = re.compile("|".join([re.escape(k) for k in rep_dict.keys()]), re.M)

def multiple_replace(string):    
    return pattern.sub(lambda x: rep_dict[x.group(0)], string)

df = pd.DataFrame({'description': ['bc single phase acr', 'conditioning accum', 'dsply value ac']})
df = pd.concat([df]*10000)

%timeit df['description'].map(multiple_replace)          # 72.8 ms per loop
%timeit df['description'].replace(rep_dict, regex=True)  # 88.6 ms per loop