python - 快速映射/修改大量 Python 字典中的值？

Question

我有一些代码正在尝试加速。也许我得到的是正确的，但每当我在 StackOverflow 上提问时，通常有人知道一个聪明的小技巧“使用 map !”、“试试这个 lambda”或“导入迭代工具”，我希望有人能在这里提供帮助。这是我关心的代码部分:

#slowest part from here....
for row_dict in json_data:
    row_dict_clean = {}
    for key, value in row_dict.items():
        value_clean = get_cleantext(value)
        row_dict_clean[key] = value_clean
    json_data_clean.append(row_dict_clean)
    total += 1
#to here...

这个概念很简单。我有一个数百万长的 list，其中包含字典，我需要通过一点清洁器来运行每个 value。然后我得到了一个很好的清理字典列表。是否有任何我不知道应该使用的聪明的 iterate 工具？这里有一个更完整的 MVE 来帮助使用它:

def get_json_data_clean(json_data):
    json_data_clean = []
    total = 0
    #slowest part from here....
    for row_dict in json_data:
        row_dict_clean = {}
        for key, value in row_dict.items():
            value_clean = get_cleantext(value)
            row_dict_clean[key] = value_clean
        json_data_clean.append(row_dict_clean)
        total += 1
    #to here...
    return json_data_clean

def get_cleantext(value):
    #do complex cleaning stuffs on the string, I can't change what this does
    value = value.replace("bad", "good")
    return value

json_data = [
    {"key1":"some bad",
     "key2":"bad things",
     "key3":"extra bad"},
    {"key1":"more bad stuff",
     "key2":"wow, so much bad",
     "key3":"who dis?"},
    # a few million more dictionaries
    {"key1":"so much bad stuff",
     "key2":"the bad",
     "key3":"the more bad"},
]

json_data_clean = get_json_data_clean(json_data)
print(json_data_clean)

每当我嵌套 for 循环时，脑子里就会响起一个小铃铛，可能有更好的方法来做到这一点。感谢您的帮助!

Answer 1

一定要问聪明人 https://codereview.stackexchange.com/ ，但作为一种快速修复，您似乎只需 map() 对字典列表的转换函数，如下所示:

def clean_text(value: str)-> str:
    # ...
    return value.replace("bad", "good")

def clean_dict(d: dict):
    return {k:clean_text(v) for k,v in d.items()}


json_data = [
    {"key1":"some bad",
     "key2":"bad things",
     "key3":"extra bad"},
    {"key1":"more bad stuff",
     "key2":"wow, so much bad",
     "key3":"who dis?"},
    # a few million more dictionaries
    {"key1":"so much bad stuff",
     "key2":"the bad",
     "key3":"the more bad"},
]

x = list(map(clean_dict, json_data))

遗漏的是您的 total 计数器，但它似乎永远不会离开 get_json_data_clean() 。

不确定为什么 @Daniel Gale 提出了 filter()，因为您没有删除任何值，只是转换它们。