python - 按行读取的文件排序平均 Python 3

Question

我有一个输入文件:(姓氏，名字，类(class)，分数)

John Smith 11 30
Anna White  9 49
Kate Balon  10 91
Кузьмин Александр 11 99

我需要根据类别对值进行分组并获得平均值

49.0 91.0 64.5

代码应该一行一行地读，我的代码可以运行但是太慢了，我该如何改进它？

from collections import defaultdict
from operator import itemgetter

import numpy

total = defaultdict(list)
with open('input', 'r', encoding='utf8') as f:
    for row in f:
        _class, range = map(float, row.rsplit(None, 2)[-2:])
        total[_class].append(range)


print(*(numpy.mean(v) for k, v in sorted(total.items(), key=itemgetter(0))))

Answer 1

正如我在评论中提到的，您无法在纯 Python 中做很多事情来加快速度。我有几个小的优化。第一个 (alt1) 不会将组标识符字符串转换为 float (这是一项开销很大的操作)。第二个 (alt2) 使用带有预定义组的标准字典。第三个 (alt3) 使用列表而不是字典。

from collections import defaultdict
from operator import itemgetter
import random
from io import StringIO 
import numpy as np

# random data for benchmarks 
data = '\n'.join('first last {} {}'.format(random.randrange(1, 12), random.random()) for _ in range(1000))

def base(handle):
    # This is your implementation
    total = defaultdict(list)
    for row in handle:
        _class, range = map(float, row.rsplit(None, 2)[-2:])
        total[_class].append(range)
    return [np.mean(v) for k, v in sorted(total.items(), key=itemgetter(0))]

def alt1(handle):
    groups = defaultdict(list)
    for row in handle:
        group, value = row.rsplit(None, 2)[-2:]
        groups[group].append(float(value))
    return [np.mean(v) for k, v in sorted(groups.items(), key=itemgetter(0))]

def alt2(handle):
    groups = {str(i): [] for i in range(1, 12)}
    for row in handle:
        key, val = row.rsplit(None, 2)[-2:]
        groups[key].append(float(val))
    return [np.mean(group) for _, group in sorted(groups.items(), key=itemgetter(0))]

def alt3(handle):
    groups = [[] for _ in range(11)]
    for row in handle:
        key, val = row.rsplit(None, 2)[-2:]
        groups[int(key)-1].append(float(val))
    return [np.mean(group) for group in groups if group]

我想不出其他重要的优化。让我们看一些基准:

In [2]: %timeit base(StringIO(data))
1.18 ms ± 36.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [3]: %timeit alt1(StringIO(data))
937 µs ± 30.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [4]: %timeit alt2(StringIO(data))
941 µs ± 30.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [5]: %timeit alt3(StringIO(data))
1.08 ms ± 40.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

所有三种替代方案都比原始实现更快。 alt1 和 alt2 具有相同的性能并且速度明显更快。你可能想给他们一个机会。