python - 将多个数组值匹配到 csv 文件中的行很慢

Question

我有一个 numpy 数组，由大约 1200 个数组组成，每个数组包含 10 个值。 np.shape = 1200, 10。每个元素的值都在 0 到 570 万之间。

接下来我有一个包含 3800 行的 .csv 文件。每行包含 2 个值。第一个值表示一个范围，第二个值是一个标识符。 .csv 文件的第一行和最后 5 行:

509,47222
1425,47220
2404,47219
4033,47218
6897,47202
...,...
...,...
...,...
5793850,211
5794901,186
5795820,181
5796176,43
5796467,33

第一列一直上升，直到达到 570 万。对于 numpy 数组中的每个值，我想检查 .csv 文件的第一列。例如，我有值 3333，这意味着属于 3333 的标识符是 47218。每行表示从前一行的第一列到本行的第一列，例如:2404 - 4033 标识符是47218。

现在我想获取 numpy 数组中每个值的标识符，然后我想保护标识符以及在 numpy 数组中找到该标识符的频率。这意味着我需要在 12000 行的 csv 文件上循环 3800 次，然后++ 一个整数。这个过程大约需要 30 秒，这太长了。

这是我目前使用的代码:

    numpy_file = np.fromfile(filename, dtype=np.int32)
    #some code to format numpy_file correctly

    with open('/identifer_file.csv') as read_file:
        csv_reader = csv.reader(read_file, delimiter=',')
        csv_reader = list(csv_reader)

        identifier_dict = {}
        for numpy_array in numpy_file:
            for numpy_value in numpy_array:
                #there are 12000 numpy_value in numpy_file

                for row in csv_reader:
                    last_identifier = 0

                    if numpy_value <= int(row[0]):
                        last_identifier = int(row[1])

                        #adding the frequency of the identifier in numpy_file to a dict
                        if last_identifier in identifier_dict:
                            identifier_dict[last_identifier] += 1
                        else:
                            identifier_dict[last_identifier] = 1
                    else:
                        continue

                    break

        for x, y in identifier_dict.items():
            if(y > 40):
                print("identifier: {} amount of times found: {}".format(x, y))

我应该实现什么算法来加速这个过程？

编辑我试过将 numpy 数组折叠成一维数组，所以它有 12000 个值。这对速度没有实际影响。最新测试33秒

Answer 1

设置:

import numpy as np
import collections
np.random.seed(100)
numpy_file = np.random.randint(0, 5700000, (1200,10))

#'''range, identifier'''
read_file = io.StringIO('''509,47222
1425,47220
2404,47219
4033,47218
6897,47202
5793850,211
5794901,186
5795820,181
5796176,43
5796467,33''')

csv_reader = csv.reader(read_file, delimiter=',')
csv_reader = list(csv_reader)

# your example code put in a function and adapted for the setup above
def original(numpy_file,csv_reader):
    identifier_dict = {}
    for numpy_array in numpy_file:
        for numpy_value in numpy_array:
            #there are 12000 numpy_value in numpy_file

            for row in csv_reader:
                last_identifier = 0

                if numpy_value <= int(row[0]):
                    last_identifier = int(row[1])

                    #adding the frequency of the identifier in numpy_file to a dict
                    if last_identifier in identifier_dict:
                        identifier_dict[last_identifier] += 1
                    else:
                        identifier_dict[last_identifier] = 1
                else:
                    continue

                break

#    for x, y in identifier_dict.items():
#        if(y > 40):
#            print("identifier: {} amount of times found: {}".format(x, y))
    return identifier_dict        

---

三个解决方案，每个都向量化一些操作。第一个函数消耗最少的内存，最后一个消耗最多的内存。

def first(numpy_file,r):
    '''compare each value in the array to the entire first column of the csv'''
    alternate = collections.defaultdict(int)
    for value in np.nditer(numpy_file):
        comparison = value < r[:,0]
        identifier = r[:,1][comparison.argmax()]
        alternate[identifier] += 1
    return alternate

def second(numpy_file,r):
    '''compare each row of the array to the first column of csv'''
    alternate = collections.defaultdict(int)
    for row in numpy_file:
        comparison = row[...,None] < r[:,0]
        indices = comparison.argmax(-1)
        id_s = r[:,1][indices]
        for thing in id_s:
            #adding the frequency of the identifier in numpy_file to a dict
            alternate[thing] += 1
    return alternate

def third(numpy_file,r):
    '''compare the whole array to the first column of csv'''
    alternate = collections.defaultdict(int)
    other = collections.Counter()
    comparison = numpy_file[...,None] < r[:,0]
    indices = comparison.argmax(-1)
    id_s = r[:,1][indices]
    other = collections.Counter(map(int,np.nditer(id_s)))
    return other

这些函数需要将 csv 文件读入一个 numpy 数组:

read_file.seek(0)    #io.StringIO object from setup
csv_reader = csv.reader(read_file, delimiter=',')
r = np.array([list(map(int,thing)) for thing in csv_reader])

one = first(numpy_file, r)
two = second(numpy_file,r)
three = third(numpy_file,r)
assert zero == one
assert zero == two
assert zero == three