python - 如何解决 numpy 中的内存 View 错误？

Question

在此代码片段中 train_dataset , test_dataset和 valid_dataset属于 numpy.ndarray 类型.

def check_overlaps(images1, images2):
    images1.flags.writeable=False
    images2.flags.writeable=False
    print(type(images1))
    print(type(images2))
    start = time.clock()
    hash1 = set([hash(image1.data) for image1 in images1])
    hash2 = set([hash(image2.data) for image2 in images2])
    all_overlaps = set.intersection(hash1, hash2)
    return all_overlaps, time.clock()-start

r, execTime = check_overlaps(train_dataset, test_dataset)    
print("# overlaps between training and test sets:", len(r), "execution time:", execTime)
r, execTime = check_overlaps(train_dataset, valid_dataset)   
print("# overlaps between training and validation sets:", len(r), "execution time:", execTime) 
r, execTime = check_overlaps(valid_dataset, test_dataset) 
print("# overlaps between validation and test sets:", len(r), "execution time:", execTime)

但这会产生以下错误:(格式化为代码以使其可读!)

ValueError                                Traceback (most recent call last)
<ipython-input-14-337e73a1cb14> in <module>()
     12     return all_overlaps, time.clock()-start
     13 
---> 14 r, execTime = check_overlaps(train_dataset, test_dataset)
     15 print("# overlaps between training and test sets:", len(r), "execution time:", execTime)
     16 r, execTime = check_overlaps(train_dataset, valid_dataset)

<ipython-input-14-337e73a1cb14> in check_overlaps(images1, images2)
      7     print(type(images2))
      8     start = time.clock()
----> 9     hash1 = set([hash(image1.data) for image1 in images1])
     10     hash2 = set([hash(image2.data) for image2 in images2])
     11     all_overlaps = set.intersection(hash1, hash2)

<ipython-input-14-337e73a1cb14> in <listcomp>(.0)
      7     print(type(images2))
      8     start = time.clock()
----> 9     hash1 = set([hash(image1.data) for image1 in images1])
     10     hash2 = set([hash(image2.data) for image2 in images2])
     11     all_overlaps = set.intersection(hash1, hash2)

ValueError: memoryview: hashing is restricted to formats 'B', 'b' or 'c'

现在的问题是我什至不知道这个错误是什么意思，更不用说考虑纠正它了。有什么帮助吗？

Answer 1

问题是您的散列数组方法仅适用于 python2。因此，一旦您尝试计算 hash(image1.data)，您的代码就会失败。报错信息告诉你只有memoryview的格式无符号字节('B')，字节('b')的单字节( 'c') 是受支持的，我还没有找到一种方法可以在不复制的情况下从 np.ndarray 中获取这样的 View 。我想出的唯一方法包括复制数组，这在您的应用程序中可能不可行，具体取决于您的数据量。话虽如此，您可以尝试将您的功能更改为:

def check_overlaps(images1, images2):
    start = time.clock()
    hash1 = set([hash(image1.tobytes()) for image1 in images1])
    hash2 = set([hash(image2.tobytes()) for image2 in images2])
    all_overlaps = set.intersection(hash1, hash2)
    return all_overlaps, time.clock()-start